Compare commits

..
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 9e7c12e0d4 Pointer helper: split the member functions into files by topic
Structure only, no behaviour change; the code moves as it is (git's
--color-moved shows it). build.sh compiles ft-pointer.cpp and
pointer_*.cpp.

- pointer.h: the includes, the helpers more than one file uses
  (HandGestures, PoseHistory, FramePanel, SendTo, JsonQuote, OverlayList;
  the three functions are now inline), Clock, Frame, and Pointer.
- ft-pointer.cpp: the header comment, setup (Init, ApplyConfig, and the
  helpers only it uses), the short per-frame sections, and main().
- pointer_input.cpp: waking, the mouse's buttons, the relay's commands.
- pointer_place.cpp: panel placement for layouts.
- pointer_gaze.cpp: gaze mode, the held-back press, the click.
- pointer_holds.cpp: holds (hands, gaze precision buttons).
- pointer_keys.cpp: keyboard clicks.
- pointer_cursor.cpp: recenter, head follow, slow work, the cursor.

Helpers only one file uses stay in its anonymous namespace. Each new file
says that "the top" in its comments is ft-pointer.cpp's header comment.
hands/README.md and update-check.py now name the files that hold the code
they point at.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 11:48:43 -06:00
DeeJanuzandClaude Opus 5.5 7066b591ff Pointer helper: one function per main loop section
Structure only, no behaviour change: every frame does the same work in the
same order. main() now sets up and runs a short loop that calls the
sections, each a Pointer member named after its comment:

  HeadsetOff, GazeAwake, RelayCommands, ReadFrame, ClaimPulse,
  PauseOverlayList, LaserMode, HandRole, LastUsedWins, Recenter, GazeMode,
  Hands, GazePrecisionButtons, KeyboardClicks, CalPanelOpened, HeadSteer,
  PinchesAndGrips, HandsStopped, MarkHeld, HeadFollow, SlowWork, Cursor,
  HeldBackPress, KeyboardClickHold, Click, PollControllerButtons,
  SteamVRQuit

- The values the sections share within a frame (the poses, hmdRaw, tnow,
  eye, and handOk from PinchesAndGrips for HandsStopped) are in a Frame,
  read by ReadFrame at the same point as before. eye moved there from just
  after "last used wins"; it only depends on the poses, read once.
  Sections open with local aliases (hmd, hm, tnow, eye), so the bodies are
  the old code, dedented.
- None of the loop's 43 continues skipped the rest of a frame: 25 were in
  the relay's recvfrom loop and 18 in the controller, press, and overlay
  scans. The recvfrom loop's body is now Command(), where they're returns.
  The quit event's "return 0" is SteamVRQuit() returning true.
- The loop's lambdas are members too: HandAngles, EndHold, StartHold,
  Steer, BeginHold, HeadAngles (taking tnow or the HMD pose as a parameter),
  and Nearest with its Hit type.
- The cursor section is Cursor(): Tilt() while tilting; otherwise
  FindCursor() (collision, drag lock, occlusion; returns a Spot), DrawDot()
  (SteamVR Settings, our dot), and SendRay() (the controller pose).

Built with -Wextra -Wshadow too: no warnings. The three shadowing warnings
the old main() had ("from" twice, "now") are gone with the separate scopes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 11:46:30 -06:00
DeeJanuzandClaude Opus 5.5 a000725021 Pointer helper: the setup-time lambdas become member functions
Structure only, no behaviour change. The lambdas main() defined before its
loop become Pointer member functions with the same bodies and comments:
Wake, PressLeft, ReleaseLeft, CanAim, AimStart, ChordDrop, LeftButton,
RightButton, MouseMoveHeld, SendPose, HeadPos, Borrow, GiveBack, FindPanel,
GrabProbe, and Place (the names in the file's CamelCase for functions).
sleepMs captured nothing, so it's the free function SleepMs. Every call
site is renamed; the loop's own lambdas stay where they are until the next
commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 11:38:51 -06:00
DeeJanuzandClaude Opus 5.5 3460e610d1 Pointer helper: runtime state in a Pointer struct
Structure only, no behaviour change. main()'s runtime locals become members
of Pointer, declared in the same order with the same initial values and
comments. The local types (Gaze, Src, Hold, DevicePress, KeyPress) move in
with them, and Clock moves to file scope.

Pointer::Init does the setup, in the original order: settings, VR_Init,
the overlays, the laser width, the sockets, the controller buttons'
manifest, vrhello, the overlay list. Members that were set during setup
(sys, overlay, the three overlay handles, the sockets, systemPointer) are
set there at the same point, and so are the timestamps that started at
"now" (lastVisible, lastDebug, followAt, lastSlow).

The lambdas and the main loop sit unchanged in Pointer::Run for now; the
next commits split them up. applyConfig becomes Pointer::ApplyConfig,
since Init and the "reload" command both call it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 11:37:24 -06:00
DeeJanuzandClaude Opus 5.5 92a72b8797 Pointer helper: settings in a PointerConfig struct
Structure only, no behaviour change. The settings and loadConfig move from
main()'s locals to PointerConfig (pointer/helper/pointer_config.h) with the
same defaults, clamps, and parsing; ReadConfig, ConfDouble, and ParseIgnore
move with them. main() uses them as cfg.<name>.

Head follow and gaze mode stay runtime state. Load only records POINTER_FOLLOW
and POINTER_GAZE; applyConfig, run after each Load (at startup and on
"reload"), turns them on or off when the setting changed, as before, so a
"follow" or "gaze" command still wins until the setting itself changes.
POINTER_LASER_WIDTH is still read once at startup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 11:35:07 -06:00
DeeJanuzandClaude Opus 5.5 ff36356992 Merge branch gaze-games (PR #13, narrowed) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:18:26 -06:00
4d50739393 Gaze: leave SteamVR's gaze action alone during VR games
From curiousjtuber's PR #13: with the gaze service running, SteamVR
restarted its eye tracker every 10 to 13 s of Beat Saber, as if the
headset came off, and each restart took input focus from the game.
The PR stopped every read in a game. Only the action path reaches
SteamVR (UpdateActionState on the gaze set at overlay-global priority,
then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are
read-only files. So only the action is skipped while a scene app runs,
and gaze keeps moving the pointer over the dashboard in a game. The
action source is only used with --source action.

Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:15:57 -06:00
DeeJanuzandClaude Opus 5.5 f60da63702 Merge PR #12 and #14 (relay udev retry, catcher crash) into experimental
From curiousjtuber's PRs: the relay leaves a new input node for the next
scan until udev gives it to the input group, rather than marking it seen
after a failed open; and ft-screens' catcher takes a screen's overlays
from one copy of All() instead of begin() and end() of two temporaries,
which crashed libc++ builds on the first click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:15:04 -06:00
CuriousJ 82d2107e1d Screens: take a screen's overlays from one copy in the catcher
While a button pressed on a screen is held, UpdateCatcher checks every tick
whether the laser is still on one of the screen's overlays. It built that list
from s.All().begin() and s.All().end(), but All() returns a std::array by
value: iterators into two different temporaries, which is undefined behaviour.
A clang build of ft-screens got a garbage length, threw std::length_error, and
aborted on the first click, taking KWin and the desktop with it.
2026-10-02 09:11:46 -06:00
CuriousJ 12f2e844d5 Input relay: retry a new device until udev gives it to the input group
A new /dev/input node is root:root 0600 until udev applies GROUP=input. The
scan probed each new node once and marked it seen even when the open failed, so
a node caught in that gap was never opened. Behind a KVM, a switch brings back a
hub of devices at once: on the Frame, four nodes failed with EACCES in one switch,
the keyboard was never grabbed, and its keys went to gamescope instead of the
desktop screens. A node that isn't readable yet now waits for the next scan.
2026-10-02 09:11:46 -06:00
DeeJanuzandClaude Opus 5.5 ee2ce1a8d2 Merge PR #11 (controller click stability) into experimental
From jlneal's PR: a trigger press on a desktop screen stays put until the
laser moves more than 8 logical pixels, so controller jitter doesn't turn
a click into a drag. On top of it, only a hand controller's press starts
the filter: the 3D mouse's laser reaches the screens the same way, and the
PR as sent turned the mouse's short drags into clicks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 23:20:06 -06:00
DeeJanuzandClaude Opus 5.5 540d8c425c Click stability: only a hand controller's press starts it
The 3D mouse drives SteamVR's laser through the ft_pointer virtual
controller, so its events reach the screens the same way a controller's
do. The filter held every press, which turned the mouse's short drags
(selecting a character or two, nudging a slider) into clicks. Mark
button events from hand controllers and start the filter only on those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 23:19:58 -06:00
Codex ba6ccdfbd1 Clear reported drag state on controller release 2026-10-01 23:18:41 -06:00
Codex 77f28f0922 Prevent small desktop overlay pointer movements from starting a drag 2026-10-01 23:18:41 -06:00
DeeJanuzandClaude Opus 5.5 85532a54f3 Wait for a new container to finish setting up before entering it
container-up.sh starts the dev container in a scope of its own, so
distrobox enter finds it running and skips its wait for distrobox-init.
On a fresh install, init was still setting up passwordless sudo when
dev-container.sh ran sudo dnf install, and sudo asked for a password
with no terminal to read it from. container-up.sh now waits for
container_setup_done itself, and the container's sudo calls use -n,
so a password prompt fails at once with a clear message.

Fixes #9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 21:58:17 -06:00
DeeJanuzandClaude Opus 5.5 3864741f53 README: link the Frametop Discord
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 21:13:19 -06:00
DeeJanuzandClaude Opus 5.5 12ad93d24c Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken
A user on a fresh install got "Calibration failed: only 0 of 21 dots" with
no reason. The installer never installed our tracker, so gaze used
SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an
eye for most of the look. The panel just showed a red ring.

- install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh)
  after gaze mode, yes by default; it needs sudo, so --yes runs it only
  when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker.
  Configs that still say GAZE_TRACKER=steam (the old template) are asked
  whether to switch.
- GAZE_TRACKER=auto, the new default: ours when it's installed (the
  frame grabber, its unit, and ft-eyes' Python), else SteamVR's.
  ft-gazed rechecks every second, so installing it switches over. Input
  Settings lists Own tracker first as recommended, and says how to
  install it when it's missing (checking the host's /etc through
  /run/host from the dev container).
- The calibration panel has a note line, orange over the instructions:
  why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for
  SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply
  for ours), what a click is still waiting for after 1.5 s, and a failed
  calibration's most common reason, which the Gaze page shows too.
  steady_samples keeps the same samples as before (checked on 2037
  windows of recordings); a lost eye is named before a blink, since its
  openness reads 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 20:48:52 -06:00
DeeJanuzandClaude Opus 5.5 a7f9a8dd80 Gaze: say why gaze mode can't work yet, and open the calibration whenever it's missing
Turning gaze mode on without a calibration opened Calibrate only on the
off-to-on change, and only if it could open right then. With the headset
off, the eye tracker silent, or the panel not built, or with gaze mode
already on when the gaze service started, nothing opened and nothing said
why: the pointer just stayed a mouse.

- The gaze service now checks every second: gaze mode on, no calibration
  for the tracker in use, eyes seen -> the full calibration opens. One
  that closes unfinished opens again only after the headset comes off and
  on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that
  fails retries every 10 s.
- Its status says why gaze mode can't work yet (checks.problem): not
  calibrated and opening, open, closed unfinished, or can't open and why.
- Input Settings shows that under the Gaze pointer switch, along with the
  gaze service not installed or not running and our tracker missing its
  frame grabber.
- ft-gazectl on notes a missing calibration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:50:52 -06:00
DeeJanuzandClaude Opus 5.5 ac142f6e4a ft-cutouts status: only the current run's tracker lines
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:26:59 -06:00
DeeJanuzandClaude Opus 5.5 b417534425 Hands: ft-cutouts, the hand cutouts without pinches and grips
hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient
user units with ft-hands' new --no-gestures: hands are published for
ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks
or drags and a closing hand doesn't raise the tracking rate. It needs a
build and ft-camd's capabilities, not hands/run.sh install. Its units
conflict with ft-handsctl's, so each stops the other, and they stop with
SteamVR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:22:52 -06:00
DeeJanuz c377133859 Merge experimental into shortcuts
# Conflicts:
#	input/input-relay.py
2026-10-01 17:08:15 -06:00
DeeJanuzandClaude Opus 5.5 675ef3f0d5 Relay: share a key combination's Meta release with frame-voice
A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release
from the desktop, and the relay skipped share_key for it too. frame-voice
saw Meta go down on @frametop_keys and never come up, so it held all
dictated text back, waiting for that release. Keys of grabbed keyboards
are now shared as pressed, before key_binding() decides what the desktop
gets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:07:38 -06:00
DeeJanuzandClaude Opus 5.5 fadc0f467c Shortcuts: Steam menu, commands, and modifier taps
Key combinations (and mouse and controller buttons) get two new actions:
- steam_menu: Open Steam menu / close dashboard (steam/ft-steam menu).
- command:CMD: run CMD with sh -c, as the relay's service, with layout/,
  float/ and steam/ on its PATH. Input Settings offers it for key
  combinations as Run a command...
Both work without pointer mode.

A modifier on its own is now a key combination too: a tap, pressed and
released with no other key, mouse button, or scroll in between. A bound
tap sends the desktop F24 before the release, so Plasma's launcher stays
shut. The defaults gain a Meta tap for the Steam menu; this replaces
META_DASHBOARD, which only worked in pointer mode.

input/test/keys-test.py runs the relay against fake devices with every
outgoing socket renamed, so it's safe next to the live relay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:06:30 -06:00
DeeJanuzandClaude Opus 5.5 2487351556 ft-steam: open Steam's menu through Steam's own UI
steam/ft-steam menu opens the SteamVR dashboard on Steam's menu, or closes
the dashboard if it's up, without pointer mode: it asks Steam's UI over its
debugging port to show its dashboard overlay (ShowVROverlay, what Steam
calls itself) and focus the Steam frame's left menu (MenuStore.OpenMainMenu).
ft-steam check says whether those calls still exist, and update-check.py
runs it, since a Steam client update can rename them.

The CDP client moves from display-settings/steam_settings.py to
steam/steamui.py, so both use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:06:30 -06:00
DeeJanuzandClaude Opus 5.5 4479fbfcc1 Input relay: typing with the pointer helper down no longer ends the relay
Typing on a pass-through keyboard tells the helper "typing". With the
helper not running (SteamVR off), that send raised ConnectionRefusedError
and the relay exited, dropping every grab until systemd restarted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 17:01:31 -06:00
DeeJanuzandClaude Opus 5.5 8b854fd424 Uninstall cleans up after itself; a shorter install command
- desktops.sh uninstall also removes Launch as Standalone's app copies
  and the title bar decoration, and Display Settings' uninstall the
  profiles' launcher entries; its install writes them back from the
  saved profiles (new: ft-layout launchers)
- the README says which settings an uninstall leaves
- the install command is now curl -fsSL https://deejanuz.github.io/
  frametop/get.sh | bash (GitHub Pages, from main)
- ft-gaze logs "action manifest ...: ok" instead of "error 0"

Found in a clean install test on the Frame (2026-10-01): uninstall,
then get.sh from the headset into a fresh clone; everything installed
and doctor.sh passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:42:37 -06:00
DeeJanuzandClaude Opus 5.5 7bcda94267 Defer hand tracking; the README leads with what Frametop does now
- install.sh no longer offers hand tracking (it's heavy on the CPU and
  needs more work); hands/ still builds and installs by hand
- the build container drops python3-opencv and python3-numpy, which
  only hand tracking's tools used: Fedora's OpenCV pulls in over a GB
- README: a new opening and feature list, the Use section by topic
  (screens, mouse, floating windows, profiles, gaze), and new known
  limitations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:20:39 -06:00
DeeJanuzandClaude Opus 5.5 c257000a23 A one-line installer, and gaze mode in install.sh
- get.sh: curl ... | bash asks for stable (main) or experimental,
  clones or updates ~/frametop, and runs install.sh; run it again to
  update or switch
- install.sh offers gaze mode (step 8, yes by default) and notes what
  to do if an SSH connection drops
- Input Settings' Gaze page says when the gaze service isn't installed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:04:49 -06:00
DeeJanuzandClaude Opus 5.5 0280e7441d Installers: ask for sudo in the terminal, and cope with SteamVR off
- frame_sudo (scripts/_env.sh) replaces three copies of sudo_run. On
  the Frame it asks in the terminal even when stdin isn't one, which
  install.sh's steps never had: saying yes to the Bluetooth fixes or
  hand tracking stopped the install with "no terminal for sudo".
  From a PC it asks through ssh -t when .env has no password.
- start_with_steamvr: the pointer, power, and gaze services restart
  when SteamVR runs (a re-install runs the new code) and are left to
  start with it when it doesn't, instead of failing the install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 15:04:49 -06:00
DeeJanuzandClaude Opus 5.5 12c42cd455 Bring the docs up to date with the code
- floating-windows.md and profiles.md describe what's built, with what
  isn't listed as such; the plans, phases, and branch notes are gone
- hands-migration.md is gone: the move is done; its open items are in
  hands/README.md's Known issues
- reference.md: Layout & profiles, every action, key combinations,
  floating windows, the gaze pointer, and hand tracking as they are
- design.md gets the KWin findings from floating-windows.md
- README, gaze/README, AGENTS, hazards, gaze-controllers, and the
  example config catch up with gaze, hands, and the stuck-key fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 14:47:53 -06:00
DeeJanuzandClaude Opus 5.5 0ac2b10cd7 Remove leftovers before a release
- gaze/tracker/lab/eyes_track.py: nothing used it
- Input Settings: the pointer role backend, whose page was removed
- ft-screens: no log line for every floating-window resize
- hands: ft-hands --help gives the palm-down default (1: off), the
  uninstall removes ft-handsctl's link, .frame-job is ignored
- Display Settings: "Save as profile…", not "Save current arrangement"
- two stale comments (ft-pointer's grabprobe, ft-gaze)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 14:47:53 -06:00
DeeJanuzandClaude Opus 5.5 3da17bc838 Open a profile's apps even when the screens can't be arranged
With no head pose (the headset off), ft-layout use stopped before
ft-floatd opened the apps. The screens now stay where they are, the
profile's hidden screens still hide, and the apps open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 14:47:53 -06:00
DeeJanuzandClaude Opus 5.5 20e8d6254a Merge gaze-calibration into experimental
The gaze checks and calibration in a headset panel (quick, five, full, headset
fit; shared-buffer drawing, click-to-capture), keyboard and mouse gaze clicks
(Meta+J/K, the mouse's buttons alike, mouse moves only while a button is held,
double right/Meta+K tilts a drag), one 55 degree learning limit with a quick
check past it, eye presence for "the headset went on", the gaze probe as a
development tool, and the relay releasing keys the desktop has down that no
keyboard holds.

Conflicts with the profiles: the relay keeps known_action/needs_pointer and
the gaze defaults (Meta+J/K and the float key); Input Settings keeps the
profile actions everywhere and the gaze actions in key combinations only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:55:04 -06:00
DeeJanuzandClaude Opus 5.5 2ae10a3b65 Input relay: release keys the desktop has down that no keyboard holds
After a gaze calibration, Meta stayed down in KWin: typing opened the overview,
and clicks on the desktop did other things. A key can be left down there when
its device vanishes with it held (release_held only let go of it on the relay's
own devices; the Z3's keyboard dropped out twice right then) or a release goes
astray. The relay now remembers which keys it told ft-screens went down, and
once a second releases any no device holds (EVIOCGKEY), and the key
combinations forget a Meta or modifier no device holds. A relay that starts
releases the modifiers on the desktop, for keys an earlier one left down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:49:51 -06:00
DeeJanuzandClaude Opus 5.5 da3e3d66a7 Gaze panel: shared buffers, no flicker; calibration dots wait for a click
The panel drew each picture with SetOverlayRaw, a new texture upload every
time: the full calibration's 1024x768 picture flickered on every change, and
in a live test the headset kept showing the last calibration picture after the
panel had drawn the fit check (2026-10-01). Now, as screens/keyboard.cpp does,
it writes into three linear DMA-BUFs SteamVR imported once (the size of the
biggest panel; texture bounds show the part in use) and switches between them.
A show makes the overlay visible once its first picture is in.

The full calibration's and the five-dot check's dots wait for a left click or
Meta+J while you look at the dot, in place of capturing any steady gaze (which
can be a look somewhere else); no capture ring, no 8 s skip, and the check
closes after 2 minutes without a click. The quick check still captures itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:27:30 -06:00
DeeJanuzandClaude Opus 5.5 45b1b06cdb Calibration all in the headset panel; the gaze probe is a development tool
Check headset fit now runs in the headset panel too ("fitcheck"): a card per
eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
seen) and the hints, from the probe's fitcheck.py, updated at most twice a
second; left click or Meta+J runs its guided check, right click or Meta+K closes
it. So Quick check, Calibrate, and Check headset fit all happen in one place.

The gaze probe moves to the Gaze page's overflow menu as "Gaze probe
(development)", and its app menu entry says it's a development tool. Texts that
sent users to it ("use Calibrate… with Own tracker") point at Calibrate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:14:33 -06:00
DeeJanuzandClaude Opus 5.5 964c250858 Gaze mode: a double right click or double Meta+K pans and tilts a drag
A drag begun by right during the left's held-back press (or Meta+K during
Meta+J's) now lasts while either button or key is held, so pressing the right
one again is free: it tilts, as a right press does during any mouse drag. Meta+K
during a keyboard drag (held still into one, or the second Meta+K) tilts while
held: the head turns the panel, and the mouse can too; let go and the head drags
again from there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:03:54 -06:00
DeeJanuzandClaude Opus 5.5 1f46616c75 Gaze mode: the mouse's buttons work like Meta+J and Meta+K
The right button's press is held back like the left's: hold it, the pointer
stops where you look, move onto the target, and the right click comes on the
release (held still, it's a real right press). Right during the left's held-back
press now starts a drag where the pointer is, as Meta+J then Meta+K does, in
place of a right click there; letting go of either drops it. So once you've
moved, the left alone only clicks, and the right starts a drag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:00:23 -06:00
DeeJanuzandClaude Opus 5.5 949f629338 Input Settings: the mouse movement explanation is a tooltip
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:57:22 -06:00
DeeJanuzandClaude Opus 5.5 a2a5a4c98e Gaze mode: the mouse only corrects, and left-then-right right-clicks
POINTER_GAZE_MOUSE_MOVE=held, the default (the Gaze page's Mouse movement
switch, with why it's on): while the gaze has the pointer, moving the mouse
does nothing; it moves the pointer only while a button is held, as a
correction. A bumped or drifting mouse can't pull the pointer off what you're
looking at, and every mouse move is a correction, so lessons aren't polluted by
mouse moves to somewhere else. With the gaze stale for a second, in a game, or
with the headset off, the mouse moves the pointer as usual. "free" is the old
behaviour.

Pressing the right button while the left one's press is held back right-clicks
where the pointer is instead (correct with the left, then right-click); both
releases are then nothing. Meta+J then Meta+K stays a drag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:56:33 -06:00
DeeJanuzandClaude Opus 5.5 8998fc1ddd Gaze learning limit: 55 degrees, past it a quick check; a solid panel
Drops the quick check's second step (3629681): the user didn't want it, and it
re-entered itself every tick, re-placing and redrawing the panel, which
flickered in the headset.

POINTER_GAZE_NUDGE_MAX is the one learning limit, now 55 degrees by default:
half of the 109 the Frame shows across. The helper measures the correction
itself (raw gaze to click), not the mouse's path; past the limit it isn't
learned and the helper sends "recheck" for the quick check, for the mouse and
keyboard alike. ft-gazed's own limits (8 and 25) follow the setting.

Test, our tracker's 409 clicks since its Sep 29 calibration: a one-dot check
set from any one of them puts the next 2 minutes' clicks within 15 degrees (99%
within 4.2) and the next 10 minutes' within 25, so it gets well under 55.

The panel's dot is still and its capture ring fills in quarters, so it's drawn
again only a few times per dot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:48:57 -06:00
DeeJanuzandClaude Opus 5.5 362968198b Quick check: put the pointer on the dot, and one limit for learning
After the quick check's capture, in gaze mode, the dot stays where it is in the
room (the panel's new "lock") and the gaze has the pointer again. You put the
pointer on the dot as you'd correct a click, with the mouse or Meta+J and the
head. That click clicks nothing: the helper sends it as "calverify", learned
whatever its size. Over POINTER_GAZE_NUDGE_MAX, the capture runs again, up to
3 times. A right click or Meta+K skips it.

POINTER_GAZE_NUDGE_MAX is now the one limit: 15 degrees by default (was 8),
and ft-gazed's own limits (8 for SteamVR's tracker, 25 for ours) follow it. A
keyboard correction past it isn't learned but opens a quick check ("recheck"),
in place of the 30 degree keyboard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:41:52 -06:00
DeeJanuzandClaude Opus 5.5 cbad7bc470 Keyboard corrections are learned up to 30 degrees
A Meta+J hold's correction is always meant, but it went through the mouse's
8 degree nudge limit. Live, our tracker was 12 degrees off and every correction
was dropped without a word. Debug output now says when one is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:30:49 -06:00
DeeJanuzandClaude Opus 5.5 580004bb00 Gaze checks: the headset going on is eyes coming back
Steam's eyetracking.txt writes "HMD on" every minute or so with nobody in the
headset, and SteamVR said it was worn for 12 hours straight, so the quick check
opened about 40 times an hour at an empty headset. It now opens when SteamVR's
tracker sees eyes for 3 s after none for 3 s, and closes when they're gone 2 s.

The log reader also drops repeated "HMD on" lines and sub-second offs, so
lessons stop counting as from an older wear every minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:21:21 -06:00
DeeJanuzandClaude Opus 5.5 5f885cbe65 Merge volume-swap into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:38:32 -06:00
DeeJanuzandClaude Opus 5.5 bfc044c1b0 Try every volume key when one swap fails
remap_volume stopped at the first EVIOCSKEYCODE_V2 that failed, so the
volume entries after it were never tried. On a keyboard where swapping
volume up failed, volume down stayed a real key that gamescope reads,
and one press with nothing focused aborts gamescope and the VR session.
Restoring had the same hole: a failed entry left the later stand-ins in
place after the relay exited.

Now every entry is tried and the first failure is raised at the end.
take_volume already marks the device remapped on that error, so the
relay routes the stand-ins that did swap and restores them on exit.

Checked against a fake keymap with a stubbed fcntl.ioctl: with volume
up's swap failing, volume down now swaps (it stayed real before), the
same for restore, and full swaps, restores and the no-keymap case come
out as before. Found by 0x1f6 in PR #8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:38:32 -06:00
DeeJanuzandClaude Opus 5.5 46e5c59cc0 Merge PR #8 fixes 1 and 2 into experimental
From 0x1f6's robustness PR: survive a malformed control datagram (adapted
to wrap experimental's textfield command too) and plan a layout when the
rows setting exceeds what the screens fill. The PR's third commit, the
volume-key swap rollback, is left out: rolling back a partial remap leaves
every real volume key exposed to gamescope instead of some, and on restore
it turns already-restored keys back into stand-ins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:34:31 -06:00
0x1f6andClaude Opus 5.5 0f4243c7f2 Plan a layout when the rows setting exceeds what the screens fill
ft-layout crashed - and so did arranging the screens - whenever the row
setting in the layout preset is above what the screens fill. plan()
builds a grid with cols = ceil(count / rows) columns; when that leaves
fewer full rows than the setting asked for, the extra rows are empty,
and the per-row height max() over an empty row raised

    ValueError: max() arg is an empty sequence
      at plan, line 300 (flat) and 312 (curved)

The smallest real case is 4 screens with 3 rows: cols = ceil(4/3) = 2,
the first two rows hold all 4 screens, the third row is empty. Both the
flat wall and the curved preset crashed, so apply, plan and the
auto-arrange at desktop start all failed until the row setting was
lowered. The Rows spinner in Frametop Display Settings (main.qml) allows
any row count up to the screen count, and clamping to the screen count
does not prevent this - 3 rows for 4 screens is within that range and
never fits a full grid - so the crash was reachable from the UI as
shipped.

Reproduced by calling plan() directly with 4 screens and 3 rows: both
kinds raised. Also verified the whole placement matrix (counts 1-10,
rows 1-4) places every screen after the fix.

The fix trims rows to the number of rows the screens actually fill,
rows = ceil(count / cols), after cols is computed. The row count is
used again for stacking (gap times rows - 1, the sum of row heights),
so the stacking matches the trimmed grid: no empty row is ever built,
and a rows setting that can't be honoured degrades to the tightest fit
instead of failing.

(cherry picked from commit f2bdbd97a0)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:34:22 -06:00
0x1f6andClaude Opus 5.5 9c2fccf0a0 Survive a malformed control datagram
One bad datagram on the control socket ended the whole relay. Its
dispatch in handle_control ran unguarded in the main loop, so any
exception propagated out of main() and the process exited. The simplest
trigger is "watch abc": float(words[1]) raises ValueError, and the relay
died with

    ValueError: could not convert string to float: 'abc'
      at handle_control, via the bare handle_control(now) call in the
      select loop

The control socket is an abstract socket bound to @frametop_relay.
Abstract sockets carry no permissions, so every local process can send
to it; nothing authenticates the sender. Dying from a datagram was bad
in three ways:

- Every grab is lost. Physical devices go back to gamescope and SteamVR,
  which read them themselves, so the mouse types into Steam and the
  desktop at once, and Meta+Shift shortcuts fire on both sides.
- The volume-key takeover is lost. gamescope aborts on a volume key when
  no window has keyboard focus (wlr_seat_keyboard_notify_enter asserts on
  a null focus surface), which ends the whole VR session - the exact
  failure the relay exists to prevent.
- systemd restarts the service, but Type=notify with READY only after
  the virtual devices exist makes the restart a visible hiccup, and a
  crash loop from repeated bad datagrams would flap SteamVR's input.

Reproduced by running the relay with stubbed uinput devices on a
non-Linux host and sending "watch abc" to the control socket: it exited
on the first datagram after answering "devices" correctly. After the
fix, the same exchange gets a log line ("bad control datagram ...") and
the relay keeps answering.

The fix wraps each datagram's dispatch in try/except inside
handle_control's loop, so one malformed message is logged and skipped
while the rest of the queue is still processed. Parsing of the common
helper commands (vrbtn, vrhello, gazeawake, keyboard) keeps its own
guards; the catch-all is only a backstop for anything the guards miss,
including float() on a non-numeric argument to "watch" and "vrcapture".

Adapted for experimental (from PR #8): the guard also wraps the textfield
command, which experimental added to this dispatch after the PR's base.

(cherry picked from commit ca7e58069b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:34:12 -06:00
DeeJanuz d54e907d37 Merge main into experimental (#6 is already here as 3aea571..4e5131d; history only) 2026-10-01 09:32:35 -06:00
DeeJanuzandClaude Opus 5.5 4c46d69b0c Check what Frametop needs from SteamOS after an update
On the Frame, a SteamOS update replaces SteamVR, KWin, and gamescope with the
rest of the OS image. scripts/update-check.py, run by doctor.sh and report.sh,
checks what Frametop uses from it: the OpenVR interface versions the installed
programs were built against, the vrcmd --overlays format, the eye tracker's
shared memory layout, host files, services, sockets, and the driver
registration. doctor.sh --mark-good records the package versions once things
work, and later runs say what changed and what to try by hand.

The session no longer stops when mesavars.sh or flatpak.sh is missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 08:51:09 -06:00
DeeJanuz 3c5bfd8d2f Merge float-tracker into experimental 2026-10-01 08:50:38 -06:00
DeeJanuzandClaude Opus 5.5 c3785bb46e Floating windows: keep KWin's placement memory out
KWin's PlacementTracker keeps each window's geometry, full screen and
maximized state per layout of the outputs, and puts windows back when a
layout it has seen comes back. A spare output resizes after its window,
so resizing a floating window back to an earlier size (or changing its
scale, or full screen) made the window and its output flip forever, and
floating or docking one window could move others onto or off a spare.

The KWin script now keeps where each window belongs, reports nothing
while KWin changes the outputs, and on screensChanged puts floating
windows back (and the screens' windows when only spares changed),
cancelling KWin's requests before the app sees them. A size asked for is
held for a second against late answers.

With that: a launched app and a profile get their remembered scale back,
scale steps keep the size in pixels, ft-floatd waits for the end of an
edge resize before resizing the output (KWin cancels the resize on any
output change), a window taken over after a restart keeps its app and
panel density, and `ft-float float ID` asks the script for the window's
current place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 08:50:38 -06:00
DeeJanuz 81b64302aa Merge profiles fixes into experimental 2026-09-30 22:37:50 -06:00
DeeJanuzandClaude Opus 5.5 a699540d95 Profiles: fixes from trying them on the live desktop
- KWin 6.2's scripts have no maximize mode: a window counts as maximized
  when it fills its output's maximize area.
- A late event for a window that just closed brought it back into
  ft-floatd's table, so a profile "found" it open, floated a window that no
  longer existed, and held a slot. The script doesn't report deleted
  windows, and ft-floatd ignores events for ids it has seen close.
- Docking a window that came from a screen that's hidden now puts it on the
  first screen that shows instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:37:50 -06:00
DeeJanuz 079390cde4 Merge screen-hide and profiles into experimental 2026-09-30 22:32:18 -06:00
DeeJanuzandClaude Opus 5.5 858dbe1562 Profiles: named layouts that open apps
A profile is a named layout plus the screens it hides and its apps' windows
(docs/profiles.md). ft-layout save captures them (ft-floatd's "windows",
after the KWin script reports every window as it is now); use opens them:
the screens move, open windows of each app go to their places (on a screen,
maximized or not, or floating), and missing apps start, once and then again
for each window still missing 3 s after the first. Nothing closes.

The desktop starts in FT_PROFILE or default_profile (ft-layout start, from
the session script). Each profile gets a launcher entry (Frametop: NAME, in
SteamVR's Launch a program list) that switches to it or starts the desktop
in it. The relay's profile:NAME action and Input Settings' "Open profile"
entries put one on a key, mouse button, or controller button. Display
Settings' Layout page becomes Layout & profiles: Save as profile, Open
profile, the profile's apps, and Start in profile. Plasma's own session
restore is off in the Frametop session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:32:10 -06:00
DeeJanuzandClaude Opus 5.5 9d91ecbcaa Hide screens one at a time
ft-screens: "conceal <screen|all>" hides a screen on its own, whatever the
visibility mode or the hotkey says, until "reveal"; "concealed" lists them.
(Not "hide N": older builds read anything starting with "hide" as the hotkey.)
ft-layout keeps it per screen ("hidden" in the layout), applies it when it
arranges the screens, and has hide/show N|all and hidden. Display Settings
gets a Shown switch per screen on the Visibility tab. ft-floatd floats a new
window that opens on a hidden screen, and puts a stray window on a screen
that shows. Profiles (next) use it to show only some screens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:25:35 -06:00
DeeJanuz 0348821384 Merge float-launch into experimental 2026-09-30 22:20:36 -06:00
DeeJanuzandClaude Opus 5.5 38843a4717 Launch apps floating, remember where they floated, and Launch as Standalone
ft-float launch APP / run COMMAND: ft-floatd starts the app and floats its
first window (matched by process or desktop file name for 30 s) where that
app last floated, or in front of you at the primary screen's density. Each
app's place (pose relative to the primary screen, size in pixels, scale) is
kept in ~/.config/frametop-float.json whenever one of its windows stops
floating; the scale isn't applied yet, since rescaling can loop (written up
in docs/floating-windows.md, Known problems).

Launch as Standalone (float/ft_apps.py): the session writes copies of the
apps' desktop files with that action and puts them first in XDG_DATA_DIRS,
so it shows in the Application Launcher's and the taskbar's right-click menus
in the Frametop desktop only. ft-floatd rewrites them when apps change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:20:36 -06:00
DeeJanuzandClaude Opus 5.5 c4c10e6d92 Gaze checks and calibration in a panel fixed to the headset
ft-gazepanel (gaze/panel) is a SteamVR overlay that stays put in your view and
shows dots at known head-relative directions; the gaze service runs it and
drives it (gaze/gazecheck.py):
- a one-dot quick check 3 s after the headset goes on, when our tracker asks
  for a click (reseat), at most every 2 minutes, and from Quick check on the
  Gaze page; five dots follow if the next 3 lessons are still over 2 degrees off
- the full calibration (the probe's three rounds, dark to bright) when gaze mode
  comes on without one, or from Calibrate; quitting it with still no calibration
  turns gaze mode off
Each dot takes the gaze once it has held still for 0.6 s; a left click or Meta+J
takes it at once, a right click or Meta+K closes the panel (the pointer helper
hides its dot and passes those presses on while "calpanel" lasts). Our tracker
gets clicks and its own calibration; SteamVR's gets lessons per eye, or a new
calibration.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:12:26 -06:00
DeeJanuz 1aa88641ec Merge float-titlebar into experimental 2026-09-30 22:11:29 -06:00
DeeJanuzandClaude Opus 5.5 00dfd61396 Title bar button: fixes from trying it in the live desktop
- The decoration's "bottom" property clashed with Item's (KWin refused it).
- No On All Desktops button with one virtual desktop, like Breeze.
- decoration/apply.sh finds the session's D-Bus through plasmashell (KWin's
  environment isn't readable), and installs each try under a new name: KWin
  keeps a decoration's QML by name until it restarts.
- ft-floatd starts its script with Scripting.start: after a reload, the new
  script gets the old one's id while the old one is still being deleted, so
  run() on /Scripting/Script<id> went to the old script and the new one never
  ran (a second ft-floatd start left floating windows without their script).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:11:29 -06:00
DeeJanuz 6ea70f32bc Merge branch 'experimental' into gaze-calibration
# Conflicts:
#	input-settings/ft_input_settings.py
#	input-settings/main.qml
#	input/input-relay.py
2026-09-30 22:07:04 -06:00
DeeJanuzandClaude Opus 5.5 dc3a1b8208 Keyboard clicks at the gaze: Meta+J and Meta+K
gaze_left and gaze_right (Meta+J and Meta+K by default) click where you look.
A tap clicks where the dot was at the press and tells the gaze tracker it was
right there. Held, the pointer stays put in your view, so turning your head
carries it onto the target; the release clicks there and the correction is a
lesson (judged by the net correction, not the head's path). Held still for
POINTER_GAZE_HOLD it's a real press that the head drags, and Meta+K while
Meta+J aims presses where the dot is now, to correct and then drag.

A Meta combination now also sends the desktop an F24 press and Meta's release
at once: Meta's release no longer opens Plasma's launcher, and the click isn't
Meta+click (KWin's window move and resize, which swallowed right clicks). The
desktop gets Meta back for the next key if it's still held.

Also fixes e16b758, which had taken the gaze actions off the key combination
list along with the controllers'; and gaze_quickcal (the gaze service's
one-dot check) joins the actions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:04:14 -06:00
DeeJanuzandClaude Opus 5.5 f32187824a A float button in every window's title bar
Frametop's own window decoration (decoration/), a QML decoration for KWin's
Aurorae engine drawn like Breeze, puts a float button left of Close. It's the
Keep Below button with its own glyph: the KWin script floats a window when
keep-below is set and docks it when it's cleared, and keeps the flag set on
every floating window, so the button shows "back to the desktop" there. The
session script installs the decoration for the Frametop desktop only;
decoration/apply.sh switches a running desktop to it or back to Breeze.

Not yet tried in a running KWin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:03:41 -06:00
DeeJanuz 86c89f90af Merge float-access into experimental 2026-09-30 21:55:47 -06:00
DeeJanuzandClaude Opus 5.5 1e83f96b0f Float key in the input relay, docking everything, and the plan for profiles
The input relay owns the float key now: float_toggle (Meta+Shift+F unless the
rules have their own key combinations) floats the window under the pointer, or
the active one over the wallpaper, and docks it if it floats. dock_all puts
every floating window back. Both are mappable to mouse and controller buttons,
and work without pointer mode. The KWin script no longer registers a shortcut,
and ft-floatd drops the old one. Key combinations move to Input Settings'
Keyboard page.

docs/floating-windows.md records the decisions from 2026-09-30 (the title bar
button, Launch as Standalone, profiles), and docs/profiles.md plans profiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:55:47 -06:00
DeeJanuzandClaude Opus 5.5 e16b7588e2 Gaze mode is a mouse feature: drop its controller parts
Steam reads the Frame controllers itself, outside SteamVR's bindings, and every
press and release it sees takes SteamVR out of laser mode, so controller clicks
at the gaze can't be done cleanly (docs/gaze-controllers.md, from the gaze-first
branch's tests).

- Gaze precision and Gaze drag take a mouse button or a key combination only;
  the controller source, its aim steering, and POINTER_PRECISION_GAIN,
  POINTER_PRECISION_DEADZONE and POINTER_GAZE_DRAG_GAIN are gone.
- The gaze actions (including Gaze pointer on/off) can't be mapped to controller
  buttons: the relay ignores them there and doesn't ask the helper for those
  buttons, and Input Settings no longer offers them.
- Last used wins in gaze mode too: picking up a controller hands it the laser,
  as docs/design.md already said.
- The gaze dot shows all the time (POINTER_GAZE_DOT=always, the default;
  moving brings back the old behaviour), with a switch on the Gaze page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:11:24 -06:00
Nikita KoptelovandClaude Opus 5.5 51b4e79318 Start Frametop's SteamVR clients only once SteamVR is up (#6)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:02:24 -06:00
95 changed files with 8077 additions and 4144 deletions

No files matched your search

+6 -2
View File
@@ -25,10 +25,14 @@ scripts/frame.sh --host '<cmd>' # runs on the SteamOS host
A Steam Frame is someone's personal headset, and they may be wearing it while you work. A Steam Frame is someone's personal headset, and they may be wearing it while you work.
- Don't kill or restart `gamescope`, `steam`, `vrserver`, `vrcompositor`, the gamescope session, or the Frametop desktop without asking. Each one ends or disrupts whatever is happening in VR. - Don't kill or restart `gamescope`, `steam`, `vrserver`, `vrcompositor`, the gamescope session, or the Frametop desktop without asking. Each one ends or disrupts whatever is happening in VR.
- Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Only the Bluetooth fixes need host `sudo`, and they ask. - Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Three installers need host `sudo`, and they ask for it: the Bluetooth fixes (`setup/bluetooth/install.sh`), hand tracking (`hands/run.sh install` and `caps`, which set ft-camd's file capabilities with `setcap`), and our own eye tracker's frame grabber (`gaze/tracker/install.sh`).
- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. - Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`).
- Never copy `.netrc`, SSH keys, or Steam config off the Frame or into this repo. - Never copy `.netrc`, SSH keys, or Steam config off the Frame or into this repo.
## SteamOS updates
A SteamOS update replaces SteamVR, KWin, and gamescope with the rest of the OS image. When a change starts depending on something from the image (a host file, an OpenVR interface outside the bundled header, an undocumented layout or output format, a SteamVR or KWin quirk), add a check for it to `scripts/update-check.py`, or a retest hint for its package there. [docs/design.md](docs/design.md) has the background.
## Names ## Names
User-facing names are "Frametop", "Frametop Display Settings", and "Frametop Input Settings". Programs and files use the `ft-` / `ft_` prefix (`ft-screens`, `ft-pointer`, `ft-layout`, the `ft_pointer` driver); config, units, and overlay keys use `frametop`. Program names must stay within 15 characters: Linux truncates process names there, and the scripts find programs with `pgrep -x` / `pkill -x`. User-facing names are "Frametop", "Frametop Display Settings", and "Frametop Input Settings". Programs and files use the `ft-` / `ft_` prefix (`ft-screens`, `ft-pointer`, `ft-layout`, the `ft_pointer` driver); config, units, and overlay keys use `frametop`. Program names must stay within 15 characters: Linux truncates process names there, and the scripts find programs with `pgrep -x` / `pkill -x`.
+89 -23
View File
@@ -1,32 +1,35 @@
# Frametop # Frametop
Frametop puts a multi-monitor KDE Plasma desktop into SteamVR on the Valve Steam Frame, and lets a Bluetooth mouse drive all of SteamVR. It installs and runs on the headset itself. Frametop is a desktop for the Steam Frame that runs on the headset itself, with no PC. Put several monitors around you, pull app windows out to float on their own, and point with a mouse, or with your eyes.
Each screen is its own monitor with its own resolution, so you can have an ultrawide in the middle and two portrait screens beside it, at whatever size and distance you like. The screens come back to your saved layout when the desktop starts. You can move, resize, curve, and roll them, pin one to your wrist, and put them all back with a shortcut. - **Screens that are real monitors.** Each KDE Plasma screen has its own resolution and shape: an ultrawide in front, portrait screens beside it. Move, resize, curve, and roll them, or pin one to your wrist or your head. They come back to your layout when the desktop starts.
- **Windows that float on their own.** Take any app window off the screens into a panel of its own with Meta+Shift+F, its title bar button, or Launch as Standalone. It stays part of the desktop, so drag and drop and the clipboard still work between floating windows and the screens.
- **Profiles.** Save where your screens are, which ones show, and which apps are open where. Switch to a profile from Display Settings, a key, or its launcher entry, or start the desktop in one.
- **One mouse for all of SteamVR.** A Bluetooth mouse drives a small dot anchored in the room. It works the screens, the dashboard, Steam, and overlays, and hands the laser back when you pick up a controller.
- **Look and click (experimental).** In gaze mode the pointer goes where you look. Meta+J and Meta+K, or the mouse buttons, click and fine-tune, and each correction teaches the tracker. Calibration runs in the headset.
- **Made for long sessions.** The displays turn off when the headset isn't used, even on a stand that makes it seem worn. It stays awake on the charger, and you can reach the desktop remotely over VNC.
The mouse shows up as a small dot anchored in the room. It works on the SteamVR dashboard, Steam, overlays, and the desktop, and it hands the laser back to your controllers when you pick one up. Two settings apps come with it: Frametop Display Settings for the screens, profiles, and power, and Frametop Input Settings for mice, keyboards, gaze, and button mappings. Optional fixes let Bluetooth LE mice and keyboards like the Swiftpoint Z3 reconnect after they sleep. Install it with one command: see [Install on the headset](#install-on-the-headset).
It comes with two settings apps, Frametop Display Settings for the screens and Frametop Input Settings for mice, keyboards, and button mappings, plus fixes that let Bluetooth LE mice and keyboards like the Swiftpoint Z3 reconnect after they sleep.
When you're not wearing the headset, Frametop can turn its displays off and keep it awake on the charger, so you can still reach it remotely. This works even on a stand or mount that makes the headset seem worn.
Frametop is an independent project, not made by or affiliated with Valve. Frametop is an independent project, not made by or affiliated with Valve.
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
## Install on the headset ## Install on the headset
You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space. You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
1. In the launcher, choose Launch a program → Desktop. 1. In the launcher, choose Launch a program → Desktop.
2. In the application menu, open System → Konsole. 2. In the application menu, open System → Konsole.
3. Clone the repo and run the installer: 3. Run:
``` ```
git clone https://github.com/DeeJanuz/frametop.git ~/frametop curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
cd ~/frametop
./install.sh
``` ```
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you two things along the way. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too. It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings.
@@ -40,11 +43,10 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
## Use ## Use
### Screens
| Do this | To get this | | Do this | To get this |
| --- | --- | | --- | --- |
| Move the mouse | The dot moves around you and snaps onto whatever panel it's over |
| Click, right-click, scroll | Acts on the panel under the dot |
| Pick up a controller | The controller gets its laser back; move the mouse to take over again |
| Point near the bottom of a screen | Its controls fade in: the bar, the curve and roll buttons, and the resize tab on the corner | | Point near the bottom of a screen | Its controls fade in: the bar, the curve and roll buttons, and the resize tab on the corner |
| Drag the bar under a screen | Moves the screen; scroll while dragging to push it away or pull it closer. With the mouse, hold right while dragging to tilt it | | Drag the bar under a screen | Moves the screen; scroll while dragging to push it away or pull it closer. With the mouse, hold right while dragging to tilt it |
| Drag the tab on a screen's bottom right corner | Resizes the screen | | Drag the tab on a screen's bottom right corner | Resizes the screen |
@@ -52,15 +54,58 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
| Drag the roll button sideways, or scroll on it | Rolls the screen; it snaps level near straight | | Drag the roll button sideways, or scroll on it | Rolls the screen; it snaps level near straight |
| While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off | | While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off |
| Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head | | Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
| Save current arrangement… (Frametop Display Settings, Layout) | Saves where the screens are, with their sizes and pins, under a name. Pick a saved layout under Arrangement and press Arrange now to switch to it |
| Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) | | Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist | | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
| Leave the headset on a stand | Its displays turn off once it has gone unused for the time set in Frametop Display Settings → Power, even if the stand covers its proximity sensor. Pick it up, or use any mouse, keyboard, or button, and they come back on | | Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
| Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
| Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard | | Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
### Mouse and controllers
| Do this | To get this |
| --- | --- |
| Move the mouse | The dot moves around you and snaps onto whatever panel it's over |
| Click, right-click, scroll | Acts on the panel under the dot |
| Pick up a controller | The controller gets its laser back; move the mouse to take over again |
You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard, Recenter pointer, or Head follow on/off on the Buttons page of Frametop Input Settings, and the Frame controllers' buttons on its Controllers page. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately. If a panel you only look at, such as a performance overlay that follows your view, keeps catching the dot, tick it (or its whole app) on the Ignored panels page, and the pointer passes through it. Head follow, which is experimental and off by default, makes the pointer come along when you turn your head: it stays put until your head turns past the leash angle, then glides back to its place in your view, and a leash of 0 keeps it fixed in your view. It's only lightly tested and not polished; tuning its settings, or improving how it feels, is open to anyone who wants to take it further. You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard, Recenter pointer, or Head follow on/off on the Buttons page of Frametop Input Settings, and the Frame controllers' buttons on its Controllers page. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately. If a panel you only look at, such as a performance overlay that follows your view, keeps catching the dot, tick it (or its whole app) on the Ignored panels page, and the pointer passes through it. Head follow, which is experimental and off by default, makes the pointer come along when you turn your head: it stays put until your head turns past the leash angle, then glides back to its place in your view, and a leash of 0 keeps it fixed in your view. It's only lightly tested and not polished; tuning its settings, or improving how it feels, is open to anyone who wants to take it further.
Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running. ### Floating windows
Any desktop window can float in VR as a panel of its own. It stays a window of the same desktop, so drag and drop and the clipboard work between floating windows and the screens, and it's still in the taskbar and Alt+Tab.
| Do this | To get this |
| --- | --- |
| Meta+Shift+F over a desktop window | Floats that window, or puts it back on its screen if it floats. It acts on the window under the pointer, or the active one if the pointer is over the wallpaper. Rebind it, or map it to a mouse or controller button, in Frametop Input Settings (Keyboard page, or Buttons and Controllers as Float window in VR) |
| Click the float button, left of Close in a window's title bar | The same. Float in VR is also in every window's menu (Alt+F3). Apps that draw their own title bar, like Chromium and Electron apps, don't have the button: use Meta+Shift+F |
| Right-click an app in the Application Launcher (or the taskbar) and pick Launch as Standalone | Starts the app with its window floating, where that app last floated, or in front of you the first time. From a terminal: `float/ft-float launch org.kde.dolphin`, or `float/ft-float run <command>` |
| Meta+scroll over a floating window | Scales it up or down |
### Profiles
A profile is a named setup: where the screens are, with their sizes and pins, which ones are hidden, and which apps are open and where their windows are, on a screen or floating.
| Do this | To get this |
| --- | --- |
| Save as profile… (Frametop Display Settings, Layout & profiles) | Saves the current setup under a name, or updates the profile you're in |
| Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes |
| Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings |
### Gaze mode (experimental)
In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
| Do this | To get this |
| --- | --- |
| Tap Meta+J, or Meta+K | A left or right click where you look |
| Hold Meta+J, turn your head onto what you meant, and let go | A click there. Held still for half a second, it becomes a press, and turning your head drags |
| Hold the left mouse button, move the mouse onto what you meant, and let go | A click there. The right button does the same for a right click. Held still, the left button drags |
| Double right click (or double Meta+K) while dragging a screen's bar | Pans and tilts the screen |
| Put the headset on | A quick check: look at the dot for a moment, and the pointer lines up again |
By default the mouse only corrects: while the gaze has the pointer, moving the mouse does nothing until you hold a button. Each correction before a click teaches the tracker where it was off. A correction past the learning limit (55 degrees by default, about half of what you can see) starts a quick check instead. Calibrate, on the Gaze page, runs a full calibration in a panel in front of you: look at each dot and click. Check headset fit shows how well the eye tracker sees your eyes. [gaze/README.md](gaze/README.md) has the details.
### Leave the headset on a stand and reach it remotely ### Leave the headset on a stand and reach it remotely
@@ -75,12 +120,15 @@ With the displays off, the headset keeps tracking and rendering, so it uses abou
This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 20260922, SteamVR 2.17.10). This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
- A SteamOS or SteamVR update can break parts of it until Frametop catches up. If something stops working after an update, please report it. - A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
- The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes. - The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
- During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead. - During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
- Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab). - Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
- Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back. - Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back.
- The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing. - The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing.
- Profiles reopen apps, not what the apps had open. Tabs, files, and folders are left to each app's own restore.
- Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels.
- Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings.
- On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md. - On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md.
- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. - Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart.
- Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power. - Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power.
@@ -93,14 +141,18 @@ In a terminal on the headset, run:
cd ~/frametop && scripts/report.sh cd ~/frametop && scripts/report.sh
``` ```
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
## Update ## Update
Run the same command again. It updates `~/frametop` to the latest of the version you have (or switches, if you pick the other one) and installs it:
``` ```
cd ~/frametop && git pull && ./install.sh curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
``` ```
Or by hand: `cd ~/frametop && git pull && ./install.sh`.
## Uninstall ## Uninstall
``` ```
@@ -111,24 +163,37 @@ power/run.sh uninstall
pointer/driver/install.sh uninstall # then restart SteamVR pointer/driver/install.sh uninstall # then restart SteamVR
input-settings/install.sh uninstall input-settings/install.sh uninstall
display-settings/install.sh uninstall display-settings/install.sh uninstall
remote/install.sh uninstall
setup/bluetooth/install.sh uninstall # if you installed the Bluetooth fixes setup/bluetooth/install.sh uninstall # if you installed the Bluetooth fixes
hands/run.sh uninstall # if you installed hand tracking by hand
gaze/run.sh uninstall # if you installed the gaze service
gaze/tracker/install.sh uninstall # if you installed our own eye tracker's frame grabber
gaze/probe/install.sh uninstall # if you installed the gaze probe
``` ```
Your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, and the gaze calibration in `~/.local/state/frametop/gaze`. So does the desktop's own Plasma setup, in `~/.config/frametop`. Delete them too for a clean slate.
## How it works ## How it works
A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). A power service (`power/`) turns the displays off while the headset isn't used. [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame. [docs/hazards.md](docs/hazards.md) lists known ways the input handling can go wrong. A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). A power service (`power/`) turns the displays off while the headset isn't used. [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame. [docs/hazards.md](docs/hazards.md) lists known ways the input handling can go wrong.
| Folder | What it is | | Folder | What it is |
| --- | --- | | --- | --- |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
| `install.sh` | The one-step installer. Safe to re-run. | | `install.sh` | The one-step installer. Safe to re-run. |
| `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. | | `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
| `screens/` | ft-screens, the compositor (wlroots and OpenVR). | | `screens/` | ft-screens, the compositor (wlroots and OpenVR). |
| `session/` | The desktop session script and its config example. | | `session/` | The desktop session script and its config example. |
| `layout/` | ft-layout: where the screens float, and their sizes. | | `layout/` | ft-layout: where the screens float, and their sizes. |
| `float/` | Floating windows: ft-floatd and the KWin script that float a desktop window in VR. |
| `decoration/` | The desktop's window decoration: Breeze's look plus the float button. |
| `input/` | The input relay (Bluetooth mice and keyboards, button maps). | | `input/` | The input relay (Bluetooth mice and keyboards, button maps). |
| `pointer/` | The 3D mouse: SteamVR driver, helper service, and a probe tool. | | `pointer/` | The 3D mouse: SteamVR driver, helper service, and a probe tool. |
| `power/` | ft-powerd: turns the displays off while the headset isn't used. | | `power/` | ft-powerd: turns the displays off while the headset isn't used. |
| `gaze/` | Gaze mode (experimental): the gaze service, its calibration panel, our own eye tracker, and the gaze probe. See [gaze/README.md](gaze/README.md). |
| `hands/` | Hand tracking (experimental, deferred: the installer doesn't offer it). See [hands/README.md](hands/README.md). |
| `display-settings/`, `input-settings/` | The two settings apps (Kirigami, Python). | | `display-settings/`, `input-settings/` | The two settings apps (Kirigami, Python). |
| `remote/` | Frametop Remote Access, the app that turns remote desktop over VNC on and off. |
| `setup/` | The build container and the Bluetooth fixes. See [setup/README.md](setup/README.md). | | `setup/` | The build container and the Bluetooth fixes. See [setup/README.md](setup/README.md). |
| `scripts/` | Helpers the installers use. They run commands locally on the Frame, or over SSH from a PC. | | `scripts/` | Helpers the installers use. They run commands locally on the Frame, or over SSH from a PC. |
@@ -146,16 +211,17 @@ The scripts also work from a Linux or WSL PC over SSH, which is easier for editi
IdentityFile ~/.ssh/<your-key> IdentityFile ~/.ssh/<your-key>
``` ```
3. The Bluetooth fixes need `sudo`, and there's no terminal on the Frame to type the password into, so put it in `.env` at the repo root. It's gitignored and never synced: 3. The Bluetooth fixes need `sudo` on the Frame. The installer asks for the password in your terminal (over `ssh -t`). To skip the question, or to install with no terminal, put it in `.env` at the repo root instead. It's gitignored and never synced:
``` ```
steamos_root_pwd="<password>" steamos_root_pwd="<password>"
``` ```
Then run `./install.sh` from the PC. Daily use: Then run `./install.sh` from the PC. If SteamVR isn't running on the Frame, the services that need it start with it later. Daily use:
``` ```
scripts/doctor.sh # is the Frame reachable and ready? scripts/doctor.sh # is the Frame reachable and ready?
scripts/doctor.sh --mark-good # and record the versions Frametop works with
scripts/sync.sh # copy the repo to ~/dev/frametop on the Frame scripts/sync.sh # copy the repo to ~/dev/frametop on the Frame
scripts/frame.sh '<cmd>' # run in the dev container, in the Frame's copy scripts/frame.sh '<cmd>' # run in the dev container, in the Frame's copy
scripts/frame.sh -C <dir> '<cmd>' # same, in a folder of the repo scripts/frame.sh -C <dir> '<cmd>' # same, in a folder of the repo
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Switch the running Frametop desktop's window decoration without restarting it: Frametop's
# own (this folder, with the float button) or back to Breeze. The session script does the
# same at every desktop start (session/frametop-session.sh), so this is for trying changes.
# decoration/apply.sh install this folder's copy and use it
# decoration/apply.sh --off back to Breeze (until the next desktop start)
set -euo pipefail
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
deco=kwin4_decoration_qml_frametop
cfg=$HOME/.config/frametop
kwinrc=$cfg/kwinrc
# The desktop's D-Bus: from its Plasma shell, which uses Frametop's config folder (KWin's own
# environment isn't readable: it runs with extra capabilities).
bus=
for pid in $(pgrep -x plasmashell); do
env=$( (tr '\0' '\n' < "/proc/$pid/environ") 2>/dev/null) || continue
grep -qx "XDG_CONFIG_HOME=$cfg" <<<"$env" || continue
bus=$(sed -n 's/^DBUS_SESSION_BUS_ADDRESS=//p' <<<"$env")
done
[ -n "$bus" ] || { echo "the Frametop desktop isn't running" >&2; exit 1; }
if [ "${1:-}" = --off ]; then
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library --delete
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete
else
# Under a new name each time: KWin keeps a decoration's QML (even a broken one) by name
# until it restarts. The next desktop start goes back to the plain name.
decos=${XDG_DATA_HOME:-$HOME/.local/share}/kwin/decorations
rm -rf "$decos/${deco}"_try*
deco=${deco}_try$(date +%s)
mkdir -p "$decos"
cp -r "$here" "$decos/$deco"
rm -f "$decos/$deco/apply.sh"
sed -i "s/\"Id\": \"[^\"]*\"/\"Id\": \"$deco\"/" "$decos/$deco/metadata.json"
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library org.kde.kwin.aurorae
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme "$deco"
fi
DBUS_SESSION_BUS_ADDRESS=$bus gdbus call --session -d org.kde.KWin -o /KWin -m org.kde.KWin.reconfigure >/dev/null
echo "decoration: $(kreadconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --default Breeze)"
+123
View File
@@ -0,0 +1,123 @@
/*
A title bar button drawn the way Breeze draws its own: a glyph in the title's colour,
a soft circle behind it on hover, and a red circle for Close.
The float button is KWin's Keep Below button with a glyph of its own: Frametop's KWin
script (float/frametop-float.js) floats a window when keep-below is set on it and docks
it when it's cleared, and keeps the flag set on every floating window, so "toggled" here
means "floating" and the glyph turns into "back to the desktop".
SPDX-License-Identifier: GPL-2.0-or-later
*/
import QtQuick
import QtQuick.Shapes
import org.kde.kwin.decoration
DecorationButton {
id: button
property real size: 20
property color fg: "white"
property color bg: "black"
readonly property bool isClose: buttonType === DecorationOptions.DecorationButtonClose
readonly property bool isFloat: buttonType === DecorationOptions.DecorationButtonKeepBelow
// The glyph, as polylines in an 18 x 18 box (Breeze's own sizes).
readonly property var glyph: {
switch (buttonType) {
case DecorationOptions.DecorationButtonClose:
return [[[5, 5], [13, 13]], [[13, 5], [5, 13]]];
case DecorationOptions.DecorationButtonMaximizeRestore:
return decoration.client.maximized ? [[[4.5, 9], [9, 4.5], [13.5, 9], [9, 13.5], [4.5, 9]]]
: [[[4, 11.5], [9, 6.5], [14, 11.5]]];
case DecorationOptions.DecorationButtonMinimize:
return [[[4, 7], [9, 12], [14, 7]]];
case DecorationOptions.DecorationButtonKeepAbove:
return [[[4, 9], [9, 4], [14, 9]], [[4, 14], [9, 9], [14, 14]]];
case DecorationOptions.DecorationButtonShade:
return [[[4, 5], [14, 5]], [[4, 13], [9, 8], [14, 13]]];
case DecorationOptions.DecorationButtonApplicationMenu:
return [[[3.5, 5], [14.5, 5]], [[3.5, 9], [14.5, 9]], [[3.5, 13], [14.5, 13]]];
case DecorationOptions.DecorationButtonKeepBelow:
// A box with an arrow leaving it (float in VR), or coming back into it (put it back).
return button.toggled
? [[[8, 3.5], [3.5, 3.5], [3.5, 14.5], [14.5, 14.5], [14.5, 10]], [[14.5, 3.5], [8.5, 9.5]],
[[8.5, 5], [8.5, 9.5], [13, 9.5]]]
: [[[8, 3.5], [3.5, 3.5], [3.5, 14.5], [14.5, 14.5], [14.5, 10]], [[8.5, 9.5], [14.5, 3.5]],
[[10, 3.5], [14.5, 3.5], [14.5, 8]]];
}
return [];
}
readonly property string tip: isFloat ? (toggled ? "Back to Desktop" : "Float in VR") : ""
width: size
height: size
Rectangle {
anchors.fill: parent
radius: width / 2
visible: button.hovered || button.pressed || (button.toggled && !button.isFloat)
color: button.isClose ? (button.pressed ? "#c0392b" : "#da4453")
: Qt.rgba(button.fg.r, button.fg.g, button.fg.b,
button.pressed ? 0.35 : button.hovered ? 0.2 : 0.12)
}
// On all desktops: a dot, filled while the window is on all of them. Help: a question mark.
Rectangle {
visible: button.buttonType === DecorationOptions.DecorationButtonOnAllDesktops
anchors.centerIn: parent
width: button.size * 0.3
height: width
radius: width / 2
color: button.toggled ? button.fg : "transparent"
border.width: Math.max(1, button.size / 18)
border.color: button.fg
}
Text {
visible: button.buttonType === DecorationOptions.DecorationButtonQuickHelp
anchors.centerIn: parent
text: "?"
color: button.fg
font.pixelSize: button.size * 0.65
font.bold: true
}
// Up to three strokes per glyph.
function stroke(i) {
const k = button.size / 18;
return i < glyph.length ? glyph[i].map(p => Qt.point(p[0] * k, p[1] * k)) : [];
}
readonly property color strokeColor: isClose && (hovered || pressed) ? "white" : fg
readonly property real strokeWidth: Math.max(1, size / 18 * 1.1)
// (An inline component can't see the ids around it, so everything comes in as properties.)
component Stroke: ShapePath {
property var points: []
fillColor: "transparent"
capStyle: ShapePath.RoundCap
joinStyle: ShapePath.RoundJoin
PathPolyline { path: points }
}
Shape {
anchors.fill: parent
preferredRendererType: Shape.CurveRenderer
Stroke { points: button.stroke(0); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
Stroke { points: button.stroke(1); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
Stroke { points: button.stroke(2); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
}
onHoveredChanged: {
if (!tip || typeof decoration.requestShowToolTip !== "function")
return;
if (hovered)
decoration.requestShowToolTip(tip);
else
decoration.requestHideToolTip();
}
Component.onCompleted: {
if (buttonType === DecorationOptions.DecorationButtonQuickHelp)
visible = Qt.binding(() => decoration.client.providesContextHelp);
if (buttonType === DecorationOptions.DecorationButtonApplicationMenu)
visible = Qt.binding(() => decoration.client.hasApplicationMenu);
// Like Breeze: no On All Desktops button with only one virtual desktop.
if (buttonType === DecorationOptions.DecorationButtonOnAllDesktops)
visible = Qt.binding(() => decorationSettings.onAllDesktopsAvailable);
}
}
+270
View File
@@ -0,0 +1,270 @@
/*
Frametop's window decoration: Breeze's flat title bar, drawn in QML for KWin's Aurorae
engine, plus a button left of Close that floats the window in VR (docs/floating-windows.md,
decision 25). Aurorae loads QML without compiling anything, so this keeps working across
SteamOS's KWin updates, which a C++ decoration wouldn't.
The float button is the Keep Below button (FtButton.qml): Frametop's KWin script floats a
window when keep-below is set and docks it when it's cleared. A Keep Below button in the
configured button order is left out, since the float button stands in for it.
SPDX-License-Identifier: GPL-2.0-or-later
*/
import QtQuick
import org.kde.kwin.decoration
Decoration {
id: root
alpha: false
DecorationOptions {
id: options
deco: decoration
}
TextMetrics {
id: metrics
font: options.titleFont
text: "Mj"
}
readonly property bool maximized: decoration.client.maximized
readonly property int buttonSize: Math.max(16, Math.round(metrics.height * 1.25))
readonly property int titleHeight: buttonSize + 8
readonly property int borderSize: decorationSettings.borderSize
readonly property int side: {
switch (borderSize) {
case DecorationOptions.BorderNone:
case DecorationOptions.BorderNoSides: return 0;
case DecorationOptions.BorderTiny: return 2;
case DecorationOptions.BorderLarge: return 6;
case DecorationOptions.BorderVeryLarge: return 8;
case DecorationOptions.BorderHuge: return 12;
case DecorationOptions.BorderVeryHuge: return 18;
case DecorationOptions.BorderOversized: return 27;
default: return 4;
}
}
readonly property int bottomBorder: borderSize === DecorationOptions.BorderNone ? 0
: borderSize === DecorationOptions.BorderNoSides ? 4 : side
readonly property color outline: Qt.tint(options.titleBarColor, Qt.rgba(0, 0, 0, 0.35))
function applyBorders() {
borders.left = side;
borders.right = side;
borders.bottom = bottomBorder;
borders.top = titleHeight;
maximizedBorders.top = titleHeight;
// Without visible side borders, keep a strip to grab for resizing.
extendedBorders.left = side ? 0 : 4;
extendedBorders.right = side ? 0 : 4;
extendedBorders.bottom = bottomBorder ? 0 : 4;
}
onTitleHeightChanged: applyBorders()
onSideChanged: applyBorders()
onBottomBorderChanged: applyBorders()
Component.onCompleted: applyBorders()
// The configured buttons, with the float button left of Close (or first on the right
// when there's no Close), and no Keep Below of their own.
function order(list, right) {
const out = [];
let placed = false;
for (let i = 0; i < list.length; ++i) {
const t = list[i];
if (t === DecorationOptions.DecorationButtonKeepBelow)
continue;
if (t === DecorationOptions.DecorationButtonClose && !placed) {
if (right) {
out.push(DecorationOptions.DecorationButtonKeepBelow, t);
} else {
out.push(t, DecorationOptions.DecorationButtonKeepBelow);
}
placed = true;
continue;
}
out.push(t);
}
return {buttons: out, placed: placed};
}
readonly property var leftOrder: order(options.titleButtonsLeft || [], false)
readonly property var rightOrder: {
const r = order(options.titleButtonsRight || [], true);
if (!r.placed && !leftOrder.placed)
r.buttons.unshift(DecorationOptions.DecorationButtonKeepBelow);
return r;
}
function componentFor(t) {
switch (t) {
case DecorationOptions.DecorationButtonMenu: return menuButton;
case DecorationOptions.DecorationButtonExplicitSpacer: return spacer;
case DecorationOptions.DecorationButtonClose: return closeButton;
case DecorationOptions.DecorationButtonMaximizeRestore: return maximizeButton;
case DecorationOptions.DecorationButtonMinimize: return minimizeButton;
case DecorationOptions.DecorationButtonKeepBelow: return floatButton;
case DecorationOptions.DecorationButtonKeepAbove: return keepAboveButton;
case DecorationOptions.DecorationButtonShade: return shadeButton;
case DecorationOptions.DecorationButtonOnAllDesktops: return allDesktopsButton;
case DecorationOptions.DecorationButtonQuickHelp: return helpButton;
case DecorationOptions.DecorationButtonApplicationMenu: return appMenuButton;
}
return null;
}
Rectangle {
anchors.fill: parent
color: options.titleBarColor
border.width: root.maximized ? 0 : 1
border.color: root.outline
}
Item {
id: titleBar
x: root.maximized ? 0 : Math.max(root.side, 1)
y: root.maximized ? 0 : 1
width: root.width - 2 * x
height: root.titleHeight - y
Row {
id: leftButtons
anchors.left: parent.left
anchors.leftMargin: 4
anchors.verticalCenter: parent.verticalCenter
spacing: 4
Repeater {
model: root.leftOrder.buttons
delegate: Loader {
required property var modelData
sourceComponent: root.componentFor(modelData)
}
}
}
Row {
id: rightButtons
anchors.right: parent.right
anchors.rightMargin: 4
anchors.verticalCenter: parent.verticalCenter
spacing: 4
layoutDirection: Qt.LeftToRight
Repeater {
model: root.rightOrder.buttons
delegate: Loader {
required property var modelData
sourceComponent: root.componentFor(modelData)
}
}
}
// Centred over the whole bar, like Breeze, but never under the buttons.
Text {
id: caption
readonly property real free: rightButtons.x - (leftButtons.x + leftButtons.width) - 16
width: Math.min(implicitWidth, free)
x: Math.max(leftButtons.x + leftButtons.width + 8,
Math.min((parent.width - width) / 2, rightButtons.x - 8 - width))
anchors.verticalCenter: parent.verticalCenter
text: decoration.client.caption
textFormat: Text.PlainText
font: options.titleFont
color: options.fontColor
elide: Text.ElideRight
renderType: Text.NativeRendering
}
Component.onCompleted: decoration.installTitleItem(titleBar)
}
Component {
id: menuButton
MenuButton {
width: root.buttonSize
height: root.buttonSize
}
}
Component {
id: spacer
Item {
width: root.buttonSize
height: root.buttonSize
}
}
Component {
id: closeButton
FtButton {
buttonType: DecorationOptions.DecorationButtonClose
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: maximizeButton
FtButton {
buttonType: DecorationOptions.DecorationButtonMaximizeRestore
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: minimizeButton
FtButton {
buttonType: DecorationOptions.DecorationButtonMinimize
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: floatButton
FtButton {
buttonType: DecorationOptions.DecorationButtonKeepBelow
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: keepAboveButton
FtButton {
buttonType: DecorationOptions.DecorationButtonKeepAbove
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: shadeButton
FtButton {
buttonType: DecorationOptions.DecorationButtonShade
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: allDesktopsButton
FtButton {
buttonType: DecorationOptions.DecorationButtonOnAllDesktops
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: helpButton
FtButton {
buttonType: DecorationOptions.DecorationButtonQuickHelp
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
Component {
id: appMenuButton
FtButton {
buttonType: DecorationOptions.DecorationButtonApplicationMenu
size: root.buttonSize
fg: options.fontColor
bg: options.titleBarColor
}
}
}
+14
View File
@@ -0,0 +1,14 @@
{
"KPackageStructure": "KWin/Decoration",
"KPlugin": {
"Authors": [
{
"Name": "Frametop"
}
],
"Description": "Breeze-style window decoration with a button that floats the window in VR (Frametop desktop)",
"Id": "kwin4_decoration_qml_frametop",
"License": "GPL",
"Name": "Frametop"
}
}
+6 -1
View File
@@ -40,7 +40,12 @@ $running && echo 'started' || { echo 'failed:'; tail -20 $log; exit 1; }" ;;
sed 's|@SESSION@|$session/frametop-session.sh|' $session/deckard-nested-desktop.desktop > ~/$override sed 's|@SESSION@|$session/frametop-session.sh|' $session/deckard-nested-desktop.desktop > ~/$override
[ -f ~/.config/frametop.conf ] || cp $session/frametop.conf.example ~/.config/frametop.conf [ -f ~/.config/frametop.conf ] || cp $session/frametop.conf.example ~/.config/frametop.conf
echo \"installed ~/$override\"; grep ^Exec= ~/$override; echo; cat ~/.config/frametop.conf" ;; echo \"installed ~/$override\"; grep ^Exec= ~/$override; echo; cat ~/.config/frametop.conf" ;;
uninstall) "$frame" --host "rm -f ~/$override && echo 'removed; the launcher uses the stock desktop again'" ;; uninstall)
# Also what the session puts in place at each start: Launch as Standalone's app copies and
# the title bar decoration (float/ft_apps.py, decoration/).
"$frame" --host "rm -f ~/$override
rm -rf ~/.local/share/frametop/apps ~/.local/share/kwin/decorations/kwin4_decoration_qml_frametop
rmdir ~/.local/share/frametop 2>/dev/null; echo 'removed; the launcher uses the stock desktop again'" ;;
screens) screens)
[[ ${2:-} =~ ^[1-9]$ ]] || { echo "usage: $0 screens N (1-9)" >&2; exit 2; } [[ ${2:-} =~ ^[1-9]$ ]] || { echo "usage: $0 screens N (1-9)" >&2; exit 2; }
"$frame" --host "set -e; f=~/.config/frametop.conf "$frame" --host "set -e; f=~/.config/frametop.conf
+73 -3
View File
@@ -409,6 +409,31 @@ class Backend(QObject):
def pins(self): def pins(self):
return self._pins return self._pins
@Property("QVariantList", notify=changed)
def screensShown(self):
"""For each screen, whether it shows (False: hidden on its own, ft-layout hide N)."""
layout = ft_layout.load_layout()
return [not ft_layout.screen_entry(layout, i).get("hidden") for i in range(ft_layout.screen_count(layout))]
@Slot(int, bool)
def setScreenShown(self, index, shown):
"""Hide screen `index` (0-based) on its own, whatever the visibility mode, or show it."""
layout = ft_layout.load_layout()
screens = layout.setdefault("screens", [])
while len(screens) <= index:
screens.append({})
if shown:
screens[index].pop("hidden", None)
else:
screens[index]["hidden"] = True
ft_layout.save_layout(layout)
self.changed.emit()
if self._running:
reply = self._ask_screens(f"{'reveal' if shown else 'conceal'} {index + 1}")
if not (reply and reply.startswith("ok")):
self.message.emit("Saved; the desktop applies it when it next starts "
"(its compositor is older than hiding screens one at a time)", False)
@Slot(str, str) @Slot(str, str)
def pin(self, which, where): def pin(self, which, where):
"""Pin screen `which` (1-based, or "all") to "left", "right", or "head" as it is """Pin screen `which` (1-based, or "all") to "left", "right", or "head" as it is
@@ -420,7 +445,7 @@ class Backend(QObject):
f"{reply or 'the desktop is not running'}", True) f"{reply or 'the desktop is not running'}", True)
elif which == "all" and where != "none": elif which == "all" and where != "none":
place = "on your head" if where == "head" else f"on your {where} wrist" place = "on your head" if where == "head" else f"on your {where} wrist"
self.message.emit(f"All screens ride {place} now. Save current arrangement (Layout) keeps it.", False) self.message.emit(f"All screens ride {place} now. Save as profile… (Layout & profiles) keeps it.", False)
self._check_running() self._check_running()
# --- power: ft-powerd and Steam's sleep setting --- # --- power: ft-powerd and Steam's sleep setting ---
@@ -559,16 +584,55 @@ class Backend(QObject):
def renameLayout(self, old, new): def renameLayout(self, old, new):
try: try:
self._edit_layout(lambda l: ft_layout.rename_named(l, old, new)) self._edit_layout(lambda l: ft_layout.rename_named(l, old, new))
except RuntimeError as e: ft_layout.write_launchers(ft_layout.load_layout())
except (RuntimeError, OSError) as e:
self.message.emit(str(e), True) self.message.emit(str(e), True)
@Slot(str) @Slot(str)
def deleteLayout(self, name): def deleteLayout(self, name):
try: try:
self._edit_layout(lambda l: ft_layout.delete_named(l, name)) self._edit_layout(lambda l: ft_layout.delete_named(l, name))
except RuntimeError as e: ft_layout.write_launchers(ft_layout.load_layout())
except (RuntimeError, OSError) as e:
self.message.emit(str(e), True) self.message.emit(str(e), True)
# --- profiles (docs/profiles.md): a named layout's apps and hidden screens ---
@Slot(str, result="QVariantList")
def profileWindows(self, name):
"""A profile's windows, as "app" and "where" for the list."""
out = []
for e in ft_layout.load_layout().get("profiles", {}).get(name, {}).get("windows", []):
app = e.get("app") or os.path.basename((e.get("cmd") or ["?"])[0])
app = app.rsplit(".", 1)[-1] if "." in app and not e.get("cmd") else app
where = "floating" if "float" in e else f"screen {e.get('screen', 1)}" + (", maximized" if e.get("maximized") else "")
out.append({"app": app, "where": where})
return out
@Slot(str, result="QVariantList")
def profileHidden(self, name):
return ft_layout.load_layout().get("profiles", {}).get(name, {}).get("hidden", [])
@Slot(str, int)
def removeProfileWindow(self, name, index):
def edit(layout):
windows = layout.get("profiles", {}).get(name, {}).get("windows", [])
if 0 <= index < len(windows):
windows.pop(index)
self._edit_layout(edit)
@Property(str, notify=changed)
def defaultProfile(self):
return ft_layout.load_layout().get("default_profile", "")
@Slot(str)
def setDefaultProfile(self, name):
def edit(layout):
if name:
layout["default_profile"] = name
else:
layout.pop("default_profile", None)
self._edit_layout(edit)
@Slot(str, "QVariant") @Slot(str, "QVariant")
def setPreset(self, key, value): def setPreset(self, key, value):
def edit(layout): def edit(layout):
@@ -582,6 +646,12 @@ class Backend(QObject):
@Slot() @Slot()
def arrange(self): def arrange(self):
"""Arrange the screens; in a profile, also open its apps (ft-layout use)."""
layout = ft_layout.load_layout()
name = layout.get("active")
if layout.get("mode") == "custom" and name in layout.get("layouts", {}):
self._run(f"Opening {name}", "use", name)
else:
self._run("Arranging the screens", "apply") self._run("Arranging the screens", "apply")
@Slot() @Slot()
+3 -1
View File
@@ -17,10 +17,12 @@ case ${1:-install} in
on_frame "chmod +x display-settings/ft-display-settings layout/ft-layout layout/ft_layout.py on_frame "chmod +x display-settings/ft-display-settings layout/ft-layout layout/ft_layout.py
mkdir -p ~/.config/frametop mkdir -p ~/.config/frametop
kwriteconfig6 --file ~/$shortcuts --group services --group ft-layout-reset.desktop --key _launch 'Meta+Shift+R' kwriteconfig6 --file ~/$shortcuts --group services --group ft-layout-reset.desktop --key _launch 'Meta+Shift+R'
kwriteconfig6 --file ~/$shortcuts --group services --group ft-screens-toggle.desktop --key _launch 'Meta+Shift+H'" kwriteconfig6 --file ~/$shortcuts --group services --group ft-screens-toggle.desktop --key _launch 'Meta+Shift+H'
layout/ft-layout launchers # each profile's entry (Frametop: NAME), if there are profiles"
echo "installed: Frametop Display Settings, Reset Screen Layout (Meta+Shift+R), Hide/Show Screens (Meta+Shift+H)" ;; echo "installed: Frametop Display Settings, Reset Screen Layout (Meta+Shift+R), Hide/Show Screens (Meta+Shift+H)" ;;
uninstall) uninstall)
on_frame "rm -f ~/$apps/ft-display-settings.desktop ~/$apps/ft-layout-reset.desktop ~/$apps/ft-screens-toggle.desktop on_frame "rm -f ~/$apps/ft-display-settings.desktop ~/$apps/ft-layout-reset.desktop ~/$apps/ft-screens-toggle.desktop
rm -f ~/$apps/frametop-profile-*.desktop # the profiles' entries (the profiles stay in ~/.config/frametop-layout.json)
[ -f ~/$shortcuts ] && for f in ft-layout-reset ft-screens-toggle; do kwriteconfig6 --file ~/$shortcuts --group services --group \$f.desktop --key _launch --delete; done [ -f ~/$shortcuts ] && for f in ft-layout-reset ft-screens-toggle; do kwriteconfig6 --file ~/$shortcuts --group services --group \$f.desktop --key _launch --delete; done
echo removed" ;; echo removed" ;;
*) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;; *) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;;
+90 -12
View File
@@ -411,8 +411,8 @@ Kirigami.ApplicationWindow {
text: spage.md text: spage.md
? "Each screen is a real monitor of its own: any resolution, portrait by choosing a tall one. Resolution, " ? "Each screen is a real monitor of its own: any resolution, portrait by choosing a tall one. Resolution, "
+ "width, and curve apply at once. In VR: move a screen by the bar underneath, curve it with the round " + "width, and curve apply at once. In VR: move a screen by the bar underneath, curve it with the round "
+ "button next to the bar, resize it by the tab on its bottom right corner; Save current arrangement on the " + "button next to the bar, resize it by the tab on its bottom right corner; Save as profile… on the "
+ "Layout page keeps all of it." + "Layout & profiles page keeps all of it."
: "gamescope draws every screen at the same resolution, at most 1920 × 1080 worth of pixels. Portrait turns " : "gamescope draws every screen at the same resolution, at most 1920 × 1080 worth of pixels. Portrait turns "
+ "a screen on its side. Rotation and scale apply at once; the rest when the desktop starts." + "a screen on its side. Rotation and scale apply at once; the rest when the desktop starts."
} }
@@ -424,7 +424,7 @@ Kirigami.ApplicationWindow {
id: layoutPage id: layoutPage
Kirigami.ScrollablePage { Kirigami.ScrollablePage {
id: lpage id: lpage
title: "Layout" title: "Layout & profiles"
property var layout: backend.layout property var layout: backend.layout
property var preset: layout.preset || {} property var preset: layout.preset || {}
property bool hasCustom: (layout.screens || []).some(s => s.pos !== undefined) property bool hasCustom: (layout.screens || []).some(s => s.pos !== undefined)
@@ -440,16 +440,17 @@ Kirigami.ApplicationWindow {
actions: [ actions: [
Kirigami.Action { Kirigami.Action {
text: "Arrange now" text: lpage.named ? "Open profile" : "Arrange now"
icon.name: "view-restore" icon.name: "view-restore"
tooltip: "Float the screens out of the dashboard and put them in this layout, around where you're facing" tooltip: lpage.named ? "Put the screens in this profile's places, around where you're facing, and open its apps (windows already open move; nothing closes)"
: "Float the screens out of the dashboard and put them in this layout, around where you're facing"
enabled: backend.desktopRunning && backend.busy === "" enabled: backend.desktopRunning && backend.busy === ""
onTriggered: backend.arrange() onTriggered: backend.arrange()
}, },
Kirigami.Action { Kirigami.Action {
text: "Save current arrangement…" text: "Save as profile…"
icon.name: "document-save" icon.name: "document-save"
tooltip: "Save where the screens are now (placed by hand) as a named layout, and use it" tooltip: "Save where the screens are now, which ones are hidden, and the open apps and where their windows are, under a name"
enabled: backend.desktopRunning && backend.busy === "" enabled: backend.desktopRunning && backend.busy === ""
onTriggered: nameDialog.openFor("save", lpage.named ? lpage.layout.active onTriggered: nameDialog.openFor("save", lpage.named ? lpage.layout.active
: "Layout " + (lpage.names.length + 1)) : "Layout " + (lpage.names.length + 1))
@@ -507,16 +508,61 @@ Kirigami.ApplicationWindow {
Controls.Label { Controls.Label {
visible: lpage.layout.mode === "custom" visible: lpage.layout.mode === "custom"
Kirigami.FormData.label: "" Kirigami.FormData.label: ""
text: lpage.named ? "Where the screens were when you saved it. Arrange now puts them there. " text: lpage.named ? "Where the screens were when you saved it, and the apps that were open. Open "
+ "Save current arrangement updates it or saves a new one." + "profile puts the screens there and opens the apps. Save as profile updates "
: lpage.hasCustom ? "Where the screens were when you saved. Save current arrangement " + "it or saves a new one."
: lpage.hasCustom ? "Where the screens were when you saved. Save as profile "
+ "names it. Pick a preset to edit." + "names it. Pick a preset to edit."
: "Nothing saved yet: place the screens by hand, then Save current arrangement." : "Nothing saved yet: place the screens by hand, open your apps, then Save as profile."
opacity: 0.7 opacity: 0.7
wrapMode: Text.Wrap wrapMode: Text.Wrap
Layout.maximumWidth: Kirigami.Units.gridUnit * 20 Layout.maximumWidth: Kirigami.Units.gridUnit * 20
} }
// The profile's apps (docs/profiles.md): each window and where it goes.
ColumnLayout {
id: profileApps
visible: lpage.named
Kirigami.FormData.label: "Apps:"
property var windows: lpage.named ? backend.profileWindows(lpage.layout.active) : []
property var hidden: lpage.named ? backend.profileHidden(lpage.layout.active) : []
Connections {
target: backend
function onChanged() {
profileApps.windows = lpage.named ? backend.profileWindows(lpage.layout.active) : []
profileApps.hidden = lpage.named ? backend.profileHidden(lpage.layout.active) : []
}
}
Controls.Label {
visible: profileApps.windows.length === 0
text: "None saved. Open the apps you want, place their windows, then Save as profile."
opacity: 0.7
wrapMode: Text.Wrap
Layout.maximumWidth: Kirigami.Units.gridUnit * 20
}
Repeater {
model: profileApps.windows
delegate: RowLayout {
required property var modelData
required property int index
Controls.Label { text: modelData.app + " (" + modelData.where + ")" }
Controls.ToolButton {
icon.name: "list-remove"
text: "Leave out"
display: Controls.AbstractButton.IconOnly
Controls.ToolTip.text: "Leave this window out of the profile"
Controls.ToolTip.visible: hovered
onClicked: backend.removeProfileWindow(lpage.layout.active, index)
}
}
}
Controls.Label {
visible: profileApps.hidden.length > 0
text: "Hides screen" + (profileApps.hidden.length > 1 ? "s " : " ") + profileApps.hidden.join(", ")
opacity: 0.7
}
}
Controls.SpinBox { Controls.SpinBox {
Kirigami.FormData.label: "Rows:" Kirigami.FormData.label: "Rows:"
visible: lpage.layout.mode !== "custom" visible: lpage.layout.mode !== "custom"
@@ -554,8 +600,19 @@ Kirigami.ApplicationWindow {
Kirigami.FormData.label: "When the desktop starts:" Kirigami.FormData.label: "When the desktop starts:"
text: "Float the screens and arrange them" text: "Float the screens and arrange them"
checked: lpage.layout.auto !== false checked: lpage.layout.auto !== false
enabled: backend.defaultProfile === ""
onToggled: backend.setAuto(checked) onToggled: backend.setAuto(checked)
} }
Controls.ComboBox {
Kirigami.FormData.label: "Start in profile:"
model: [{ text: "None", value: "" }].concat(lpage.names.map(n => ({ text: n, value: n })))
textRole: "text"
valueRole: "value"
currentIndex: Math.max(0, indexOfValue(backend.defaultProfile))
onActivated: backend.setDefaultProfile(currentValue)
Controls.ToolTip.text: "The desktop starts in this profile: its screens, and its apps open. Each profile also has its own entry in SteamVR's Launch a program list"
Controls.ToolTip.visible: hovered
}
} }
// Preview: from above (you at the bottom) and from the front. // Preview: from above (you at the bottom) and from the front.
@@ -802,6 +859,27 @@ Kirigami.ApplicationWindow {
} }
} }
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Screens shown" }
Repeater {
model: backend.screensShown
delegate: Controls.Switch {
required property var modelData
required property int index
Kirigami.FormData.label: "Screen " + (index + 1) + ":"
text: modelData ? "Shown" : "Hidden"
checked: modelData
onToggled: backend.setScreenShown(index, checked)
}
}
Controls.Label {
text: "A hidden screen stays hidden whatever the choices above say, and Meta+Shift+H doesn't bring it back. Windows on it stay there; new ones that would open on it float instead."
opacity: 0.7
font: Kirigami.Theme.smallFont
wrapMode: Text.Wrap
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Pinned screens" } Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Pinned screens" }
Repeater { Repeater {
@@ -871,7 +949,7 @@ Kirigami.ApplicationWindow {
+ "it.\n\nPin a screen to your head: choose On your head above. It rides on the headset where it " + "it.\n\nPin a screen to your head: choose On your head above. It rides on the headset where it "
+ "is now, like a HUD, and shows whenever the screens do. Grab its bar to move it; it stays on " + "is now, like a HUD, and shows whenever the screens do. Grab its bar to move it; it stays on "
+ "your head where you let go. Choosing a pin above keeps the screen where it is now, so place " + "your head where you let go. Choosing a pin above keeps the screen where it is now, so place "
+ "it first. Save current arrangement (Layout) keeps pins." + "it first. Save as profile… (Layout) keeps pins."
} }
} }
} }
+10 -101
View File
@@ -4,116 +4,25 @@ Steam, not systemd, puts the Frame to sleep: after "When Plugged In and Idle ->
(an hour by default) without input, even while it charges. That's a Steam client setting, (an hour by default) without input, even while it charges. That's a Steam client setting,
`system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam `system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam
on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. There (SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080
`settingsStore.clientSettings` has the current values, and `SteamClient.Settings.SetSetting` (steam/steamui.py). There `settingsStore.clientSettings` has the current values, and
takes a change as a serialized CMsgClientSettings protobuf, which is what Steam's own `SteamClient.Settings.SetSetting` takes a change as a serialized CMsgClientSettings protobuf,
Settings -> Power page sends. Standard library only (a minimal WebSocket client). which is what Steam's own Settings -> Power page sends.
""" """
import base64
import json
import os import os
import socket import sys
import struct
import urllib.request sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "steam"))
from steamui import SteamUnreachable, evaluate # noqa: E402,F401 (callers catch SteamUnreachable here)
CDP_PORT = 8080
# CMsgClientSettings field numbers (Steam's UI bundle maps the names to these). # CMsgClientSettings field numbers (Steam's UI bundle maps the names to these).
FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003} FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003}
class SteamUnreachable(Exception):
pass
class _WebSocket:
def __init__(self, url, timeout=5):
host_port, path = url[len("ws://"):].split("/", 1)
host, port = host_port.rsplit(":", 1)
self.sock = socket.create_connection((host, int(port)), timeout=timeout)
key = base64.b64encode(os.urandom(16)).decode()
self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
head = b""
while b"\r\n\r\n" not in head:
chunk = self.sock.recv(4096)
if not chunk:
raise SteamUnreachable("Steam closed the connection")
head += chunk
if b" 101 " not in head.split(b"\r\n", 1)[0]:
raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
self.buf = head.split(b"\r\n\r\n", 1)[1]
def send(self, text):
data, mask = text.encode(), os.urandom(4)
n = len(data)
if n < 126:
head = struct.pack(">BB", 0x81, 0x80 | n)
elif n < 65536:
head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
else:
head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
def _take(self, n):
while len(self.buf) < n:
chunk = self.sock.recv(65536)
if not chunk:
raise SteamUnreachable("Steam closed the connection")
self.buf += chunk
out, self.buf = self.buf[:n], self.buf[n:]
return out
def recv(self):
message = b""
while True:
b0, b1 = self._take(2)
n = b1 & 0x7F
if n == 126:
n = struct.unpack(">H", self._take(2))[0]
elif n == 127:
n = struct.unpack(">Q", self._take(8))[0]
message += self._take(n)
if b0 & 0x80:
return message.decode(errors="replace")
def close(self):
self.sock.close()
def _evaluate(expression):
"""Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
try:
with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
targets = json.load(r)
except OSError as e:
raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
if not url:
raise SteamUnreachable("Steam's UI isn't running")
try:
ws = _WebSocket(url)
try:
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
"params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
while True:
reply = json.loads(ws.recv())
if reply.get("id") == 1:
break
finally:
ws.close()
except OSError as e:
raise SteamUnreachable(str(e)) from e
result = reply.get("result", {})
if "exceptionDetails" in result:
details = result["exceptionDetails"]
raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
return result.get("result", {}).get("value")
def sleep_settings(): def sleep_settings():
"""{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input, """{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input,
plugged in and on battery (0 = never).""" plugged in and on battery (0 = never)."""
value = _evaluate("(() => { const c = settingsStore.clientSettings; " value = evaluate("(() => { const c = settingsStore.clientSettings; "
"return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()") "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()")
if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")): if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")):
raise SteamUnreachable("Steam's settings don't have the sleep timeouts") raise SteamUnreachable("Steam's settings don't have the sleep timeouts")
@@ -125,7 +34,7 @@ def set_sleep_setting(name, seconds):
field, seconds = FIELDS[name], int(seconds) field, seconds = FIELDS[name], int(seconds)
if seconds < 0: if seconds < 0:
raise ValueError("seconds must be 0 (never) or more") raise ValueError("seconds must be 0 (never) or more")
ok = _evaluate(f"""(async () => {{ ok = evaluate(f"""(async () => {{
const bytes = []; const bytes = [];
const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }}; const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }};
varint({field} * 8); varint({seconds}); varint({field} * 8); varint({seconds});
+34
View File
@@ -0,0 +1,34 @@
# Controller desktop click stability
Trigger presses reach KDE immediately, but controller motion within 8 logical
pixels of the press stays at that position until release. Releasing without a motion outside this
zone delivers the click at the original position, even if the hand moved during
release. Moving outside the zone begins a normal drag immediately; returning to
the zone does not turn it back into a click. There is no hold-duration timer.
This filters overlay pointer content events on desktop monitors only, and only
presses from hand controllers start it. The 3D mouse (whose laser comes from the
`ft_pointer` virtual controller), SteamVR UI, separate screen grab bars and
floating-app title-bar carrying are unaffected. Multi-button gestures keep their
existing behavior. A motion onto another desktop monitor starts a drag;
cross-monitor motion is not stabilized.
CLI (runtime preferences, reset to 8 on desktop restart):
```sh
input/ft-clickctl status
input/ft-clickctl threshold 8
input/ft-clickctl threshold 0 # disable without a restart
```
Thresholds are 0–64 logical pixels, normalized to each panel's KDE scale.
Status reports held state, suppressed motions, stabilized clicks and drags.
Changing the threshold while a controller button is held is refused.
This is a separate contribution from desktop mouse/controller ownership. Its
hardware validation must check small controls, intentional text selection,
long presses, cross-monitor dragging and simultaneous mouse use. The existing
renderer laser remains tracked; this change stabilizes desktop input rather
than smoothing the visual laser. Default threshold is a starting point to test.
Run `scripts/test-controller-click.sh` for the isolated gesture-state tests.
+49 -4
View File
@@ -52,7 +52,9 @@ A head pin is the same pin on the headset (device index 0): the screen's transfo
### Named layouts ### Named layouts
A named layout is the custom arrangement under a name: each screen's pose relative to your head, width, curve, and pin, but not its resolution or scale, which need a desktop restart or belong to KWin. Using one copies it into the custom arrangement, so everything that applies the layout (desktop start, Meta+Shift+R, Arrange now) works unchanged, and `active` remembers which name it came from. Saving without a name (`ft-layout capture`) clears `active`, because the screens have been placed by hand since. Layouts are kept per screen number, so one saved with a different screen count still applies: missing screens keep their last saved place or the preset's. Named layouts are now profiles, which also hold which screens are hidden and which apps to open, with where their windows go. See [profiles.md](profiles.md).
A profile's screen part is the custom arrangement under a name: each screen's pose relative to your head, width, curve, and pin, but not its resolution or scale, which need a desktop restart or belong to KWin. Using one copies it into the custom arrangement, so everything that applies the layout (desktop start, Meta+Shift+R, Arrange now) works unchanged, and `active` remembers which name it came from. Saving without a name (`ft-layout capture`) clears `active`, because the screens have been placed by hand since. Layouts are kept per screen number, so one saved with a different screen count still applies: missing screens keep their last saved place or the preset's.
### Visibility and VR games ### Visibility and VR games
@@ -60,6 +62,44 @@ A named layout is the custom arrangement under a name: each screen's pose relati
`IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting. `IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting.
## Floating windows
[floating-windows.md](floating-windows.md) describes the feature and its parts. Drag and drop and the clipboard only work between windows of one compositor, so a floating window stays a KWin window and gets a KWin output of its own: one of the spare outputs KWin opens after the screens, shown by ft-screens as a panel cropped to the window. What follows is how KWin 6.2.5 behaves underneath that, from its source (`src/backends/wayland/`) and from trying it on the Frametop desktop.
### KWin's nested outputs
- Disabling a nested output keeps its host window. `Output::applyChanges` only flips `enabled`, KWin stops rendering it (no more commits), and Plasma drops its desktop view. So ft-screens keeps the same toplevel, and its screen numbers stay put. Enabling the output again resumes on the same toplevel.
- Each output's host window is titled `KDE Wayland Compositor WL-<n>`, with `- Output disabled` appended while it's disabled (`WaylandOutput::updateWindowTitle`, on every `enabledChanged`). ft-screens reads the title to tell screens (`WL-0` to `WL-<SCREENS-1>`) from spares, and to see a spare turn on and off.
- A spare resized while it's disabled comes up at the new size on its first frame, so floating a window needn't blink. Outputs with gaps between them are accepted, so ft-floatd places spares apart from the screens and from each other, within Xwayland's 32767-pixel limit.
- KWin keeps a Wayland popup inside its parent's output (`XdgPopupWindow::updateRelativePlacement` uses the output's placement area), and X11 apps place their menus within the monitor. That's why a floating window's output has a margin around the window: menus and dropdowns open past the window's edges, into the margin.
- KWin makes a nested output the size it's configured to times its scale, rounded (at 1.5 it lays out 1067 × 667 on a 1600 × 1000 buffer), and gives the buffer a whole buffer scale (1.2 becomes 2). A buffer whose size isn't a multiple of that is a protocol error that disconnects KWin, so ft-floatd sizes spares in multiples of it. After a scale change, ft-floatd asks for the output's size again in the new scale's terms, or the next configure would make it the old size times the scale.
- **Virtual outputs don't work.** `createVirtualOutput` makes an output window but never adds it to the backend's `m_outputs`, so `findOutput()` returns null when the pointer enters it, and the next line dereferences it (`Q_ASSERT` is compiled out). A click on such a panel would crash KWin. This rules out virtual outputs (`stream_virtual_output`) for floating windows without a patched KWin.
### The pointer
- Pointer positions reach KWin only through motion events: the output's position in the layout plus the position on its window. When ft-screens stops sending motion, KWin's pointer stays put.
- KWin starts an interactive move on the press itself, before any motion. So ft-screens stops sending motion as soon as a press lands in a floating window's title bar (from the frame and client rectangles ft-floatd sends it), with no round trip, and carries the panel instead. KWin gets the release at the press point, and the window moves by nothing on its output.
- KWin's nested backend ignores the position in `wl_pointer.enter`, and wlroots drops a motion to the position it entered at, so the first click after crossing onto another panel landed where KWin's pointer had been. ft-screens enters one unit off.
### The KWin script
The KWin side is a script (`float/frametop-float.js`), not a C++ effect, because a script keeps working across KWin updates and an effect would have to match the host's exact KWin build. KWin scripts can call D-Bus but can't serve it, so ft-floatd's commands come back through a long poll: the script calls `NextCommand`, which answers when a command is ready, or empty after 20 seconds, under KWin's 25-second D-Bus timeout. A few things about KWin's script engine:
- `windowAdded` reports popups as windows of their own (`popupWindow` true, `transient` true) with their geometry.
- Setting `frameGeometry` applies asynchronously: the app has to answer the new size first.
- A script can't read a window's maximize mode, so the script counts a window as maximized when it fills its output's maximize area.
- `globalThis` isn't defined. `print` goes to the journal unless `QT_FORCE_STDERR_LOGGING=1`.
The title bar's float button is Frametop's own window decoration (`decoration/`), written in QML for KWin's Aurorae engine, which loads it without compiling. A C++ fork of Breeze would have to match SteamOS's exact KDecoration build. A decoration can only make the window requests KWin offers it, so the button toggles keep-below, which has no visible effect on a window alone on its own output, and the script treats keep-below as the floating flag.
### KWin's placement memory
KWin keeps each window's geometry, full screen, and maximized state for each layout of the outputs (its `PlacementTracker`, keyed by every enabled output's name and geometry). When the outputs come back to a layout it has seen, it puts every window back as it was in it. That's for plugging monitors in and out, and it does harm here. A spare output changes size after its window does, so what KWin keeps for a spare's size is the window's next size. Resizing a floating window back to an earlier size made the window and its output flip between two sizes for good (Dolphin went between 1187 and 1424 logical pixels wide, its output between 1687 and 1925). Full screen flipped the same way, and floating or docking one window could move others, even onto a spare or off one.
So the KWin script keeps where each window belongs: where ft-floatd put it, or where it went outside an output change. While KWin changes the outputs, it reports nothing to ft-floatd. Once KWin is done (`screensChanged` comes after its restore), it puts floating windows back, and the screens' windows too when only spares changed. KWin's resize request hasn't reached the app by then, so the app never sees it. A size asked for is held for a second, since an app can still answer an older request, and then the script takes the size the window has. If `screensChanged` doesn't come within 2 seconds, the script stops waiting for it.
KWin also ends an interactive move or resize whenever the outputs change. So during a resize by a floating window's edge, ft-floatd only crops the panel to the window, and resizes the output when the drag ends. The margin is the room to grow until then.
## The 3D mouse ## The 3D mouse
The mouse works like the pointer on the Apple Vision Pro: a small cursor floats in the room, lands on whatever panel it meets, and acts on it like a controller's laser. The mouse works like the pointer on the Apple Vision Pro: a small cursor floats in the room, lands on whatever panel it meets, and acts on it like a controller's laser.
@@ -91,7 +131,7 @@ A few overlays need special handling:
Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping. Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured. Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a right click or MetLine truncated
Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place. Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
@@ -135,7 +175,7 @@ Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens
The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused. The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused.
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`. Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
@@ -149,6 +189,12 @@ Movement is judged within 10-second windows. On the mount, the head pose jittere
Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values. Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values.
## SteamOS updates
On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.
`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's sample timestamp is still at the offset ft-gaze reads, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
## Approaches we dropped ## Approaches we dropped
- WayVR, an existing Wayland desktop for VR. It built and connected to SteamVR on the Frame, but nothing showed in the headset. It has no bindings for the Frame's controllers, and its KDE screen capture needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship. - WayVR, an existing Wayland desktop for VR. It built and connected to SteamVR on the Frame, but nothing showed in the headset. It has no bindings for the Frame's controllers, and its KDE screen capture needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship.
@@ -157,7 +203,6 @@ Staying awake while charging uses Steam's own setting rather than a logind sleep
## Open questions ## Open questions
- A head-locked screen, like a HUD.
- A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens. - A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens.
- Drawing KWin's cursor on the screens. - Drawing KWin's cursor on the screens.
- Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back. - Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back.
+93 -132
View File
@@ -1,76 +1,79 @@
# Floating windows (plan) # Floating windows
Status: design settled 2026-09-29 (see "Decisions"); being built on the `floating-windows` branch. Any desktop app can float in VR in a panel of its own, like SteamVR's floating windows, while it stays part of the Frametop desktop. Drag and drop, the clipboard, and focus keep working between floating windows and the screens.
Built so far (2026-09-29; the KWin side tested on the headless test desktop, `screens/test/headless.sh`; nothing yet tried in the headset): - **Float a window** with "Float in VR" in its window menu (Alt+F3), the float button left of Close in its title bar, or the float key (Meta+Shift+F by default). Start an app floating with "Launch as Standalone" in its right-click menu in the Application Launcher or the taskbar, with `ft-float launch` or `ft-float run`, or from a profile ([profiles.md](profiles.md)).
- **Put it back** with the dock button under its panel, the title bar button, the float key, or "Back to Desktop" in the window menu. It returns to the screen, position, and size it came from. The `dock_all` action docks every floating window.
- The catcher (a release off every panel still reaches KWin), and the pointer helper's "up" backstop. - **Move it** by its title bar or the bar under its panel, **resize it** by its edges or the corner tab, and **change its scale** with Meta+scroll over it. Each app's last floating place, size, and scale are remembered.
- `float/frametop-float.js` (the KWin script), `float/ft-floatd`, and `float/ft-float`. "Float in VR" is in the window menu under Extensions, and Meta+Shift+F toggles the active window. Floating, docking (back where it came from), closing, full screen, per-window scale (Meta+scroll), popups reported with their rectangles, windows of a floating app floating too, and the notification when every spare is in use.
- ft-screens: a panel per spare output (`frametop.float.N`) with the crop, density, popups and dialogs as small panels over it, title-bar carrying, the corner tab resizing the window, and dock and close buttons. `--spares`, and the commands `float`, `unfloat`, `pose`, `sub`, `minimized`, `carry`.
- The session adds `FLOAT_SLOTS` spares and starts ft-floatd from the desktop's autostart; ft-layout leaves the spares alone.
- The 3D mouse's drag lock crosses onto other Frametop panels (not while carrying one).
Not built yet: phase 2 (the ghost, tear-off by dragging, push-flush docking), phase 3 (launching floating, the Frametop Apps entry and picker, remembered placement), and phase 4. Frametop Apps (decision 10) needs a per-screen hide, which ft-screens doesn't have yet: its hide and show are for all screens.
The goal is to let any desktop app float in VR in a panel of its own, like SteamVR's floating windows, while it stays part of the Frametop desktop. That means drag and drop, the clipboard, and focus keep working between floating windows and the screens.
- There are two ways to get a floating window. Launch the app floating, or drag a desktop window by its title bar off a screen and let go in the air.
- There are two ways to put one back. Push it flush against a screen and let go, or press its "back to desktop" button.
- Files, text, and images drag between any two floating windows, and between floating windows and the screens. - Files, text, and images drag between any two floating windows, and between floating windows and the screens.
How KWin behaves underneath all this, and what the KWin script does about it, is in [design.md](design.md#floating-windows).
## Not built
These were decided (see the table) but aren't built:
- Tearing a window off a screen by dragging its title bar into the air, with a ghost of it on the laser (decision 3; see "Tearing a window off a screen").
- Docking by pushing a floating window flush against a screen, with the landing spot highlighted (decisions 4 and 17).
- New windows of a floating app placed where that app's windows went last time, or to the parent's right (decision 15).
- +/- scale buttons on the floating panel's bar (decision 18). Meta+scroll changes the scale.
- The glow at the edge of your view toward a floating window activated out of sight, and the setting that brings it in front of you instead (decision 19).
- A Floating windows section in Frametop Display Settings (decision 1). `FLOAT_SLOTS` and `FLOAT_MARGIN` are set in `~/.config/frametop.conf`.
- A drag proxy on the catcher, so a drag's icon shows while the laser is between panels.
- Keeping floating windows out of Show Desktop (Meta+D) (decision 9). Nothing handles it yet.
## Decisions ## Decisions
Settled with the user on 2026-09-29. The sections below follow them. The numbers are cited in the code, so they stay as they are. Struck-out text was replaced by a later decision.
| # | Question | Decision | | # | Question | Decision |
|---|---|---| |---|---|---|
| 1 | How many windows can float at once | 8 spare outputs by default, configurable (`FLOAT_SLOTS`, and Display Settings); a change needs a desktop restart | | 1 | How many windows can float at once | 8 spare outputs by default, configurable with `FLOAT_SLOTS` (at most 16); a change needs a desktop restart. A Display Settings control isn't built |
| 2 | Menus and dropdowns | Each floating output has a margin around the window. The panel shows only the window, and each open popup gets a small overlay of its own, cut from the same buffer | | 2 | Menus and dropdowns | Each floating output has a margin around the window. The panel shows only the window, and each open popup gets a small overlay of its own, cut from the same buffer |
| 3 | Tearing off | Drag the title bar past a screen's edge and let go in the air, with a small dead zone past the edge | | 3 | Tearing off | Not built. Drag the title bar past a screen's edge and let go in the air, with a small dead zone past the edge |
| 4 | Docking by dragging | Push the window flush against a screen (within about 10 cm), with the landing spot highlighted, and let go | | 4 | Docking by dragging | Not built. Push the window flush against a screen (within about 10 cm), with the landing spot highlighted, and let go |
| 5 | Visibility | Floating windows follow the same rules as the screens: the hide hotkey, the visibility modes, and the games rule | | 5 | Visibility | Floating windows follow the same rules as the screens: the hide hotkey, the visibility modes, and the games rule |
| 6 | Windows a floating app opens | They float too | | 6 | Windows a floating app opens | They float too |
| 7 | Launching floating from the headset | One "Frametop Apps" launcher entry with a picker | | 7 | Launching floating from the headset | ~~One "Frametop Apps" launcher entry with a picker~~ Replaced by 26 and profiles (27) |
| 8 | Build order | The catcher first, as a fix that stands on its own; `pointer-ignore` and `layouts-headpin` merged before phase 1; the hand cutouts stay out | | 8 | Build order | Not kept here: it only set the order of the work |
| 9 | Show Desktop (Meta+D) | Floating windows stay | | 9 | Show Desktop (Meta+D) | Floating windows stay. Not built |
| 10 | Frametop Apps and visibility | The entry starts the desktop with each screen hidden on its own (the existing per-screen hide), so only floating windows show. No special mode | | 10 | Frametop Apps and visibility | ~~The entry starts the desktop with each screen hidden on its own, so only floating windows show~~ Replaced: a profile can hide screens (27) |
| 11 | Window frame | KWin's title bar and border stay. Frametop's bar, close, and "back to desktop" are extras | | 11 | Window frame | KWin's title bar and border stay. Frametop's bar, close, and "back to desktop" are extras |
| 12 | Resizing | The window's own edges and Frametop's corner tab both change the size in pixels at the same density; the output follows | | 12 | Resizing | The window's own edges and Frametop's corner tab both change the size in pixels at the same density; the output follows |
| 13 | Margin | 300 px on each side, configurable | | 13 | Margin | 300 px on each side, configurable (`FLOAT_MARGIN`) |
| 14 | All spares in use | The window opens on the screens, with a notification | | 14 | All spares in use | The window stays on the screens, with a notification |
| 15 | Where a floating app's new windows go | Where that app's windows went last time; otherwise to the parent's right, curving around you | | 15 | Where a floating app's new windows go | Not built: where that app's windows went last time, otherwise to the parent's right, curving around you. For now, a new window that opens on a floating window's output floats a little in front of it |
| 16 | Where the code is written | A branch in the PC's clone of the repo | | 16 | Where the code is written | Not kept here: it was about the work, not about Frametop |
| 17 | Size when docked by dragging | The current floating size in pixels, shrunk to fit the screen | | 17 | Size when docked by dragging | Not built (see 4): the current floating size in pixels, shrunk to fit the screen |
| 18 | Bigger text | A scale for each window (KWin's output scale): Meta+scroll over the window, or +/- on its bar. Remembered for each app | | 18 | Bigger text | A scale for each window (KWin's output scale): Meta+scroll over the window. Remembered for each app. +/- buttons on its bar aren't built |
| 19 | Switching to a window you can't see | It's focused, and a glow at the edge of your view points to it. Moving it in front of you is a setting | | 19 | Switching to a window you can't see | It's focused. Not built: a glow at the edge of your view that points to it, and a setting that moves it in front of you |
| 20 | Full screen | The window fills its own panel. The margin drops to zero while it's full screen, and the panel keeps its size and place | | 20 | Full screen | The window fills its own panel. The margin drops to zero while it's full screen, and the panel keeps its size and place |
| 21 | Named layouts | They cover the screens only. Floating windows use the placement remembered for each app | | 21 | Named layouts | ~~They cover the screens only~~ Replaced by profiles (27). Outside a profile, floating windows use the placement remembered for each app |
| 22 | The float key | One toggle: it floats a window, or docks it if it already floats. The input relay owns it (`float_toggle`), Meta+Shift+F by default, rebindable in Frametop Input Settings and mappable to mouse and controller buttons. KWin has no shortcut of its own for it, so one press can't fire twice |
| 23 | Which window the key acts on | The window under the desktop's pointer; the active window if there's none there (the wallpaper, the taskbar) |
| 24 | Docking everything | A `dock_all` action, with no default binding |
| 25 | A button on every window | A float button left of Close in the title bar, from Frametop's own QML window decoration, made to look like Breeze. It shows a dock icon on floating windows. Apps that draw their own title bar (Chromium, Electron, GTK) use the key |
| 26 | Launching one app floating | "Launch as Standalone" in the right-click menu of every app in the Application Launcher and the taskbar, from copies of the apps' desktop files that only the Frametop desktop reads. It replaces the Frametop Apps entry (7, 10) |
| 27 | Profiles | Named layouts become profiles: the screens' places, which screens show, and the apps and their windows, floating or not. See `docs/profiles.md` |
Also assumed: floating windows get the wrist pin, the head pin, and pass-through (`pointer-ignore`) like screens. Every gesture works with the controllers as well as the 3D mouse. A window launched floating uses the primary screen's density. VNC shows only the primary screen, as now. Anything that restarts the live desktop waits for the user's OK. Also: floating windows get the wrist pin, the head pin, and pass-through (`pointer-ignore`) like screens. Every gesture works with the controllers as well as the 3D mouse. A window launched floating uses the primary screen's density. VNC shows only the primary screen.
## The approach: each floating window gets a KWin output of its own ## The approach: each floating window gets a KWin output of its own
Drag and drop and the clipboard only work between windows of the same compositor. A Wayland window can't move from one compositor to another. So a floating window has to stay a KWin window. Drag and drop and the clipboard only work between windows of the same compositor. A Wayland window can't move from one compositor to another. So a floating window has to stay a KWin window.
ft-screens already shows each KWin output as a panel. It sets the output's size with an `xdg_toplevel` configure, and KWin resizes the output to match. So a floating window can get an output of its own, sized to fit it, and ft-screens shows that output as a panel with its own controls. To KWin this is an ordinary desktop with more monitors. Dragging between two floating windows is the same as dragging between two monitors, which KWin already handles. ft-screens already moves the pointer between panels in the middle of a drag: `handle_vr_event` moves pointer focus to another KWin window even while a button is held. ft-screens already shows each KWin output as a panel. It sets the output's size with an `xdg_toplevel` configure, and KWin resizes the output to match. So a floating window gets an output of its own, sized to fit it, and ft-screens shows that output as a panel with its own controls. To KWin this is an ordinary desktop with more monitors. Dragging between two floating windows is the same as dragging between two monitors, which KWin already handles. ft-screens moves the pointer between panels in the middle of a drag: `handle_vr_event` moves pointer focus to another KWin window even while a button is held.
Alternatives we considered: Alternatives considered:
- **Run floating apps directly on ft-screens.** It's a wlroots compositor, so apps could connect to it and get a panel per window. But they would get no drag and drop or clipboard with desktop apps unless we wrote a bridge. Also, a window that's already on the desktop could never be torn off, because a Wayland client can't change compositors. Rejected. - **Run floating apps directly on ft-screens.** It's a wlroots compositor, so apps could connect to it and get a panel per window. But they would get no drag and drop or clipboard with desktop apps without a bridge. Also, a window that's already on the desktop could never float, because a Wayland client can't change compositors. Rejected.
- **One large hidden "canvas" output.** Every floating window would sit on one big output, and each panel would show a crop of it (`SetOverlayTextureBounds`). That needs only one extra output, with no copies. But an 8K canvas uses about 128 MB per buffer, with two or three buffers in KWin's swapchain. It would also have to repack windows whenever one resized, full screen would fill the whole canvas, and every window would share one scale. This is the fallback if per-window outputs don't work. - **One large hidden "canvas" output.** Every floating window would sit on one big output, and each panel would show a crop of it (`SetOverlayTextureBounds`). That needs only one extra output, with no copies. But an 8K canvas uses about 128 MB per buffer, with two or three buffers in KWin's swapchain. It would also have to repack windows whenever one resized, full screen would fill the whole canvas, and every window would share one scale. It was the fallback in case per-window outputs didn't work.
- **Screencast single windows** (`zkde_screencast` `stream_window`, over PipeWire). This adds copies and latency, and the window still needs a real place in KWin's layout to receive input. Rejected. - **Screencast single windows** (`zkde_screencast` `stream_window`, over PipeWire). This adds copies and latency, and the window still needs a real place in KWin's layout to receive input. Rejected.
- **SteamOS's own floating windows** (Launch a program from the dashboard). Those apps run in gamescope, outside KWin, so they can't drag and drop with the desktop. - **SteamOS's own floating windows** (Launch a program from the dashboard). Those apps run in gamescope, outside KWin, so they can't drag and drop with the desktop.
### Where the extra outputs come from: spare outputs ### Where the extra outputs come from: spare outputs
KWin's nested backend opens its outputs at start (`--output-count`). The session starts KWin with the screen count plus `FLOAT_SLOTS` outputs (default 8). Each spare is disabled until it's needed, with `kscreen-doctor` (or in the session's `kwinoutputconfig.json`, so it starts disabled). Floating a window enables a spare, and docking the window disables it again. `FLOAT_SLOTS` limits how many windows can float at once, and changing it means restarting the desktop. KWin's nested backend opens its outputs at start (`--output-count`). The session starts KWin with the screen count plus `FLOAT_SLOTS` outputs (default 8, at most 16). ft-floatd turns off the spares nothing floats on with `kscreen-doctor` once it starts. Floating a window enables a spare, and docking the window disables it again. `FLOAT_SLOTS` limits how many windows can float at once, and changing it means restarting the desktop. How KWin's nested backend treats disabled outputs, and why its virtual outputs can't be used instead, is in [design.md](design.md#floating-windows).
Checked in KWin 6.2.5's source (`src/backends/wayland/`, 2026-09-29):
- Disabling a nested output keeps its host window. `Output::applyChanges` only flips `enabled`, KWin stops rendering it, and Plasma drops its desktop view. So ft-screens keeps the same toplevel, and its screen numbers stay put.
- Each output's host window is titled `KDE Wayland Compositor WL-<n>`, with `- Output disabled` appended while it's disabled (`WaylandOutput::updateWindowTitle`, on every `enabledChanged`). ft-screens reads the title to tell screens (`WL-0` to `WL-<SCREENS-1>`) from spares, and to see a spare turn on and off.
- Pointer positions reach KWin only through motion events: the output's position in the layout plus the position on its window. When ft-screens stops sending motion, KWin's pointer stays put.
- **Virtual outputs don't work.** `createVirtualOutput` makes an output window but never adds it to the backend's `m_outputs`, so `findOutput()` returns null when the pointer enters it, and the next line dereferences it (`Q_ASSERT` is compiled out). A click on such a panel would crash KWin. This rules out the virtual-output fallback (`stream_virtual_output`) without a patched KWin.
ft-screens creates a `screen` for each toplevel in the order they appear, and indexes its settings by that order. Spares come after the screens, so they get indices `SCREENS` and up. Their panels are hidden while their output is disabled. ft-screens creates a `screen` for each toplevel in the order they appear, and indexes its settings by that order. Spares come after the screens, so they get indices `SCREENS` and up. Their panels are hidden while their output is disabled.
@@ -79,35 +82,38 @@ ft-screens creates a `screen` for each toplevel in the order they appear, and in
``` ```
KWin script "frametop-float" ft-floatd (host, Python) ft-screens KWin script "frametop-float" ft-floatd (host, Python) ft-screens
window events, moves, menus ── D-Bus ──▶ window ↔ output ↔ panel table ── @ft_screens ──▶ panels, controls, window events, moves, menus ── D-Bus ──▶ window ↔ output ↔ panel table ── @ft_screens ──▶ panels, controls,
runs commands ◀─ long poll ─ spare outputs (kscreen-doctor) ◀─ @frametop_float ─ lasers, ghost, catcher runs commands ◀─ long poll ─ spare outputs (kscreen-doctor) ◀─ @frametop_float ─ lasers, catcher
``` ```
- **KWin script `frametop-float`** (JavaScript). A script keeps working across KWin updates. A C++ effect would have to match the host's exact KWin build, and our build container is Fedora, not SteamOS. The script watches windows (`windowAdded`/`windowRemoved`, `interactiveMoveResizeStarted`/`Stepped`/`Finished`, `outputChanged`, `minimizedChanged`, `windowActivated`, `fullScreenChanged`). It runs commands: move a window to an output, set its geometry, put it on all virtual desktops, and restore it. It adds "Float in VR" to the window menu (`registerUserActionsMenu`) and registers a shortcut (`registerShortcut`, Meta+Shift+F). KWin scripts can call D-Bus but can't serve it, so commands come back through a long poll. The script calls ft-floatd's `NextCommand`, which answers when a command is ready, and then the script calls it again. The fallback is loading one-shot scripts through `org.kde.kwin.Scripting`, the way kdotool does. All of these API names are present in the host's KWin 6.2.5. - **KWin script `frametop-float`** (`float/frametop-float.js`). ft-floatd loads it into the desktop's KWin over D-Bus (`org.kde.kwin.Scripting`). A script keeps working across KWin updates. A C++ effect would have to match the host's exact KWin build, and the build container is Fedora, not SteamOS. The script watches windows (`windowAdded`/`windowRemoved`, `frameGeometryChanged`, `outputChanged`, `interactiveMoveResizeStarted`/`Finished`, `fullScreenChanged`, `maximizedChanged`, `minimizedChanged`, `keepBelowChanged`, `windowActivated`) and the outputs (`screensChanged`). It runs commands: move a window to an output, set its geometry, put it on all virtual desktops, and restore it. It adds "Float in VR" ("Back to Desktop" on a floating window) to the window menu (`registerUserActionsMenu`). It registers no shortcut: the float key belongs to the input relay. KWin scripts can call D-Bus but can't serve it, so commands come back through a long poll. The script calls ft-floatd's `NextCommand`, which answers when a command is ready, and then the script calls it again. It also keeps KWin's placement memory from moving windows (see design.md).
- **ft-floatd** (Python). The host has dbus-python and PyGObject. It owns `org.frametop.Float` on the session's private bus, and it keeps the table of which window is on which output and panel. It enables and disables spare outputs and sets their size, scale, and position with `kscreen-doctor`, as ft-layout does. It tells ft-screens where each floating window goes and tells the script which window goes where. It also remembers each app's placement and scale, keyed by desktop file name. - **ft-floatd** (`float/ft-floatd`, Python). The host has dbus-python and PyGObject. It runs inside the desktop's Plasma session, started from its autostart. It owns `org.frametop.Float` on the session's private bus, and it keeps the table of which window is on which output and panel. It enables and disables spare outputs and sets their scale and position with `kscreen-doctor`, and their size through ft-screens. It tells ft-screens where each floating window goes and tells the script which window goes where. It launches apps floating, opens profiles' apps, and remembers each app's placement and scale, keyed by desktop file name. Commands come in on `@frametop_float`, from `ft-float`, the input relay, and ft-screens.
- **ft-screens.** `Screen` becomes a panel with a kind: screen or floating window. Floating panels get the same bar, curve, roll, resize tab, and wrist and head pins, plus close, "back to desktop", and scale buttons. New parts are the tear-off ghost, the catcher, popup overlays, carrying a panel during a KWin move, and the dock target highlight. `MAX_SCREENS` goes from 8 to 16. Commands arrive on `@ft_screens`. Events go out to `@frametop_float` from an unbound socket, the same way ft-screens talks to the input relay. - **ft-screens.** A spare output's panel is a floating window's. Floating panels get the same bar, curve, roll, resize tab, and wrist and head pins as screens, plus dock and close buttons left of the bar. Other parts: the catcher, popup and dialog overlays, and carrying a panel during a KWin move. `MAX_SCREENS` (screens and spares together) is 24. Commands arrive on `@ft_screens`. Events go out to `@frametop_float` from an unbound socket, the same way ft-screens talks to the input relay.
- **Session script.** Adds `FLOAT_SLOTS` to the output count, starts ft-floatd, and enables the KWin script in the session's `kwinrc`. - **Session script.** Adds `FLOAT_SLOTS` to KWin's output count, starts ft-floatd from the desktop's autostart, installs Frametop's window decoration and chooses it in the session's `kwinrc`, and writes the Launch as Standalone copies of the apps' desktop files.
- **ft-layout.** Arranges only the screens' outputs. Today it arranges everything in `kscreen-doctor -j`, so it has to skip the spares (`WL-<SCREENS>` and up), enabled or not. - **ft-layout.** Arranges only the screens' outputs, and leaves the spares (`WL-<SCREENS>` and up) to ft-floatd, enabled or not.
- **ft-pointer.** Changes to the drag lock (see "Drag and drop between panels"). - **ft-pointer.** The drag lock crosses onto other Frametop panels (see "Drag and drop between panels"), and a left release also goes to ft-screens as a backstop for the catcher.
- **Frametop Display Settings.** Gets a Floating windows section: slots, margin, and "bring a window in front of you when it's activated". - **Input relay.** Owns the float key: `float_toggle` and `dock_all` send `float pointer` and `dock all` to ft-floatd.
Program names stay within 15 characters (`ft-floatd`). Overlay keys are `frametop.float.N` and `frametop.float.N.bar`, and so on. Program names stay within 15 characters (`ft-floatd`). Overlay keys are `frametop.float.N` and `frametop.float.N.bar`, and so on; a floating window's popups and dialogs are `frametop.float.N.sub.K`.
## A floating window ## A floating window
- **Output and margin.** Its output is the window's frame plus a margin on each side (`FLOAT_MARGIN`, default 300 px). KWin keeps a Wayland popup inside its parent's output (`XdgPopupWindow::updateRelativePlacement` uses the output's placement area), so the margin gives menus and dropdowns room past the window's edges. X11 apps place their own menus within the monitor, so the same applies. Enabled spares sit apart from the screens and from each other in KWin's layout, so nothing spills from one to the next. Memory: a 1600 × 1000 window with a 300 px margin is about 14 MB per buffer, 42 MB for three. - **Output and margin.** Its output is the window's frame plus a margin on each side (`FLOAT_MARGIN`, default 300 px). KWin keeps a Wayland popup inside its parent's output, so the margin gives menus and dropdowns room past the window's edges. X11 apps place their own menus within the monitor, so the same applies. Enabled spares sit apart from the screens and from each other in KWin's layout, so nothing spills from one to the next. Memory: a 1600 × 1000 window with a 300 px margin is about 14 MB per buffer, 42 MB for three.
- **What the panel shows.** Only the window's frame: ft-screens crops the output's buffer with `SetOverlayTextureBounds` and maps mouse positions through the crop. Each open popup gets a small overlay of its own, cut from the same buffer and placed a few millimetres in front of the window, so the main panel never changes size. KWin tells scripts about popups as windows of their own (`windowAdded` with `popupWindow`), so the script reports their rectangles. - **What the panel shows.** Only the window's frame: ft-screens crops the output's buffer with `SetOverlayTextureBounds` and maps mouse positions through the crop. Each open popup or dialog gets a small overlay of its own, cut from the same buffer and placed a few millimetres in front of the window, so the main panel never changes size. KWin tells scripts about popups as windows of their own (`windowAdded` with `popupWindow`), so the script reports their rectangles.
- **Size and scale.** The panel's width is the window's pixel width times the source screen's metres per pixel, so text stays the same size in VR. A window launched floating uses the primary screen's density. Each window also has a scale (KWin's output scale), changed with Meta+scroll over the window or +/- on its bar and remembered for each app. A bigger scale makes the content bigger at the same panel size. - **Where it appears.** Floated from a screen, the panel starts where the window was on that screen, 30 cm in front of it. Launched floating, it goes where that app last floated, or in front of you, 0.8 to 2 m away.
- **Size and scale.** The panel's width is the window's pixel width times the source screen's metres per pixel, so text stays the same size in VR. A window launched floating uses the primary screen's density. Each window also has a scale (KWin's output scale), changed with Meta+scroll over the window in steps of 10% and remembered for each app. A bigger scale makes the content bigger at the same panel size.
- **Window state.** An ordinary window, not maximized, placed inside its output with the margin around it, and set to show on all virtual desktops. It keeps its title bar and border. Apps that draw their own title bar (GTK, Chromium) keep theirs. - **Window state.** An ordinary window, not maximized, placed inside its output with the margin around it, and set to show on all virtual desktops. It keeps its title bar and border. Apps that draw their own title bar (GTK, Chromium) keep theirs.
- **Moving.** Press the title bar. KWin starts an interactive move and the script reports it. ft-screens then stops forwarding pointer motion to KWin, so KWin's pointer stays at the press point and the window moves by nothing. Meanwhile ft-screens carries the panel with the pressing device, the same way the bar does today: it follows rigidly, scroll pushes and pulls, and the 3D mouse's right-drag tilts. When the button comes up, KWin gets the release at the press point. The bar under the panel works too. - **Moving.** Press the title bar. KWin starts an interactive move on the press itself, before any motion, so ft-screens stops forwarding pointer motion to KWin as soon as a press lands in a floating window's title bar (from the frame and client rectangles ft-floatd sends it). KWin's pointer stays at the press point and the window moves by nothing. For apps that draw their own title bar, the script reports the move and ft-screens stops then (`carry`); ft-floatd puts back any few pixels the window slipped before that. Meanwhile ft-screens carries the panel with the pressing device, the same way the bar does: it follows rigidly, scroll pushes and pulls, and the 3D mouse's right-drag tilts. When the button comes up, KWin gets the release at the press point. The bar under the panel works too.
- **Resizing.** The window's own edges (inside the margin, so KWin's resize works as on the desktop) and Frametop's corner tab both change the window's size in pixels at the same density, so the app lays itself out again. ft-floatd resizes the output to keep the margin, and the panel grows or shrinks around the window's top-left corner. A screen's tab only scales the panel. Resizing is throttled to about 20 updates a second, with a minimum of 320 × 200, like screens. - **Resizing.** The window's own edges (inside the margin, so KWin's resize works as on the desktop) and Frametop's corner tab both change the window's size in pixels at the same density, so the app lays itself out again. ft-floatd resizes the output to keep the margin, and the panel grows or shrinks around the window's top-left corner. KWin ends a resize by the window's edge whenever an output changes, so during one the panel follows the window and the output follows only when the drag ends: the margin is the room to grow until then. A screen's tab only scales the panel. Resizing is throttled to about 20 updates a second, with a minimum of 320 × 200, like screens.
- **KWin's placement memory.** KWin puts windows back where they were for each layout of the outputs it has seen, which fights spare outputs that follow their windows' sizes. The KWin script undoes it ([design.md](design.md#kwins-placement-memory)).
- **Full screen.** The window fills its own panel: the margin drops to zero while it's full screen, and the output is the panel's size in pixels. The panel keeps its size and place. On leaving full screen, the margin comes back. - **Full screen.** The window fills its own panel: the margin drops to zero while it's full screen, and the output is the panel's size in pixels. The panel keeps its size and place. On leaving full screen, the margin comes back.
- **Buttons.** The close button closes the window. "Back to desktop" docks it where it came from. - **Buttons.** The close button closes the window. The dock button docks it where it came from.
- **Minimize.** Minimizing, from the title bar or the taskbar, hides the panel, and restoring it shows the panel again. Floating windows stay in the desktop's taskbar and in Alt+Tab. - **Minimize.** Minimizing, from the title bar or the taskbar, hides the panel, and restoring it shows the panel again. Floating windows stay in the desktop's taskbar and in Alt+Tab.
- **Activated out of view.** When a floating window is activated (taskbar, Alt+Tab, a notification) and it's more than about 60° from where you're looking, it's focused and a glow at the edge of your view points to it. A setting moves it in front of you instead. - **New windows.** Popups and dialogs of a floating window (`transientFor`) show as small overlays over it. Another window of a floating app that opens on its output floats too, a little in front of it. Any other window that opens on a floating window's output goes to the first screen that shows. When every spare is in use, the window stays on the screens and a notification says so.
- **New windows.** A dialog of a floating window (`transientFor`) floats in front of its parent. Other new windows of a floating app float too: where that app's windows went last time, otherwise to the parent's right at the same distance, curving around you, and to its left if that's taken. When every spare is in use, the window opens on the screen used last and a notification says so. - **On a hidden screen.** A new window that opens on a screen hidden on its own floats instead, where that app last floated or in front of you.
- **Show Desktop.** Meta+D leaves floating windows alone.
## Tearing a window off a screen ## Tearing a window off a screen (not built)
The design for decision 3:
1. Press a desktop window's title bar and drag it. KWin starts a move, and the script tells ft-floatd, which tells ft-screens: `move-start <output> <window> <rect>`. 1. Press a desktop window's title bar and drag it. KWin starts a move, and the script tells ft-floatd, which tells ft-screens: `move-start <output> <window> <rect>`.
2. While the button is held, the laser leaves every Frametop panel by more than a small dead zone (a few centimetres past the edge). Letting go inside the dead zone is an ordinary drop. 2. While the button is held, the laser leaves every Frametop panel by more than a small dead zone (a few centimetres past the edge). Letting go inside the dead zone is an ordinary drop.
@@ -117,93 +123,48 @@ Program names stay within 15 characters (`ft-floatd`). Overlay keys are `frameto
## Putting it back ## Putting it back
- **Button.** "Back to desktop" returns the window to the screen, position, and size it had before it was torn off (saved at tear-off). - **Button.** The dock button returns the window to the screen, position, and size it had before it floated. If that screen is hidden now, it goes onto the first screen that shows.
- **Dragging.** Carry the floating window, by its title bar or its bar, until the spot you're pointing at is on a screen. Then push it flush with the screen, within about 10 cm of its surface: scroll away with the mouse, or move the controller forward. The screen shows where the window will land, and letting go docks it there at its current size in pixels, shrunk to fit if the screen is smaller. A carried panel keeps its distance, so moving a floating window in front of a screen never docks it by accident. - **Dragging (not built).** Carry the floating window, by its title bar or its bar, until the spot you're pointing at is on a screen. Then push it flush with the screen, within about 10 cm of its surface: scroll away with the mouse, or move the controller forward. The screen shows where the window will land, and letting go docks it there at its current size in pixels, shrunk to fit if the screen is smaller. A carried panel keeps its distance, so moving a floating window in front of a screen never docks it by accident.
- Docking disables the output and removes the panel. - Docking disables the output and hides the panel.
## Getting at it: the float key and the title bar button
Decisions 22 to 25.
- **The float key.** The input relay owns it: the action `float_toggle`, bound to Meta+Shift+F unless the rules file says otherwise (a rules file with no `key_bindings` gets that default; one with its own list, even an empty one, doesn't). It can be rebound or removed in Frametop Input Settings, and mapped to a mouse button or a Frame controller button like any other action. The relay takes the combination before it reaches the desktop and sends `float pointer` to ft-floatd, which asks the script for the window under KWin's pointer (`workspace.cursorPos`, top of `workspace.stackingOrder`, popups and dialogs counting as their parent). With none there, the wallpaper or the taskbar, it's the active window. KWin's pointer is where the 3D mouse or a laser last was on a Frametop panel. A window that floats docks; any other floats. The KWin script has no shortcut of its own (ft-floatd removes one that an older script registered), so one press can't float a window and dock it again.
- **Docking everything.** `dock_all` (no default binding) sends `dock all`, which docks every floating window where it came from.
- **The title bar button.** Breeze can't take a button of its own, and a C++ fork of it would have to match SteamOS's exact KDecoration build (Plasma 6.3 replaces KDecoration2 with KDecoration3). So the Frametop desktop gets its own window decoration, written in QML for KWin's Aurorae engine, which loads it without compiling (`decoration/`, installed to `~/.local/share/kwin/decorations/kwin4_decoration_qml_frametop`, chosen in the session's `kwinrc` only, so Desktop Mode keeps Breeze; `decoration/apply.sh` switches the running desktop to it or back to Breeze). It's drawn to look like Breeze, with a float button left of Close. The button calls `requestToggleKeepBelow()`, the one window request a decoration can make that has no visible effect here, and the KWin script reads the change: keep-below set on a window on the screens floats it, cleared on a floating window docks it. The script keeps keep-below set on every floating window, however it was floated, so the button shows its dock icon there. A window alone on its own output loses nothing by being kept below (only the wallpaper is under it). If the window can't float (every spare is in use), the script clears the flag again. Keep Below Others in a window's menu does the same as the button.
- **Apps that draw their own title bar** (Chromium and Electron apps, GTK apps) never show KWin's decoration, so they don't get the button. They use the key, or the window menu (Alt+F3).
## Launching an app floating ## Launching an app floating
- **In the desktop.** Use "Float in VR" in any window's menu, or press Meta+Shift+F for the active window. - **In the desktop.** Use "Float in VR" in any window's menu, its title bar button, or the float key.
- **From a command.** `ft-float run <command>` and `ft-float launch <app.desktop>` start an app and float its first window. ft-floatd records the process it started, and the script matches new windows by PID, including child processes. Some single-instance apps (Firefox, D-Bus-activated apps) open the window from a process that was already running. Those are matched by desktop file name within a few seconds, or by `XDG_ACTIVATION_TOKEN` where the app honors it. - **From the menu.** Right-click an app in the Application Launcher, or in the taskbar (where it starts another window of that app), and pick "Launch as Standalone" (decision 26). The launcher has no way to add an entry to every app's menu, but its menu shows each app's own desktop actions. So the Frametop desktop reads copies of the apps' desktop files with one more action added (`float/ft_apps.py`). They're written to `~/.local/share/frametop/apps/applications` from every desktop file in `XDG_DATA_DIRS`: by the session script before Plasma starts, and by ft-floatd whenever an app is installed, changed, or removed. The session puts `~/.local/share/frametop/apps` first in `XDG_DATA_DIRS`. Plasma's app cache is keyed by those directories, so Desktop Mode never sees the copies. Desktop files in `~/.local/share/applications` come before every data dir, so an app you've customized there keeps your copy and has no Launch as Standalone. The action runs `ft-float launch <desktop file name>`.
- **From the headset without the desktop open.** One new launcher entry, "Frametop Apps". It starts the Frametop session with each screen hidden on its own (the per-screen hide that already exists), and opens an app picker as a floating window. Picking an app launches it floating. The hide hotkey and visibility modes still apply to everything, and a screen comes back with one click. Everything runs in one session, so dragging between a standalone app and a desktop app works. If the desktop is already running, the entry just opens the picker. - **From a command.** `ft-float run <command>` and `ft-float launch <app.desktop>` start an app and float its first window. ft-floatd records the process it started, and new windows are matched by PID, including child processes. Some single-instance apps (Firefox, D-Bus-activated apps) open the window from a process that was already running. Those are matched by desktop file name. Either way, the window has to show up within 30 seconds.
- **The picker.** Either KRunner, floated, or a small Kirigami app like the settings apps, with a grid of apps and their icons. - **From the headset with the desktop off.** A profile's launcher entry starts the desktop in that profile, and a profile can hide every screen and hold only floating apps (`docs/profiles.md`). There's no separate Frametop Apps entry or picker.
- **Remembered placement.** Each app's last floating pose, size, and scale, keyed by desktop file name. Named layouts don't include floating windows. - **Remembered placement.** Each app's last floating pose (relative to the primary screen's panel, so it moves with the screens' layout), size in pixels, and scale, keyed by desktop file name, in `~/.config/frametop-float.json`. It's kept whenever one of the app's windows stops floating. With nothing remembered, the window opens in front of you, at the primary screen's density, 0.8 to 2 m away. A profile's own placement wins when the profile opens the app.
## Drag and drop between panels ## Drag and drop between panels
KWin handles the protocols: Wayland, X11 through Xwayland, and the portal's file transfer. Frametop has to get the pointer right between panels. KWin handles the protocols: Wayland, X11 through Xwayland, and the portal's file transfer. Frametop has to get the pointer right between panels.
- **Crossing panels.** When the laser moves onto another panel mid-drag, ft-screens gives that panel's KWin window pointer focus. KWin puts its cursor at that output's position, and the drop target gets enter and motion events. Floating windows add nothing new here, but they make gaps between panels the normal case. - **Crossing panels.** When the laser moves onto another panel mid-drag, ft-screens gives that panel's KWin window pointer focus. KWin puts its cursor at that output's position, and the drop target gets enter and motion events. Floating windows add nothing new here, but they make gaps between panels the normal case.
- **Gaps (the catcher).** While the laser is between panels, none of our overlays get its events, and ft-screens clears pointer focus on `FT_LEAVE` even with a button held. If you let go in empty space, the release never reaches KWin, and the drag or move stays stuck until the next click. The fix: while a button is held on a Frametop panel and the laser leaves all of them, ft-screens puts an invisible catcher overlay on the laser (the tear-off ghost is the same thing with a picture on it). A release on the catcher releases in KWin wherever the pointer last was. Dropping in a gap cancels, just as dropping outside any window does. The pointer helper also tells ft-screens when the mouse's left button comes up, in case the catcher misses it. This also fixes window moves and drags that end off a panel today. - **Gaps (the catcher).** While the laser is between panels, none of Frametop's overlays get its events, and ft-screens clears pointer focus on `FT_LEAVE` even with a button held. A release in empty space would never reach KWin, and the drag or move would stay stuck until the next click. So while a button is held on a Frametop panel and the laser leaves all of them, ft-screens puts an invisible catcher overlay on the laser. A release on the catcher releases in KWin wherever the pointer last was. Dropping in a gap cancels, just as dropping outside any window does. The pointer helper also tells ft-screens when the mouse's left button comes up ("up"), in case the catcher misses it. This also covers window moves and drags on the screens that end off a panel.
- **The 3D mouse's drag lock.** While the button is held, the drag lock keeps the cursor at its distance and stops hit tests. So a drag onto a nearer panel passes behind it, and a farther panel works only if SteamVR's laser happens to reach it. The change: while the button is held, keep testing the other panels (not the one pressed on) and move onto a panel the ray meets. Off the edge of the pressed panel, keep that panel's plane, as now, so moves and resizes past the edge still work. - **The 3D mouse's drag lock.** While the button is held, the drag lock keeps the cursor at its distance and stops hit tests, so a drag onto a nearer panel would pass behind it. So while the button is held, the helper keeps testing the other Frametop panels (not the one pressed on, and not while carrying one) and moves onto a panel the ray meets. Off the edge of the pressed panel, it keeps that panel's plane, so moves and resizes past the edge still work.
- **Drag icon.** KWin 6 draws the drag icon as part of its scene, so it should show on the panel under the laser (to check). In a gap it stops at the source panel's edge. A later version can show a small drag proxy on the catcher. - **Drag icon.** KWin 6 draws the drag icon as part of its scene, on the output its pointer is on. In a gap it stays at the source panel's edge.
- **Flatpak apps.** Dropping files into a sandboxed app goes through the document portal, the same path that the session script's file-picker fix covers. Test it explicitly, for example Dolphin to Brave. - **Flatpak apps.** Dropping files into a sandboxed app goes through the document portal, the same path that the session script's file-picker fix covers ([design.md](design.md#the-desktop-session)).
## Things that must keep working ## Things that must keep working
- **Typing follows the last click.** A click on a floating panel counts as a click on the desktop, since the window is a KWin window. - **Typing follows the last click.** A click on a floating panel counts as a click on the desktop, since the window is a KWin window.
- **Visibility.** Floating windows follow the screens' rules: the hide hotkey, the visibility modes, and hiding during a VR game unless the dashboard is open. Controllers' lasers are off in games. - **Visibility.** Floating windows follow the screens' rules: the hide hotkey, the visibility modes, and hiding during a VR game unless the dashboard is open. Controllers' lasers are off in games.
- **Headset standby.** Nothing new may poll SteamVR with new clients, so no new `vrcmd` loops. - **Headset standby.** Nothing new may poll SteamVR with new clients, so no new `vrcmd` loops.
- **The pointer helper's overlay list.** The helper learns about overlays by running `vrcmd --overlays` in the background, so a new floating panel appears in its next listing. Check the delay after a tear-off. If it's too long, ft-screens can send the helper new overlay keys directly. - **The pointer helper's overlay list.** The helper learns about overlays by running `vrcmd --overlays` in the background, so a new floating panel appears in its next listing.
- **Plasma.** An enabled floating output gets a desktop view (wallpaper) under its window, hidden by the crop. Plasma doesn't add panels to new outputs by default. A floating output must never become primary. With spare outputs, the output count stays the same, which avoids the lost-taskbar problem in design.md's open questions. - **Plasma.** An enabled floating output gets a desktop view (wallpaper) under its window, hidden by the crop. Plasma doesn't add panels to new outputs by default. A floating output must never become primary. With spare outputs, the output count stays the same, which avoids the lost-taskbar problem in design.md's open questions.
- **Restarting the desktop** closes every window, floating ones included. Each app's placement is remembered, so an app launched floating again comes back where it was. - **Restarting the desktop** closes every window, floating ones included. Each app's placement is remembered, so an app launched floating again comes back where it was.
## Plan
### Step 1: the catcher and the branches
- The catcher (see "Gaps"), as a fix that stands on its own, so it can go to `main` by itself.
- Merge `pointer-ignore` and `layouts-headpin`.
### Phase 0: find out
Each item has a pass condition. Items that need a desktop restart with extra outputs wait until the headset is free, or run in the headless test mode from 0.1.
- **0.1 Headless test mode.** `ft-screens --no-vr` runs the compositor without SteamVR. It logs toplevels, titles, and sizes, and answers commands on a separate control socket name. This lets KWin and output experiments run without the headset, and without touching the running desktop.
- **0.2 Outputs.** Start KWin with spare outputs, then disable and re-enable one with `kscreen-doctor`. Pass: the toplevel stays and its title changes (as the source says), a disabled output costs no frames, resizing a spare through configure works, and outputs with gaps between them are accepted. Also: an output larger than its window with the window placed inside, and a popup placed in the margin.
- **0.3 KWin script API on 6.2.5.** Move signals fire for moves from both KWin's title bars and apps' own. `sendClientToScreen` and `frameGeometry` work on another output, the `callDBus` long poll works, and `registerUserActionsMenu` works. Popups show up in `windowAdded` with their geometry. The observers can load into the running desktop through `org.kde.kwin.Scripting` and move nothing, so this is safe while the headset is in use.
- **0.4 Frozen-pointer move.** Pass: a KWin move with no pointer movement doesn't shift the window.
- **0.5 SteamVR's laser.** Find out which overlay gets MouseMove and ButtonUp when a held laser moves from overlay A to overlay B, and when it's released over nothing, for both a controller and the 3D mouse. Pass: an interactive overlay placed on the laser reliably catches the release.
- **0.6 Texture bounds.** Pass: `SetOverlayTextureBounds` crops a DMA-BUF (`SharedTextureHandle`) overlay correctly, mouse positions map to the cropped area, and two overlays can show different crops of one buffer.
#### Results (2026-09-29, headless, KWin 6.2.5)
`screens/test/headless.sh` runs these: ft-screens `--no-vr` with a bare nested KWin next to the running desktop, with an `input` command that feeds pointer events as if from a panel, KWin scripts loaded over D-Bus, and screenshots through ScreenShot2.
- **0.1 passes.** `--no-vr`, `--control`, `toplevels`, and `input` are in ft-screens.
- **0.2 passes.** Disabling a spare with `kscreen-doctor` keeps its toplevel, its title gets `- Output disabled`, and it stops committing; enabling it resumes on the same toplevel. A spare resized (`size`) while disabled comes up at the new size on its first frame, so a tear-off needn't blink. Live resizing works, output scale works (1.5: KWin lays out 1067 × 667 on a 1600 × 1000 buffer), and outputs with gaps between them (x = 5000, 8000, 10000) are accepted. A window placed inside a 1600 × 1200 output with a 300 px margin opens a context menu past its bottom and right edges, into the margin.
- **0.3 mostly passes.** `workspace.screens`, `sendClientToScreen`, `windowList`, `frameGeometry` (set; it applies asynchronously), `callDBus`, `registerShortcut`, `registerUserActionsMenu`, and `readConfig` exist. `windowAdded` reports popups (`popupWindow` true, `transient` true) with their geometry. Move signals fire for KWin's title bars (`interactiveMoveResizeStarted` with `move` true, `Stepped` with the geometry, `Finished`). Not yet checked: apps' own title bars, the `callDBus` long poll, and the window menu entry. `globalThis` isn't defined in KWin's script engine. `print` goes to the journal unless `QT_FORCE_STDERR_LOGGING=1`.
- **0.4: KWin starts the move on the press itself,** before any motion. So ft-screens freezes pointer motion as soon as a press lands in a floating window's title bar band (from the frame and client rectangles ft-floatd sends it), with no round trip. For apps that draw their own title bars, the script reports the move and ft-screens freezes then; the script puts back any few pixels the window slipped before that.
- **Found and fixed:** KWin's nested backend ignores the position in `wl_pointer.enter`, and wlroots drops a motion to the position it entered at, so the first click after crossing onto another screen landed where KWin's pointer had been. ft-screens now enters one unit off.
- **0.5 and 0.6** need SteamVR and the headset.
### Phase 1: float a window from its menu
Build the KWin script, ft-floatd, and floating panels in ft-screens: the margin and popup overlays, controls, moving by the title bar, resizing (edges and tab), scale, full screen, close, and back to desktop. Add drag and drop across panels, with the pointer helper change.
Done when:
- Dolphin and Kate float from "Float in VR".
- A file drags from the floating Dolphin to the floating Kate, to a screen, and back.
- The clipboard works between them.
- A menu near a floating window's edge opens past the edge.
- Closing and docking give the output back.
- Frame pacing and GPU memory are measured with several floating windows.
### Phase 2: tear off and dock by dragging
The ghost and tear-off, and the dock highlight with push-flush docking.
### Phase 3: launch floating
`ft-float run` and `ft-float launch`, window matching, new windows of floating apps, the Frametop Apps launcher entry, the picker, remembered placement, and the notification when every spare is in use.
### Phase 4: polish
The drag proxy, the glow toward a window activated out of view (and the setting to bring it in front), the Display Settings section, and docs (design.md, reference.md, and the Use table in the README).
## Risks ## Risks
- A SteamOS update can change KWin's script API or its nested backend. The script and the output handling are the parts to recheck after one. - A SteamOS update can change KWin's script API or its nested backend. The script and the output handling are the parts to recheck after one, and KWin's placement memory with them (the script undoes it, see [design.md](design.md#kwins-placement-memory)).
- GPU memory: each floating output has its own swapchain of two or three buffers, including the margin. The Frame has 16 GB shared, with about 4 GB free in normal use (2026-09-29). - GPU memory: each floating output has its own swapchain of two or three buffers, including the margin. The Frame has 16 GB shared, with about 4 GB free in normal use (2026-09-29).
- Frame pacing with many panels hasn't been measured (already an open question in design.md). Each output is a separate render pass in KWin. - Frame pacing with many panels hasn't been measured (already an open question in design.md). Each output is a separate render pass in KWin.
+28
View File
@@ -0,0 +1,28 @@
# Gaze with the controllers: a dead end
Gaze mode is a mouse and keyboard feature. On 2026-09-30 we tried to make the Frame controllers its buttons: with gaze mode on and no game running, either controller's trigger would click where you look (a tap clicks, moving the hand steers the pointer, holding still drags), the controllers' lasers would be muted, and SteamVR's dashboard would follow the gaze too. It can't be done cleanly, for the reasons below. The work wasn't merged, and it's kept outside the published history.
## What worked
- Our `ft_pointer` device can hold SteamVR's laser without a hand role, in the treadmill role. It has to hint that role when SteamVR activates it; a hint changed later never gets the `/user/treadmill` path.
- A trimmed copy of the Frame controller's compositor binding mutes the controllers' laser buttons. It's chosen with `POST /input/selectconfig.action` on vrserver's port 27062 (a JSON body; a form-encoded one gets "Parse failed"). The helper's global action sets can't do it, because SteamVR marks them inactive while its laser mouse has focus.
- vrserver's web socket on 127.0.0.1:27062 reports every controller component without taking it from anyone.
- Steam's UI can be kept from acting on the controllers by wrapping its gamepad input source (webpack module 17900) through Steam's CEF debugger.
- Our device takes the laser back 10 to 15 ms after a controller takes it.
## What broke it
- Steam reads the Frame controllers itself. They aren't devices on the host: vrserver owns their radio and passes their raw reports to Steam through SteamVR's private Steam interface. Steam's client library turns them into a virtual device ("SteamFrameVirtual", at `/steamvr/virtual`), outside every SteamVR binding. Steam's own SteamVR action manifest asks only for haptics.
- Every press and every release that Steam sees takes SteamVR's dashboard, and Frametop's panels, out of laser mode about 40 ms later. That happens whatever the bindings say and whatever Steam's UI does with the event. None of these stopped it: dropping the events in Steam's UI, removing the controller's `dualanalog` bindings, binding every button to a harmless compositor action, or setting `dashboard.modalGamepadAndLaser` to false.
- Taking the laser back after each switch leaves a gap of 20 to 40 ms, and panels treat it as the pointer leaving, so clicks and drags break.
- Steam can't be told to ignore the controllers. It won't save a controller layout for the virtual controller ("Saved Binding Selection Failed - No Identity"), and its menus don't go through the layout anyway: a live preview of the empty layout for Steam's UI (app 769) changed nothing.
- SteamVR hands the controllers to a VR app instead of Steam only while that app has the input focus, as games do. A dashboard overlay with overlay flag `1 << 4` is given the focus only in gamepad mode, and only for gamepad input.
## Options not taken
- Frametop as a transparent VR app (a scene application using OpenXR's alpha blend mode) whenever gaze mode is on. That would cut Steam off while the dashboard is closed, but Steam's dashboard pages would still take the controllers, and it costs a scene layer all the time.
- Patching Steam's running process, with an eBPF probe that writes its memory or an injected hook, to drop the controller reports while gaze mode is on. It would cover everything, but it changes Valve's software, needs Steam's client library reverse-engineered, and breaks with Steam updates.
## What was built
The attempt had a plan and a test log, probes for the laser, input focus, and SteamVR settings, a reader for vrserver's web socket, a filter for Steam's UI, and controller code in the relay and the helper. None of it is in this repo. Only the gaze dot setting (`POINTER_GAZE_DOT`) came over.
-158
View File
@@ -1,158 +0,0 @@
# Gaze first: plan
**Abandoned on 2026-09-30.** Steam reads the Frame controllers itself, outside SteamVR's bindings, and every press and release it sees takes SteamVR out of laser mode. Gaze mode stays a mouse feature; `docs/gaze-controllers.md` on `experimental` explains what was tried and why it doesn't work. This branch keeps the plan, the probes, and the code for reference; it isn't merged.
Gaze first makes the eyes the pointer for everything flat in VR, and the controllers its buttons. With gaze on and no game running, the gaze drives Frametop's 3D pointer (the `ft_pointer` driver) everywhere the mouse can go, SteamVR's dashboard included. Either controller's trigger clicks where you look, and the controllers stop showing lasers. The work happens on branch `gaze-first` (worktree `~/frametop/.worktrees/gaze-first`, from `experimental`) and is merged into `experimental` after it's been tested in the headset.
The decisions below were made with the user on 2026-09-30. Nothing is built yet: the first step is a headset session with four tests (see "Tests before building").
## Decisions
Input, with gaze on and outside games:
- The gaze drives `ft_pointer`, so anything the mouse does today works with the gaze, on Frametop's screens, floating windows, the keyboard, and SteamVR's and Steam's panels. The mouse keeps working as it does today in gaze mode.
- The gaze wakes the pointer by itself: the headset is worn and the tracker sees your eyes. No mouse movement is needed.
- Trigger, on either controller:
- A tap clicks where you look.
- Moving the controller within `POINTER_GAZE_HOLD` (0.5 s) of the press is precision: the pointer stops where you looked, and your hand's movement steers it, relative, like tap and drag on the Apple Vision Pro (the controller's position seen from the eye, not where it points). The release clicks there, and the correction goes to the gaze service as a lesson.
- Holding still for 0.5 s makes a real press; then the hand's movement drags 1:1.
- Bumper, on either controller: the same, with the right button.
- Right thumbstick: scrolls at the pointer (vertical, and horizontal when pushed sideways).
Controllers:
- They keep their hand roles and their stock SteamVR bindings: the Steam button (tap for the dashboard, double tap for the room view, hold to recenter, the screenshot chord), gamepad mode (both grips), locomotion, room setup, and every per-hand feature.
- Only the trigger and the bumper are muted from SteamVR while gaze first is on, so they never click or take the laser.
- When something still moves the laser to a controller (a Steam button summoning the dashboard, a grip), the helper moves it straight back to our device.
- Later: mute the grips too, and maybe have turning gaze off switch the controllers to gamepad mode.
Turning gaze on and off:
- Gaze is opt-in and off by default. On or off is remembered across restarts (`POINTER_GAZE`), whatever turned it on or off.
- Any game gets the controllers: a VR game (a scene application) or a flatscreen Steam game. Gaze first is off during one.
- The toggle macro, both thumbstick clicks held 1 s by default, turns gaze on or off. In a game it turns gaze first on for that game only, until the game exits.
- You can record your own toggle macro on the Gaze page of Frametop Input Settings: a chord of buttons on one or both controllers, held together for 0.5 to 2 s. The Steam button can't be part of it. It needs 2 or more buttons, or 1 button held at least 1.5 s. Firing it cancels a gaze click in progress. "Reset to default" brings back the thumbsticks. The one-button "Gaze pointer on/off" mapping and key combinations keep working.
- Gaze can't turn on without a calibration for the tracker in use, or without a working tracker service. Turning it on then opens the calibrator, or points to "Repair eye tracker".
Eye tracker:
- Our own tracker (`gaze/tracker/`) is the default. Choosing SteamVR's tracker on the Gaze page turns ours off. The same calibration rule applies to SteamVR's.
- Its root service, `frametop-eyegrab`, is installed by Frametop's installer (a step that defaults to yes) and stays enabled. Turning our tracker off means no longer asking it for frames: it then holds none of SteamVR's buffers, and nothing needs root.
- The tracker waits for a calibration before tracking, and idles whenever gaze input is off (tracking costs roughly 7 to 36 % of a core).
- If the service is missing or broken, gaze can't turn on, and the Gaze page offers "Repair eye tracker". There's no silent switch to SteamVR's tracker.
- Updates: the gaze service compares the installed `ft-eyegrab` with the build, and when they differ the Gaze page offers "Update eye tracker". Both repair and update ask for the password each time, through polkit (`pkexec`). There's no passwordless sudoers or polkit rule: the build output is writable by the user, so such a rule would let any program running as the user get root.
Calibration, in one head-locked panel:
- Quick check: one centre dot. It's captured by a dwell (about 0.6 s of steady fixation; steadiness, not position, so it works however far off the tracker is), the trigger accepts early, and it closes itself after about 4 s if ignored. It opens when the headset is put on, when our tracker notices the headset slipping (at most once every 2 minutes), and from a mappable action. If the first 3 nudges after it are still more than 2 degrees off, it asks for 5 dots.
- Full calibration: the same panel, about 60 degrees across, running the probe's calibration (21 dots in dark, medium, and bright rounds). Frametop's screens hide while it runs. It opens when turning gaze on finds no calibration. Quitting it leaves gaze off; turning gaze on again reopens it. Resetting the calibration while gaze is on turns gaze off and opens it.
- The dots move with your head, so there's no "keep your head still", and the calibration doesn't depend on where the screens are.
- The gaze probe stays as the lab tool.
## What exists
- Gaze mode (`POINTER_GAZE`, `pointer/helper/ft-pointer.cpp`): the gaze aims the pointer; the mouse's held-back press, precision, hold to drag, and nudge lessons are the model for the trigger.
- Holds in the helper (`struct Hold`): pinches and grips steer by the hand's movement seen from the eye, in the room, from where the eye was when the gesture began (`PoseHistory`). The trigger's steering is the same with the controller's position instead of the hand's.
- `gaze_precision` and `gaze_drag` (relay actions, "precision|gazedrag <source> 1|0" to the helper) steer by the controller's aim. Controller holds switch to position steering.
- Controller buttons through the helper's global action sets (`pointer/helper/vrbuttons.h`), one action set per button, active only for mapped buttons and only outside games.
- ft-screens' `controllers always|outside_games|dashboard` and its `hide`/`show` switch.
- Our tracker's headset-moved detection (`gaze/tracker/eyes_model.py`: a glint slip change over 3 px held 1 s) and its reseat after the frames stop.
- `gaze/tracker/install.sh` already installs `ft-eyegrab` as a system service with sudo.
## What SteamVR does (found 2026-09-30)
- The Frame controller's compositor bindings (`/opt/steamvr/drivers/frame_controller/resources/input/vrcompositor_bindings_frame_controller.json`) put the laser on a controller in only two ways: a button that clicks or switches the laser (trigger and bumper click; trigger, bumper, and grip move the laser to that hand with `switchlaserhand`), or summoning the dashboard with its Steam button. Everything else there doesn't touch the laser.
- The gamepad and laser modes are the `/actions/dualanalog` action set (`ModeSwitch1` and `ModeSwitch2` on the grips), with `dashboard.modalGamepadAndLaser`. Those actions are application-scoped: the mode lives in Steam's UI, and we can't set it from outside.
- The laser doesn't need a hand. The headset's own binding (`/opt/steamvr/drivers/frame_hmd/resources/input/vrcompositor_bindings_frame_hmd.json`) runs the laser from `/user/head/pose/raw`. Our device has only tried the right, left, and stylus roles; the stylus attempt got no role at all.
- A Frame controller in the hand takes its hand's role back through its touch sensors, so a device that needs a hand role loses it whenever both controllers are held. That's why the laser has to live somewhere else.
- vrserver's web socket (`/input/getstate.json` and `request_input_state_updates`, as frame-voice uses) lists each controller's `/input/trigger/click`, `/input/bumper/click`, `/input/grip/click`, `/input/thumbstick/click`, `/input/thumbstick/x` and `y`, `/input/system/click`, and each device's `side`. The headset has `/proximity`, but it flickers off for 0.3 to 0.5 s at a time while worn, so the quick check follows SteamVR's activity level (as the helper already does) rather than the raw sensor. Reading the socket takes nothing from anyone. A controller's path is `/devices/cv/<serial>` instead of `/user/hand/<side>` while our device holds that hand.
## Test results (2026-09-30)
Run with the headset on its stand and the controllers on (`pointer/probe/lasertest`, `input/vrws.py`):
1. **Laser on the treadmill role: works.** Our device held the dashboard laser with no hand role. SteamVR gives a device the `/user/treadmill` path only if it hints treadmill when it's added, so the driver hints a role that's no hand from `Activate`; a hint changed on connecting kept it at `/devices/ft_pointer/ft_pointer_0`. `GetControllerRoleForTrackedDeviceIndex` reports no role for it, so the helper's "no hand role" release skips treadmill.
2. **Muting through the helper's action sets: doesn't work.** SteamVR reported those actions inactive (`vrstatus`: `"active": []`) while its laser mouse had input focus, as frame-voice found on 2026-09-26, and a trigger pull moved the laser to its controller until the release. **Muting through the compositor binding works:** `pointer/bindings/vrcompositor_frame_controller_gazefirst.json` is the stock binding without its trigger and bumper laser entries, selected with `POST /input/selectconfig.action` on vrserver's port 27062 (a JSON body `{"app_key": "openvr.component.vrcompositor", "controller_type": "frame_controller", "url": "file:///..."}`; a form-encoded one gets "Parse failed"). `GET /input/getactions.json?app_key=openvr.component.vrcompositor` shows the choice (`current_binding_url`). With it, the triggers showed no laser. Selecting the stock file puts it back.
3. **Snap back: works** in 11 to 15 ms after a grip or a Steam button summon. A trigger held the laser until its release (0.4 to 1 s), which the muting removes. Our device also takes the laser from "none".
4. **Web socket: works.** Every click, the thumbstick axes, and both thumbsticks clicked together arrive. The headset's `/proximity` flickers off for 0.3 to 0.5 s at a time while worn, so the quick check goes by SteamVR's activity level instead.
Also found: the bumpers, the thumbsticks' movement, and their clicks switch Steam's dashboard into its controller (gamepad) mode, and the laser owner goes to "none"; both grips go back to laser mode. Steam's own Frame controller binding (`steam_vrgamepad_bindings_frame_controller.json`, app `steam.client`) has only haptics, so that input reaches Steam's UI some other way (most likely Steam Input's virtual gamepad), and a SteamVR binding can't mute it.
Decided after the tests: gaze replaces the laser pointer's controls, never the controller mode's. Controller mode takes precedence while it's on, and leaving it gives the laser back to the gaze. Right click is one grip held with a trigger (the bumpers belong to controller mode).
## Tests before building
One headset session, about 30 minutes, with the user wearing the headset. Installing the test driver needs a SteamVR restart, which closes everything in VR, so it's done at the start of the session and only when the user says so.
1. **Laser on the treadmill role.** The driver takes a `role treadmill` command, and its compositor bindings repeat the right hand's under `/user/treadmill`. Pass: after its `switchlaserhand` (`/input/a`), `GetPrimaryDashboardDevice()` is our device, and the pointer clicks Frametop's screens and the dashboard while both controllers are held. Fail: the fallback (below).
2. **Muting.** The helper activates its trigger and bumper action sets on both sides. Pass: a real trigger or bumper, pointed at a panel, neither clicks nor moves the laser, with the dashboard open and closed, and the helper sees the press. Fail: we need our own compositor binding for the Frame controller, switched when gaze first turns on and off.
3. **Snap back.** A Steam button tap on either controller, and a grip. Pass: the helper sees the laser move to a controller and moves it back within about 100 ms, with no stray click.
4. **Web socket.** Update rates for the thumbstick axes and clicks while held, and `/proximity` at don and doff.
Fallback if test 1 fails: gaze input goes straight into ft-screens for Frametop's own panels (the helper already knows which panel it hits and where), with both controllers held. SteamVR's and Steam's panels then take the gaze only while one hand is empty.
## Design by component
### Driver (`pointer/driver`)
- `role treadmill` joins `right`, `left`, and `stylus`. The hint stays OptOut while disconnected, as now.
- `ft_pointer_vrcompositor.json` and `ft_pointer_steam.json` get the same bindings under `/user/treadmill`. They're additive, so nothing changes while the device is a hand.
### Pointer helper (`pointer/helper/ft-pointer.cpp`)
- Gaze first = gaze mode on, the gaze service ready, no game, or a game with the macro's override.
- Waking: in gaze first, fresh gaze with the headset worn wakes the pointer (connect, treadmill role, `switchlaserhand`), with no mouse counts. The "no hand role" release doesn't apply to the treadmill role.
- The laser: while awake in gaze first, if the dashboard's primary device becomes a controller, press `/input/a` on ours again, at most every 100 ms. Last used wins stays off in gaze mode, as now.
- Muting: the trigger and bumper action sets on both sides are active whenever gaze first is on, whatever the relay's mappings say. Their presses go to the built-in state machine, not to the relay.
- Trigger and bumper: a hold with a new source, a controller's position. It starts as a held-back press at the gaze; moving past `POINTER_TRIGGER_DEADZONE` (about 1 degree, seen from the eye; the pull jolts the controller) within `POINTER_GAZE_HOLD` is precision at `POINTER_TRIGGER_GAIN` (0.5); still until `POINTER_GAZE_HOLD` is a real press, and dragging at `POINTER_GAZE_DRAG_GAIN` (1). Release: click, lesson (as with the mouse, up to `POINTER_GAZE_NUDGE_MAX`), or release the press. The existing controller holds (`gaze_precision`, `gaze_drag`) move to position steering too.
- Scroll: "scroll <dx> <dy>" from the relay goes to the driver, as the mouse's wheel does.
- Calibration: while ft-gazed says its panel is up ("calpanel 1|0"), the pointer hides, and a trigger press goes to ft-gazed as "calaccept" instead of clicking.
- The macro's "cancel": drops a held-back press without clicking.
- Headset worn or not goes to ft-gazed ("headset 1|0"), for the quick check.
### Input relay (`input/input-relay.py`)
- A web socket reader for vrserver, in the standard library (the host's Python has no `websockets` module, and the relay needs nothing but Python). It follows the controllers by `side`, since their paths change with the roles.
- The toggle macro: `"gaze_macro": {"buttons": ["left/thumbstick", "right/thumbstick"], "hold": 1.0}` in the rules file. Outside games it writes `POINTER_GAZE` and tells the helper; in a game it toggles the override for that game. It sends the helper "cancel" when it fires.
- Recording: Input Settings asks for "macro record" and gets back the chord and how long it was held, after the checks in "Decisions".
- Scroll: the right thumbstick's axes, while gaze first is on, as "scroll" lines to the helper.
- Games: a Steam game running (a `reaper` process with `SteamLaunch AppId=`, checked every 2 s; to confirm with a flatscreen game) goes to the helper, which already knows scene applications.
- `gaze_toggle` writes `POINTER_GAZE` instead of lasting until a restart.
### Gaze service (`gaze/ft-gazed`)
- `GAZE_TRACKER` defaults to `own`.
- Readiness goes to the helper ("gazeready 1|0"): the tracker's service works (for ours: `frametop-eyegrab` active, and its binary the same as the build) and the tracker in use has a calibration. When gaze is on but not ready, it opens the calibrator, or asks for the repair.
- Our tracker runs only while gaze is on and ready, while a calibration runs, or for the probe's lease.
- The quick check opens on "headset 1" from the helper, on our tracker's jump (at most once every 2 minutes), and on "quickcal" (the mappable action). For our tracker, the dot is a click on `@ft_eyes`, like the probe's one-dot check; for SteamVR's, it's a lesson.
- The full calibration's logic (the dots, rounds, sample rejection, and fits) moves out of the probe into `gazecal.py`, so the probe and the service share it. Quitting writes `POINTER_GAZE=0`.
- Hiding Frametop's screens during a full calibration goes through ft-screens' `hide` and `show`, keeping the user's own switch as it was.
### Calibration panel (`gaze/panel/ft-gazepanel`, new)
A small C++ OpenVR overlay program in the dev container: a head-locked overlay (placed relative to the headset) of a fixed angular size, drawn on the CPU and uploaded with `SetOverlayRaw`, like ft-screens' keyboard, with labels from stb_truetype. ft-gazed drives it over `@ft_gazepanel` (show quick or full, dot at yaw and pitch with its state, background brightness, a line of text, hide). It takes no input: the trigger comes through the helper, and the dwell is ft-gazed's. It's a separate program because the helper is already 2,000 lines, and the panel has nothing to do with pointing.
### Frametop Input Settings, Gaze page
- The gaze switch, blocked with the reason while not ready ("Calibrate first", "Repair eye tracker").
- Eye tracker: ours (default) or SteamVR's.
- Status: the service, the calibration, and when it was made. Buttons for Calibrate, Quick check, and Repair or Update eye tracker (`pkexec gaze/tracker/install.sh`, which then runs without sudo inside).
- The toggle macro: what it is, Record (a 3 s countdown, hold the chord, confirm), and Reset to default.
### Installer (`install.sh`)
- A step for our eye tracker ("It needs your password (sudo)"), defaulting to yes, and the gaze service, which needs no root. Gaze itself stays off.
## Order of work
1. The tests above, then this plan updated with the results.
2. Driver and helper: the treadmill role, muting, snap back, waking by gaze, and the trigger and bumper holds. Gaze still turns on the old way.
3. Relay: the web socket reader, scroll, the macro, games and the override, `POINTER_GAZE` remembered.
4. Gaze service: readiness, idling, the default tracker, the service check, and the quick check's triggers.
5. The calibration panel and calibrator: the quick check, then the full calibration and the move to 5 dots.
6. Input Settings, the installer, and the docs (`README.md`, `docs/design.md`, `docs/reference.md`, `gaze/README.md`).
Each step is tested in the headset before it's merged into `experimental`.
## Risks
- SteamVR may refuse the laser on a treadmill device, or a SteamVR update may change the Frame's compositor bindings.
- The muting relies on "Enable global input from overlays" (`steamvr/globalActionSetPriority`), which SteamVR calls experimental.
- `pkexec` runs a script the user can write. That's acceptable only because every run asks for the password.
- Our tracker's CPU cost while gaze is on, with SteamVR and a busy desktop; it idles otherwise.
- Head-locked panels can be uncomfortable; the panel stays small and short-lived, except for the full calibration.
-140
View File
@@ -1,140 +0,0 @@
# Hands in Frametop: migration plan
Hand tracking from the headset's own cameras has been built as a separate project, frame-hands (`~/Desktop/Projects/frame-hands` on the Frame, a local git repo with no remote). The plan is to make it a native Frametop component, like `gaze/` and `power/`, instead of a separate module. The work happens on branch `hands-migration` (worktree `frametop-hands/` in the PC workspace) and is merged into `experimental` after it's been tested in the headset.
Builds from this worktree must sync to their own folder on the Frame, never `~/dev/frametop`: run every script with `FRAME_REPO=/home/steamos/dev/frametop-hands`.
## Status (2026-09-30)
Steps 1-5 are done:
- frame-hands' pending work was committed there (6c63c9e).
- Its filtered history was merged under `hands/` (1a76d15), then laid out (`trackd/` to `track/`).
- The renames, the Frametop paths, and ft-camd's file capabilities are done. So are `hands/Makefile`, `build.sh`, `run.sh`, the two units, the README, the settings, the installer step, and ft-screens on the shared header.
- Built in the dev container on the Frame, and on the 7i.
- Checked without the headset:
- `ft-handreplay` against frame-hands' `fh-replay`, both x86 with `--cost`, on the whole dim recording and the first 60 s of the bright one: identical summaries and byte-identical depth dumps. The Makefile's own ncnn build is included in that.
- `ft-ringplay` into `ft-hands` on the 7i tracked, pinched, and wrote `/run/user/UID/frametop-hands/{hands,gestures}`.
- On the Frame, ft-hands in the container finds the calibration through `/run/host/persist`, and ft-camd without its capabilities refuses with a clear message.
Step 6 has started (2026-09-30 10:30):
- `hands/run.sh install` is done, and both services run from this worktree.
- The files moved to `/run/user/UID/frametop-hands/`, because `/run/user/UID/frametop` is the desktop session's own runtime folder, deleted at every desktop start.
- Until the desktop restarts from a build with this branch's ft-screens, the link `/run/user/UID/frame-hands -> frametop-hands` feeds the running one. It's tmpfs, so it's gone at reboot.
Found in the headset:
- The side cameras were swapped (`HANDS_SWAP_SIDES=1`).
- The cutout copy shader lost resolution at `mediump` (now `highp`).
- Colour capture isn't reliable (see the README).
- Two pinch fixes: one hand no longer pinches both sides, and the palm-down limit stops typing pinches.
## What frame-hands is today
| Part | What it is | Size |
| --- | --- | --- |
| `camd/` | `fh-camd`, the camera broker (C). It borrows XRService's camera DMA-BUFs read-only with `pidfd_getfd`, times them with the `v4l2_dqbuf` tracepoint, and publishes the four IR cameras (and optionally the two colour cameras) to a shared-memory ring. It starts as root and drops to the user after setup. Adapted in part from FrameEyeCameraFeed (MIT, licence file kept). `fh-camprobe` is its discovery and recording probe. | camd 1.1k lines, tp 0.4k, xrcams 0.8k, camprobe 1.1k |
| `trackd/` | `fh-tracker` (C++): the tracker, the models on ncnn, the calibration (jsoncpp), the pinch detector, the publisher, and the recorder. Also `fh-replay` (offline replay and scoring), `fh-ringplay` (plays a recording into a ring), and `nettest`. | 2.9k lines |
| `include/` | The hands file (`fh_hands.h`, read by ft-screens) and the gestures file (`fh_gestures.h`, pinches). | |
| `models/ncnn/` | MediaPipe's palm detector and hand landmark model, from the OpenCV Zoo ONNX ports (Apache-2.0), converted to ncnn in float and int8. | 5.9 MB |
| `tools/` | Python analysis: side-camera check, colour calibration check, frame viewer, gesture watcher, depth report, model comparison, int8 calibration, model conversion. | ~1.1k lines |
| `tracker/` | The Python prototype of the tracker. Some tools import its `calib.py` and `models.py`. | 1.3k lines |
| `probes/`, `notes/`, `re/`, `shim/` | One-off experiments, reverse-engineering notes on SteamVR's passthrough internals, a disassembly (not in git), and a header from an abandoned XRService shim approach. | |
| `vendor/`, `captures/` | ncnn and FrameEyeCameraFeed clones, and recordings of the user's hands and room (tens of GB). Neither is in git. | |
Today it runs by hand: `sudo camd/fh-camd`, then `trackd/fh-tracker`. There are no units and no installer. Files: `/run/frame-hands/ir-ring` (the ring, in a root-owned folder), and `$XDG_RUNTIME_DIR/frame-hands/hands` and `gestures`.
Frametop already has the consumer side on `experimental`: `screens/handcut.{h,cpp}` cuts the hands out of the screens, with its own copy of the hands file layout, and `screens/handtest.cpp` tries it on a test panel.
## Where it goes
A top-level `hands/` folder, laid out like `gaze/`:
```
hands/
README.md # from trackd/README.md and camd/README.md
build.sh # ft-camd, ft-hands; --tools also builds the replay tools
run.sh # install|uninstall|start|stop|restart|status|log
frametop-camd.service # user units (templates, @REPO@)
frametop-hands.service
include/ # fhring.h, fh_hands.h, fh_gestures.h: shared with screens/ and pointer/
camd/ # ft-camd: camd.c tp.c xrcams.c, LICENSE.FrameEyeCameraFeed
track/ # ft-hands: tracker, nets, calib, pinch, publish, record; replay.cpp
# (ft-handreplay) and ringplay.cpp (ft-ringplay) for recordings
models/ # the ncnn models, with NOTICE (Apache-2.0, MediaPipe / OpenCV Zoo)
tools/ # the Python checks, watch_gestures, depth_report, calib.py, ring.py
```
Left behind in frame-hands, which stays as the lab: the recordings, the Python prototype (the tools that need `calib.py` or `models.py` get a trimmed copy in `hands/tools/`), `probes/`, `notes/`, `re/`, `shim/`, `camprobe`, and `vendor/`. The reverse-engineering notes don't belong in a public repo, and recordings are images of the user's hands and room, so they never go into git.
## Names
Programs within 15 characters, `ft-` prefix; files under `frametop`:
| Now | In Frametop |
| --- | --- |
| `fh-camd` | `ft-camd` |
| `fh-tracker` | `ft-hands` |
| `fh-replay`, `fh-ringplay` | `ft-handreplay`, `ft-ringplay` |
| `/run/frame-hands/ir-ring` | `/run/user/UID/frametop-hands/cam-ring` |
| `$XDG_RUNTIME_DIR/frame-hands/hands`, `gestures` | `/run/user/UID/frametop-hands/hands`, `gestures` |
The source keeps its `fh_` identifiers and header names (`fh_hands.h`, `fh_gestures.h`, `fhring.h`), and the file formats keep their magic strings, so recordings and tools from frame-hands keep working. Programs, units and runtime paths change.
## Build
- `hands/build.sh` builds in the dev container through `scripts/frame.sh --build`, into `hands/build/`, like the other components. `FRAME_BUILDER=pc` can take the ncnn build.
- ncnn: fetched at a pinned tag (20260526, as now) into `hands/build/ncnn` and built once, the way `screens/build.sh` fetches the OpenVR header, with frame-hands' options so results match. `NCNN=` points the build at an existing install instead. Every net runs single-threaded (`num_threads = 1`), with the tracker spreading nets over its own pinned threads, so OpenMP could go later.
- ft-hands runs in the dev container like ft-pointer and ft-powerd (`distrobox enter dev --`, after `scripts/container-up.sh`). Today's fh-tracker runs on the host and works only because the host happens to have the same `libjsoncpp.so.25` and libgomp as the container. Inside the container the calibration is at `/run/host/persist`, and calib.cpp (and `tools/calib.py`) fall back to it when `/persist` isn't there.
- ft-camd has to run on the host (below), so it's linked statically (only libc and libm; `glibc-static` goes into `setup/dev-container.sh`). The host has an older glibc than the container.
## Running it
**ft-camd needs privileges**, only while it sets up: `pidfd_getfd` on XRService (the Frame has `ptrace_scope=1`), system-wide tracepoints (`perf_event_paranoid=2`), and the tracepoint files, which are root-only (`/sys/kernel/tracing/events/v4l2/v4l2_dqbuf/{id,format}` are mode 0440). A rootless container's root can't do any of that, so it runs on the host. Two ways:
- **A. File capabilities (chosen, 2026-09-30).** The installer runs `sudo setcap cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep hands/build/ft-camd` once. ft-camd then runs as the user, in a user unit `PartOf=steamvr.service`, so it starts and stops with SteamVR, and its ring lives in the user's runtime folder. It drops all capabilities after setup, as it drops root today. Nothing ever runs as root. Writing the file clears its capabilities, so a rebuilt ft-camd needs the setcap again. It changes rarely. `/home` on the Frame is ext4 without `nosuid`, so file capabilities work there.
- **B. Root system service**, like the Bluetooth fixes: a root-owned copy in `/var/lib/frametop/`, a unit in `/etc/systemd/system/`. It would have to watch for XRService itself, because a system unit can't follow the user's `steamvr.service`.
Either way the password is needed once at install, through the same `sudo -S` path the Bluetooth fixes use, and only after asking.
**ft-hands** is a user unit, `frametop-hands.service`: after `frametop-camd.service`, `PartOf=steamvr.service`, nice 5, model threads on CPUs 5-7 (measured best on 2026-09-29).
**Settings** in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES=1` and `HANDS_CPUS=5,6,7`, read by ft-hands. It's on while its services are installed (`hands/run.sh install`, `uninstall`), so there's no `HANDS` switch. There's no setting for colour yet. Later, a switch in Frametop Display Settings.
**Installer:** an optional last step in `install.sh`, off by default, which asks first because it needs sudo.
## Interfaces
- `screens/handcut.cpp` includes `hands/include/ft_hands.h` instead of its own copy of the layout, and reads the new path. ft-screens and ft-hands change together on this branch.
- Pinches go to the pointer helper. It maps the gestures file and checks the begin and end counters each tick. A begin is a press, an end the release, and the pinch point's movement a drag. In gaze mode, the press lands where you look. The counters mean a quick tap between two ticks isn't missed. The tracker knows nothing about the pointer.
## Open items that aren't part of the move
These block shipping hands to other people, not the migration:
- **The side-camera swap.** After some XRService restarts, fh-camd publishes the two side cameras under each other's names. Today it's caught by hand (`tools/check_sides.py --ring`, then `--swap-sides`). It needs fixing at the source (tell the buffers apart by the `dqbuf` tracepoint's device, the way the colour pair is split), or at least an automatic check at start-up.
- **The colour cameras' calibration mapping** (`tools/check_color.py` on a recording with texture).
- **Depth when one camera loses the hand.** From the 2026-09-30 replay measurements: drifting 10% per update toward the one-camera guess (`kMonoDepthGain`) makes the depth worse than keeping the last distance. Try 0.02.
## Public repo
Frametop is public. **Not pushed to GitHub until the user says it's ready** (user decision, 2026-09-30). When it is, it publishes:
- The camera borrowing (`pidfd_getfd` on XRService's buffers) and the tracepoint timing. FrameEyeCameraFeed already does the same publicly. Its MIT licence and credit stay with the code.
- The models, under Apache-2.0, with a NOTICE.
It doesn't publish the reverse-engineering notes, the probes, or any recording. They stay in frame-hands.
## History
frame-hands' work was committed there first (6c63c9e, its 4th commit). Its history was then filtered to drop what stays behind (`notes/`, `probes/`, `shim/`, `camd/camprobe.c`, the camprobe tools, the Python prototype except `calib.py` and `models.py`, and `.frame-job`) from every commit. It was merged into this branch under `hands/` (a subtree merge), so blame still leads to where each line came from. The renames come after, as their own commits.
## Steps
1. In frame-hands: commit the pending work, as its last state before the move (needs the user's OK).
2. On this branch: import it under `hands/`, then rename the programs and paths. The behaviour stays identical.
3. `hands/build.sh`, `run.sh`, the two units, the README, the settings, and the installer step.
4. ft-screens' hand cutouts on the shared header and the new path.
5. Check without the headset. `ft-handreplay` on the 2026-09-29 recordings with `--cost` is repeatable, so its summary must match `fh-replay`'s exactly. And `ft-ringplay` into ft-hands must publish the same hands as into fh-tracker.
6. In the headset, with the user and after asking: stop fh-camd and fh-tracker, install the services from `~/dev/frametop-hands`, and restart the desktop from this branch so ft-screens reads the new path.
7. Pinch into the pointer helper (it can also follow the merge). The gaze work is on the Frame's `~/frametop` main: on 2026-09-30 that branch had 4 commits `experimental` doesn't have, plus uncommitted work in the pointer helper's gaze mode. Build this step on wherever that work lands, not on this branch's older copy.
8. Merge into `experimental`. It's checked out in a worktree on the Frame (`~/frametop/.worktrees/experimental`, where the live desktop runs), so the merge happens there, or `experimental` is switched away first.
+2 -2
View File
@@ -16,8 +16,8 @@ The input relay takes the volume keys from every device that has them, so gamesc
ft-screens drops keys while no screen has focus or the SteamVR dashboard is open, but always lets through the release of a key the desktop saw pressed, so a modifier held as the dashboard opens doesn't stay down. ft-screens drops keys while no screen has focus or the SteamVR dashboard is open, but always lets through the release of a key the desktop saw pressed, so a modifier held as the dashboard opens doesn't stay down.
- **A release that never arrives leaves the key held in the desktop.** KWin repeats held keys itself, so a stuck letter repeats and a stuck modifier changes every later key (Ctrl+Alt held turns T into Konsole). Pressing and releasing the key again clears it. - **A key whose release never arrives stays held in the desktop until the relay clears it, within about a second.** KWin repeats held keys itself, so a stuck letter repeats and a stuck modifier changes every later key (Ctrl+Alt held turns T into Konsole). The relay remembers which keys it told the desktop went down, and once a second it releases any that no keyboard holds (`reconcile_desktop_keys`, which asks the kernel with `EVIOCGKEY`). Pressing and releasing the key again also clears it.
- **A keyboard that disconnects mid-press is one way to get there.** The relay forgets the held key without telling ft-screens. The same goes for the relay restarting while a key is down. - **A keyboard that disconnects mid-press, or a relay restart with a key down, is how it happens.** The once-a-second check catches the first. A relay that starts doesn't know what an earlier one left down, so it releases the modifiers on the desktop; another key left down that way stays until it's pressed and released again.
- **To see where a key went,** run `scripts/keys-report.py` and reproduce the problem while it records. It logs the modifiers, Tab, and Esc (no other keys) as the relay reads them and as its virtual keyboard sends them on, with the device roles and grabs, which programs have each keyboard open, and the relay's and desktop's logs. - **To see where a key went,** run `scripts/keys-report.py` and reproduce the problem while it records. It logs the modifiers, Tab, and Esc (no other keys) as the relay reads them and as its virtual keyboard sends them on, with the device roles and grabs, which programs have each keyboard open, and the relay's and desktop's logs.
- **Switching where typing goes waits for keys to come up.** The relay changes a keyboard's grab only while none of its keys are down, so a press and its release go to the same side. A key held for a long time delays the switch until it's let go. - **Switching where typing goes waits for keys to come up.** The relay changes a keyboard's grab only while none of its keys are down, so a press and its release go to the same side. A key held for a long time delays the switch until it's let go.
+54
View File
@@ -0,0 +1,54 @@
# Profiles
Profiles are built: Display Settings, `ft-layout`, each profile's launcher entry, and the input relay's `profile:NAME` action open them, and the desktop can start in one.
A profile is a named layout that also opens apps. It holds:
- where each screen goes, with its size in metres, curve, roll, and pin (what a named layout held before profiles);
- which screens show and which are hidden;
- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale.
So a "Work" profile can put three screens around you with a browser, two terminals, and an editor on them, and a "Couch" profile can hide every screen and float one video player in front of you.
## Decisions
| # | Question | Decision |
|---|---|---|
| 1 | Profiles and named layouts | One list. Named layouts grow into profiles; a layout saved before profiles existed is a profile with no apps and every screen shown |
| 2 | Making one | Capture what's open: the screens and every app's windows. Display Settings lists a profile's apps, so one can be removed. No editor beyond that |
| 3 | Apps with several windows | One entry per window. The app is launched once; when its first window shows up, it's launched again for each window still missing. A browser that restores its own windows gets launched once, a terminal twice |
| 4 | What's recorded of an app | Its desktop file name, or its command line if it has none. Not what it had open: tabs, files, and folders are left to the app's own restore |
| 5 | Switching while apps are open | Additive: launch what's missing, move the windows that match into place, leave the rest alone. Nothing is ever closed |
| 6 | Saving changes | Only on an explicit save. Moving things after switching doesn't change the profile |
| 7 | Starting one | Four ways: a default profile when the desktop starts, Display Settings, a launcher entry for each profile, and a mappable action |
| 8 | Plasma's session restore | Off in the Frametop session, so a profile is the only thing that reopens apps |
| 9 | Screen count and resolution | Not part of a profile. They're global, because changing them restarts the desktop, which closes every window |
## Where profiles live
`~/.config/frametop-layout.json` keeps its `layouts` as they are (`{"Work": [screen places]}`), so older copies of ft-layout still read it. What a profile adds goes in a parallel `profiles` map under the same names:
```
"layouts": {"Work": [{"pos": ..., "face": ..., "roll": ..., "metres": ..., "curve": ..., "pin": ...}, ...]},
"profiles": {"Work": {"hidden": [3],
"windows": [
{"app": "org.kde.konsole", "screen": 2, "rect": [40, 60, 1200, 800], "maximized": false},
{"app": "com.brave.Browser", "screen": 1, "maximized": true},
{"cmd": ["/opt/tool/run"], "class": "tool", "screen": 1, "rect": [...]},
{"app": "org.kde.dolphin", "float": {"rel": [12 numbers], "pixels": [1400, 900],
"scale": 1.2, "mpp": 0.00097}}]}},
"default_profile": "Work"
```
- `screen` is 1-based, as everywhere in Frametop. `rect` is the window's frame in KWin's logical units, relative to its screen's output, so it survives the screens being arranged differently.
- A floating window's place (`rel`) is its panel's centre and axes in the frame of the primary screen's panel, the same way ft-floatd remembers each app's place. The screens go relative to your head when the profile is applied, and the floating windows follow them. `mpp` is its density in metres per pixel, and `scale` its scale, put back with the rest.
- Renaming or deleting a layout renames or deletes its profile entry with it.
## How it works
- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name: single-instance and D-Bus-activated apps open their windows from a process that was already running.
- **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
- **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
- **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.
- **Display Settings.** On the Layout & profiles page, the arrangement list has the profiles, which can be renamed and deleted. Open profile and Save as profile… are the page's actions. A profile's apps are listed with where each goes and a button to leave one out, plus which screens it hides. Start in profile picks the one the desktop starts with. The Visibility tab's Screens shown switches hide screens one at a time.
+57 -17
View File
@@ -48,7 +48,7 @@ The Visibility & pins tab of Frametop Display Settings decides when the screens
- While you look at a chosen controller (the wrist gesture). - While you look at a chosen controller (the wrist gesture).
- Only after you show them with the hotkey. - Only after you show them with the hotkey.
In the last three modes the hotkey shows the screens anyway. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps: In the last three modes the hotkey shows the screens anyway. A screen can also be hidden on its own (Screens shown on the same tab, or `ft-layout hide N`): it stays hidden whatever the mode or the hotkey says, until it's shown again there. Windows on it stay put, and a new window that would open on it floats instead (ft-floatd). Profiles use this to show only some screens. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:
- During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up. - During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps. - Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
@@ -64,11 +64,15 @@ ft-screens listens for datagrams on the abstract socket `@ft_screens` and replie
``` ```
place N x y z yaw pitch roll width N metres curve N radius|on|off place N x y z yaw pitch roll width N metres curve N radius|on|off
pin N|all left|right|head [matrix] unpin N|all size N w h pin N|all left|right|head [matrix] unpin N|all size N w h
get N screens head state key code value scale N s vrkeyboard show|hide|toggle get N screens head state key code value scale N s vrkeyboard show|hide|toggle|close
visibility always|dashboard|gesture|toggle wrist degrees gesture left|right degrees visibility always|dashboard|gesture|toggle wrist degrees gesture left|right degrees
hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible
conceal N|all reveal N|all concealed cutouts on|off|state cutouts predict on|off cutouts lead ms
float N mpp x y w h title unfloat N pose N matrix sub N k x y w h | sub N k off minimized N 0|1 carry N
``` ```
`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
## Input relay ## Input relay
SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps and reconnects gets new device nodes, SteamVR keeps reading the dead ones, and the mouse stops working until SteamVR restarts. `input/input-relay.py` avoids this. It creates two virtual devices, `frametop virtual mouse` and `frametop virtual keyboard`, through `/dev/uinput` before SteamVR starts. It then grabs USB and Bluetooth mice and keyboards as they come and go and forwards their events, so SteamVR only ever sees the virtual devices, which never go away. SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps and reconnects gets new device nodes, SteamVR keeps reading the dead ones, and the mouse stops working until SteamVR restarts. `input/input-relay.py` avoids this. It creates two virtual devices, `frametop virtual mouse` and `frametop virtual keyboard`, through `/dev/uinput` before SteamVR starts. It then grabs USB and Bluetooth mice and keyboards as they come and go and forwards their events, so SteamVR only ever sees the virtual devices, which never go away.
@@ -81,6 +85,8 @@ The relay also owns the volume keys, on every device that has them, the headset'
desktops.sh relay install # enable it (starts with the next reboot or SteamVR start) desktops.sh relay install # enable it (starts with the next reboot or SteamVR start)
desktops.sh relay status | log | uninstall desktops.sh relay status | log | uninstall
input/input-relay.py --no-grab # try it without taking devices from SteamVR input/input-relay.py --no-grab # try it without taking devices from SteamVR
input/test/keys-test.py # key combinations and modifier taps, against fake devices (safe next to the live relay)
steam/ft-steam menu # what Open Steam menu does; ft-steam check: Steam's UI still has the calls
``` ```
The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices. The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices.
@@ -95,7 +101,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again. Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up.
``` ```
pointer/driver/build.sh && pointer/driver/install.sh install # then restart SteamVR pointer/driver/build.sh && pointer/driver/install.sh install # then restart SteamVR
@@ -104,19 +110,19 @@ pointer/helper/run.sh status | log | restart
pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer
``` ```
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, and `POINTER_GAZE_SHOW`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes). The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
## Frametop Input Settings ## Frametop Input Settings
A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages: A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages:
- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device. - Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, open or close the keyboard, head follow on or off, gaze pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep. - Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through. Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`. - Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. - Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button. - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove. - Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Check headset fit (opens the probe's Headset fit mode), Reload calibration, and Forget nudges. - Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
- Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam. - Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam.
Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it. Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.
@@ -130,7 +136,7 @@ The desktop's own screen arrangement follows where the screens are around you, w
Frametop Display Settings has four tabs (three with the gamescope backend, which has no Visibility & pins): Frametop Display Settings has four tabs (three with the gamescope backend, which has no Visibility & pins):
- Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers. - Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers.
- Layout: a curve around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, or a flat wall. Both take rows, distance, gap, and height. Save current arrangement saves the positions, sizes, curves, and pins you set by hand under a name instead. Named layouts are listed with the presets: pick one and Arrange now to switch to it, and rename or delete it with the buttons next to the list. A layout saved with fewer screens than you have now leaves the others where they were saved last, or where the preset would put them. A preview shows the layout from above and from the front, and a switch turns auto-arrange at startup on or off. - Layout (the Layout & profiles page): the Arrangement list starts with two presets: Curved around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, and Flat wall. Both take rows, distance, gap, and height, and Arrange now applies them. Save as profile… saves where the screens are now (positions, sizes, curves, and pins), which ones are hidden, and the open apps and where their windows are, under a name. Profiles are listed in Arrangement after the presets: pick one and Open profile switches to it, and the buttons next to the list rename or delete it. Below it, Apps lists the profile's windows and where they go, and Leave out drops one. A profile saved with fewer screens than you have now leaves the others where they were saved last, or where the preset would put them. Start in profile picks the profile the desktop starts in; with None, a switch turns auto-arrange at startup on or off. A preview shows the layout from above and from the front. See [profiles.md](profiles.md).
- Visibility & pins: the visibility, game, and controller settings described above, the wrist angle, where each screen is pinned (in the room, a wrist, or your head), and buttons to pin all screens or unpin them. - Visibility & pins: the visibility, game, and controller settings described above, the wrist angle, where each screen is pinned (in the room, a wrist, or your head), and buttons to pin all screens or unpin them.
- Power: when the displays turn off while the headset isn't used, their state now, Turn displays off now (to try it), and Stay awake while plugged in. See [Displays off and sleep](#displays-off-and-sleep). - Power: when the displays turn off while the headset isn't used, their state now, Turn displays off now (to try it), and Stay awake while plugged in. See [Displays off and sleep](#displays-off-and-sleep).
@@ -139,18 +145,39 @@ Frametop Display Settings has four tabs (three with the gamescope backend, which
``` ```
layout/ft-layout apply # arrange every screen layout/ft-layout apply # arrange every screen
layout/ft-layout capture # save the current arrangement and sizes as the layout layout/ft-layout capture # save the current arrangement and sizes as the layout
layout/ft-layout save NAME # ...under a name too, and use it layout/ft-layout save NAME # ...under a name too, with the open apps and hidden screens (a profile, docs/profiles.md), and use it
layout/ft-layout use NAME # switch to a named layout and arrange the screens in it layout/ft-layout use NAME # switch to a profile: arrange the screens in it and open its apps
layout/ft-layout open NAME # a profile's launcher entry: use it, or start the desktop in it
layout/ft-layout default NAME|none # the profile the desktop starts with (start --wait runs it at desktop start)
layout/ft-layout layouts # list the named layouts (* = in use); rename OLD NEW, delete NAME layout/ft-layout layouts # list the named layouts (* = in use); rename OLD NEW, delete NAME
layout/ft-layout pin N|all left|right|head # pin as they are now; unpin N|all layout/ft-layout pin N|all left|right|head # pin as they are now; unpin N|all
layout/ft-layout plan # print the arrangement as JSON (no VR needed) layout/ft-layout plan # print the arrangement as JSON (no VR needed)
layout/ft-layout scale # per-screen scale, positions (as the screens are around you), and taskbar screen, to KWin layout/ft-layout scale # per-screen scale, positions (as the screens are around you), and taskbar screen, to KWin
layout/ft-layout toggle # hide or show all screens layout/ft-layout toggle # hide or show all screens
layout/ft-layout hide N|all # hide a screen on its own, whatever the visibility mode; show N|all brings it back, hidden lists them
display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H shortcuts display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H shortcuts
``` ```
The layout is stored relative to your head when it's applied. `/tmp/frametop-layout.log` has the run from the last desktop start. The layout is stored relative to your head when it's applied. `/tmp/frametop-layout.log` has the run from the last desktop start.
## Floating windows
A desktop window can float in VR as a panel of its own, away from the screens. Meta+Shift+F floats the window under the pointer (or the active one, over the wallpaper), or puts it back on its screen if it floats. So do Float in VR in every window's menu (Alt+F3; Back to Desktop on a floating one), the button left of Close in its title bar, and a mouse button, controller button, or key combination mapped to Float window in VR in Frametop Input Settings; Put all floating windows back is mappable too. Launch as Standalone, in an app's right-click menu in the Application Launcher or the taskbar, starts the app with its first window floating, where that app last floated or in front of you. [floating-windows.md](floating-windows.md) explains how it works.
`float/ft-floatd` does this. It runs inside the desktop's Plasma session (log: `/tmp/frametop-floatd.log`), loads the KWin script `float/frametop-float.js`, and moves each floating window to a spare KWin output of its own, which ft-screens shows as a panel cropped to the window. The settings are in `~/.config/frametop.conf`: `FLOAT_SLOTS` is how many windows can float at once (8, at most 16; 0 turns floating off; restart the desktop after a change), and `FLOAT_MARGIN` the pixels around each window on its output, so menus have room past its edges (300). Each app's last floating place, size, and scale are kept in `~/.config/frametop-float.json` by desktop file name, relative to the primary screen, so they move with the screens. `FT_FLOAT_DEBUG=1` in ft-floatd's environment logs every event from the KWin script.
With floating on, the session gives the desktop's windows Frametop's own decoration (`decoration/`): Breeze's look plus the float button. Apps that draw their own title bar, like Chromium and Electron apps, don't have the button. `decoration/apply.sh` puts a changed copy into the running desktop, and `decoration/apply.sh --off` goes back to Breeze until the next desktop start.
```
float/ft-float float [ID|active] # float a window (default: the active one, as a toggle)
float/ft-float float pointer # the float key: the window under the pointer, floated or put back
float/ft-float dock [ID|active|all] # put a floating window back (default: the active one), or all of them
float/ft-float launch org.kde.dolphin # start an app (its desktop file name) floating
float/ft-float run COMMAND [ARG...] # the same for a command
float/ft-float close ID # close a window
float/ft-float list # the spare outputs and what floats on them
```
## Displays off and sleep ## Displays off and sleep
SteamVR turns the displays off a few seconds after the headset's proximity sensor says it came off. A stand or display mount that covers the sensor makes the headset seem worn, so its displays stay on, and Steam, which then counts someone as present, never puts it to sleep either. SteamVR turns the displays off a few seconds after the headset's proximity sensor says it came off. A stand or display mount that covers the sensor makes the headset seem worn, so its displays stay on, and Steam, which then counts someone as present, never puts it to sleep either.
@@ -172,14 +199,27 @@ power/run.sh off | on # the displays off now, or back on
power/run.sh log power/run.sh log
``` ```
## Hand tracking (experimental) ## Gaze pointer (experimental)
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker also detects pinches, for clicking where you look with the gaze pointer (not wired to the pointer yet). It's optional: `hands/run.sh install`, or the last step of `install.sh`. In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
- Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
- The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
- A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
[gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
## Hand tracking (experimental, deferred)
Deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it. It still builds and runs, installed by hand, for working on it.
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode a pinch clicks where you look when it opens; hold it and move the hand to correct the pointer first. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`.
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. - `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`.
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches) next to the ring. - `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
- Both start and stop with SteamVR. `hands/run.sh status` and `hands/run.sh log` show how they're doing. - The install leaves both off, and they don't start with SteamVR. `ft-handsctl on` starts them while SteamVR runs, and `ft-handsctl off` stops them; they also stop with SteamVR. The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells) and `HANDS_CPUS`. - Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md). Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
+267 -13
View File
@@ -2,7 +2,7 @@
// loads it into the desktop's KWin over D-Bus (org.kde.kwin.Scripting) and talks to it: // loads it into the desktop's KWin over D-Bus (org.kde.kwin.Scripting) and talks to it:
// - events go to ft-floatd as JSON strings (org.frametop.Float.Event), for the windows it // - events go to ft-floatd as JSON strings (org.frametop.Float.Event), for the windows it
// cares about: floating windows (the ones on a spare output, WL-<screens> and up), their // cares about: floating windows (the ones on a spare output, WL-<screens> and up), their
// popups and dialogs, new windows, and "Float in VR" requests; // popups and dialogs, new windows, and requests to float or dock one;
// - commands come back through a long poll: the script calls NextCommand, ft-floatd // - commands come back through a long poll: the script calls NextCommand, ft-floatd
// answers when it has one (or after a while with nothing), and the script calls again. // answers when it has one (or after a while with nothing), and the script calls again.
// KWin scripts can call D-Bus but can't serve it, hence the poll. Window ids are KWin's // KWin scripts can call D-Bus but can't serve it, hence the poll. Window ids are KWin's
@@ -12,6 +12,11 @@ const SERVICE = "org.frametop.Float", PATH = "/Float", IFACE = "org.frametop.Flo
let screens = 0; // outputs WL-0 .. WL-<screens - 1> are screens; the rest are spares let screens = 0; // outputs WL-0 .. WL-<screens - 1> are screens; the rest are spares
let polling = false; let polling = false;
const watched = {}; // id -> true once its signals are connected const watched = {}; // id -> true once its signals are connected
let marking = false; // the script itself is setting keep-below (see mark)
const settled = {}; // id -> {output, frame, fullScreen, maximized}: where a window belongs (see putBack)
const held = {}; // id -> {w, h, until, asked}: a size asked for, for a second (see hold)
const moved = {}; // id -> true, or "back" once put back: it moved while KWin changed the outputs
let layout = "", layoutOutputs = {}, layoutSince = 0; // the outputs at the last screensChanged
function send(ev) { function send(ev) {
callDBus(SERVICE, PATH, IFACE, "Event", JSON.stringify(ev)); callDBus(SERVICE, PATH, IFACE, "Event", JSON.stringify(ev));
@@ -47,29 +52,203 @@ function info(w) {
frame: rect(w.frameGeometry), client: rect(w.clientGeometry), popup: w.popupWindow, frame: rect(w.frameGeometry), client: rect(w.clientGeometry), popup: w.popupWindow,
transient: w.transient, parent: w.transientFor ? String(w.transientFor.internalId) : "", transient: w.transient, parent: w.transientFor ? String(w.transientFor.internalId) : "",
normal: w.normalWindow, dialog: w.dialog, fullScreen: w.fullScreen, minimized: w.minimized, normal: w.normalWindow, dialog: w.dialog, fullScreen: w.fullScreen, minimized: w.minimized,
onAllDesktops: w.onAllDesktops onAllDesktops: w.onAllDesktops, maximized: isMaximized(w)
}; };
} }
// KWin 6.2's scripts have no maximize mode to read: a window is maximized when it fills its
// output's maximize area.
function isMaximized(w) {
if (!w.normalWindow || !w.output) return false;
const a = workspace.clientArea(KWin.MaximizeArea, w), g = w.frameGeometry;
return g.x === a.x && g.y === a.y && g.width === a.width && g.height === a.height;
}
function report(type, w) { function report(type, w) {
if (w.deleted) return; // a window on its way out still changes output and size
const ev = info(w); const ev = info(w);
ev.ev = type; ev.ev = type;
send(ev); send(ev);
} }
// KWin's placement memory (its PlacementTracker) keeps each window's geometry for each layout of
// the outputs (every enabled output's name and geometry), and when the outputs come back to a
// layout it has seen, it puts the windows back where they were in it. That's for plugging monitors
// in and out, and it does harm here. A spare output changes size after its window does, so what
// KWin keeps for a spare's size is the window's next size: resizing a floating window back to a
// size it had set off an endless flip between two sizes. And floating or docking one window could
// move others, even onto a spare or off one. So the script keeps where each window belongs
// (settled), tells ft-floatd nothing while KWin changes the outputs, and once KWin is done
// (screensChanged comes after its restore) puts the floating windows back, and the screens' windows
// too when only spares changed.
function outputsNow() {
const all = workspace.screens, out = {};
for (let i = 0; i < all.length; ++i) {
const g = all[i].geometry;
out[all[i].name] = g.x + "," + g.y + " " + g.width + "x" + g.height;
}
return out;
}
function keyOf(outputs) {
return Object.keys(outputs).sort().map(n => n + "=" + outputs[n]).join(" ");
}
function takeLayout() {
layoutOutputs = outputsNow();
layout = keyOf(layoutOutputs);
layoutSince = 0;
}
// KWin is changing the outputs: they differ from the last screensChanged.
function changingOutputs() {
if (keyOf(outputsNow()) === layout) {
layoutSince = 0;
return false;
}
if (!layoutSince) {
layoutSince = Date.now();
} else if (Date.now() - layoutSince > 2000) {
takeLayout(); // screensChanged should have come by now: don't stay quiet for good
return false;
}
return true;
}
function settle(w) {
if (w.output) {
settled[String(w.internalId)] = {output: w.output.name, frame: rect(w.frameGeometry), fullScreen: w.fullScreen};
}
}
// ft-floatd put the window here: it's where it belongs now.
function expect(w, output, c, fullScreen) {
settled[String(w.internalId)] = {output: output, frame: {x: c.x, y: c.y, w: c.w, h: c.h}, fullScreen: fullScreen};
hold(w, c.w, c.h);
}
// A size asked for (by ft-floatd, or by the script putting a window back) comes in when the app
// answers, and until then the app can still answer older requests: one KWin's restore made, or,
// just after it opened, its own. For a second, the script asks again instead of taking those;
// then it takes the size the window has (an app can refuse a size, below its minimum).
const holdTimer = new QTimer();
holdTimer.singleShot = true;
holdTimer.timeout.connect(() => {
const now = Date.now();
Object.keys(held).forEach(id => {
const h = held[id];
if (h.until > now) return;
delete held[id];
const w = byId(id);
if (!h.asked || !w || w.deleted) return; // (nothing held back: nothing to tell)
settle(w);
if (isSpare(w.output)) report("geometry", w);
});
if (Object.keys(held).length) holdTimer.start();
});
function hold(w, width, height) {
held[String(w.internalId)] = {w: width, h: height, until: Date.now() + 1000};
holdTimer.interval = 1100;
holdTimer.start();
}
// A size change while a size is held: true when it isn't that size (the script asked again).
// (ft-floatd's sizes can be fractional, the window's are whole: within a pixel is the same.)
function holding(w) {
const id = String(w.internalId), h = held[id], s = settled[id];
if (!h) return false;
const g = w.frameGeometry;
if (!s || h.until < Date.now() || w.move || w.resize || (Math.abs(g.width - h.w) < 1 && Math.abs(g.height - h.h) < 1)) {
delete held[id];
return false;
}
w.frameGeometry = {x: s.frame.x, y: s.frame.y, width: h.w, height: h.h};
h.asked = true;
return true;
}
// Runs while KWin's output change still counts as going on (nothing reported), so the steps on
// the way don't reach ft-floatd: told only where the window ends up (see reportMoved).
function putBack(w, screensChanged) {
const id = String(w.internalId), s = settled[id];
if (w.deleted || !s || screens === 0 || !marked(w)) return;
const o = outputByName(s.output);
// KWin's restore sets full screen (and maximized) as it was in that layout too: with a
// floating window that flipped forever, its output changing size with it. Ask for the
// state it had: KWin's request hasn't reached the app yet, so it never sees it.
w.fullScreen = s.fullScreen;
if (o && isSpare(o) && !s.fullScreen) w.setMaximize(false, false);
// Not where KWin had to move it: its output went, a screen changed, or it's full screen or
// maximized (KWin fits those to their output).
const back = o && !s.fullScreen && !w.fullScreen && !w.move && !w.resize && !s.maximized
&& (isSpare(o) || (!screensChanged && !isMaximized(w)));
if (!back) return;
if (!w.output || w.output.name !== s.output) workspace.sendClientToScreen(w, o);
w.frameGeometry = {x: s.frame.x, y: s.frame.y, width: s.frame.w, height: s.frame.h};
if (isSpare(o)) hold(w, s.frame.w, s.frame.h);
// Moved during the change (by KWin, by the lines above, or the size ft-floatd asked for came
// in): report where it is, and keep settled as it is.
if (moved[id]) moved[id] = "back";
}
// After an output change: tell ft-floatd where the windows that moved during it are now.
function reportMoved(w) {
const id = String(w.internalId), s = settled[id], how = moved[id];
if (!how) return;
delete moved[id];
if (w.deleted) return;
if (how !== "back") settle(w);
if (!w.output || !s || w.output.name !== s.output) {
report("output", w);
mark(w);
} else if (isSpare(w.output)) {
report("geometry", w);
}
}
workspace.screensChanged.connect(() => {
const before = layoutOutputs, now = outputsNow();
if (keyOf(now) === layout) return;
let screensChanged = screens === 0;
Object.keys(Object.assign({}, before, now)).forEach(name => {
const m = /^WL-(\d+)$/.exec(name);
if (before[name] !== now[name] && !(m && parseInt(m[1]) >= screens)) screensChanged = true;
});
const all = workspace.windowList();
all.forEach(w => putBack(w, screensChanged));
takeLayout();
all.forEach(reportMoved);
});
// Floating windows, and popups and dialogs on a spare output: tell ft-floatd about changes. // Floating windows, and popups and dialogs on a spare output: tell ft-floatd about changes.
function watch(w) { function watch(w) {
const id = String(w.internalId); const id = String(w.internalId);
if (watched[id]) return; if (watched[id]) return;
watched[id] = true; watched[id] = true;
const onSpare = () => isSpare(w.output); const onSpare = () => isSpare(w.output);
w.frameGeometryChanged.connect(() => { if (onSpare()) report("geometry", w); }); w.frameGeometryChanged.connect(() => {
w.outputChanged.connect(() => report("output", w)); if (changingOutputs()) {
moved[id] = true;
return;
}
if (holding(w)) return;
settle(w);
if (onSpare()) report("geometry", w);
});
w.outputChanged.connect(() => {
if (changingOutputs()) {
moved[id] = true;
return;
}
if (!held[id]) settle(w); // (held: the place asked for is settled already)
report("output", w);
mark(w);
});
w.keepBelowChanged.connect(() => keepBelowChanged(w));
w.interactiveMoveResizeStarted.connect(() => { w.interactiveMoveResizeStarted.connect(() => {
if (onSpare()) send({ev: "move-start", id: id, move: w.move, resize: w.resize, frame: rect(w.frameGeometry)}); if (onSpare()) send({ev: "move-start", id: id, move: w.move, resize: w.resize, frame: rect(w.frameGeometry)});
}); });
w.interactiveMoveResizeFinished.connect(() => { if (onSpare()) report("move-end", w); }); w.interactiveMoveResizeFinished.connect(() => { if (onSpare()) report("move-end", w); });
w.fullScreenChanged.connect(() => { if (onSpare()) report("fullscreen", w); }); w.fullScreenChanged.connect(() => {
if (changingOutputs()) {
moved[id] = true;
return;
}
if (settled[id]) settled[id].fullScreen = w.fullScreen;
if (onSpare()) report("fullscreen", w);
});
w.minimizedChanged.connect(() => { if (onSpare()) report("minimized", w); }); w.minimizedChanged.connect(() => { if (onSpare()) report("minimized", w); });
w.maximizedChanged.connect(() => { w.maximizedChanged.connect(() => {
// A floating window stays an ordinary window: its output is its size plus a margin. // A floating window stays an ordinary window: its output is its size plus a margin.
@@ -79,16 +258,52 @@ function watch(w) {
workspace.windowAdded.connect(w => { workspace.windowAdded.connect(w => {
watch(w); watch(w);
settle(w);
report("added", w); report("added", w);
}); });
workspace.windowRemoved.connect(w => { workspace.windowRemoved.connect(w => {
send({ev: "removed", id: String(w.internalId)}); const id = String(w.internalId);
delete watched[String(w.internalId)]; send({ev: "removed", id: id});
delete watched[id];
delete settled[id];
delete held[id];
delete moved[id];
}); });
workspace.windowActivated.connect(w => { workspace.windowActivated.connect(w => {
if (w && isSpare(w.output)) send({ev: "activated", id: String(w.internalId)}); if (w && isSpare(w.output)) send({ev: "activated", id: String(w.internalId)});
}); });
workspace.windowList().forEach(watch); takeLayout();
workspace.windowList().forEach(w => {
watch(w);
settle(w);
});
// Keep-below means "floating" in the Frametop desktop. The title bar's float button (Frametop's
// window decoration, decoration/) is the Keep Below button, so setting the flag on a window on
// the screens floats it, and clearing it on a floating one docks it. The script keeps the flag
// set on every floating window, its dialogs included, and cleared everywhere else, however the
// window got there. Kept below, a window alone on its own output only has the wallpaper under it.
function marked(w) {
return w.managed && !w.deleted && !w.specialWindow && !w.popupWindow;
}
function topOf(w) {
let top = w;
for (let n = 0; top.transientFor && n < 10; ++n) top = top.transientFor;
return top;
}
function mark(w) {
if (screens === 0 || !marked(w)) return;
const want = isSpare(w.output);
if (w.keepBelow === want) return;
marking = true;
w.keepBelow = want;
marking = false;
}
function keepBelowChanged(w) {
if (marking || screens === 0 || !marked(w)) return;
const top = topOf(w);
if (w.keepBelow !== isSpare(top.output)) requestFloat(top);
}
function requestFloat(w) { function requestFloat(w) {
if (!w || !w.normalWindow || w.popupWindow) return; if (!w || !w.normalWindow || w.popupWindow) return;
@@ -104,15 +319,35 @@ registerUserActionsMenu(w => {
triggered: () => requestFloat(w) triggered: () => requestFloat(w)
}; };
}); });
registerShortcut("Frametop Float Window", "Frametop: Float Window in VR (or put it back)", "Meta+Shift+F", // The float key is the input relay's (float_toggle, Meta+Shift+F by default): it reaches us as
() => requestFloat(workspace.activeWindow)); // "request-pointer". No shortcut of KWin's own, so one press can't float a window and dock it again.
// The window under KWin's pointer (where the 3D mouse or a laser last was on a panel): the top
// one there, a popup or dialog standing for the window it belongs to. Null over the wallpaper
// or the taskbar.
function underPointer() {
const p = workspace.cursorPos;
const order = workspace.stackingOrder;
for (let i = order.length - 1; i >= 0; --i) {
const w = order[i];
if (w.deleted || w.minimized || w.hidden || !w.managed) continue;
const g = w.frameGeometry;
if (p.x < g.x || p.y < g.y || p.x >= g.x + g.width || p.y >= g.y + g.height) continue;
const top = topOf(w);
return top.normalWindow && !top.popupWindow ? top : null;
}
return null;
}
function run(c) { function run(c) {
const w = c.id ? byId(c.id) : null; const w = c.id ? byId(c.id) : null;
switch (c.cmd) { switch (c.cmd) {
case "config": case "config":
screens = c.screens; screens = c.screens;
workspace.windowList().forEach(w => report("window", w)); workspace.windowList().forEach(w => { report("window", w); mark(w); });
break;
case "mark": // after a float that didn't happen: keep-below back as it was
if (w) mark(w);
break; break;
case "place": { // onto an output, at a frame rectangle (logical, global) case "place": { // onto an output, at a frame rectangle (logical, global)
if (!w) break; if (!w) break;
@@ -120,13 +355,22 @@ function run(c) {
if (!o) break; if (!o) break;
if (w.fullScreen && !c.keepFullScreen) w.fullScreen = false; if (w.fullScreen && !c.keepFullScreen) w.fullScreen = false;
w.setMaximize(false, false); w.setMaximize(false, false);
expect(w, o.name, c, w.fullScreen && !!c.keepFullScreen);
workspace.sendClientToScreen(w, o); workspace.sendClientToScreen(w, o);
w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h}; w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h};
if (c.onAllDesktops !== undefined) w.onAllDesktops = c.onAllDesktops; if (c.onAllDesktops !== undefined) w.onAllDesktops = c.onAllDesktops;
if (c.maximized) {
// Maximized: KWin picks the size, and the place above is only where it goes.
delete held[c.id];
settled[c.id].maximized = true;
w.setMaximize(true, true);
}
break; break;
} }
case "geometry": case "geometry":
if (w) w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h}; if (!w) break;
expect(w, settled[c.id] ? settled[c.id].output : (w.output ? w.output.name : ""), c, w.fullScreen);
w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h};
break; break;
case "close": case "close":
if (w) w.closeWindow(); if (w) w.closeWindow();
@@ -140,9 +384,19 @@ function run(c) {
case "info": case "info":
if (w) report("window", w); if (w) report("window", w);
break; break;
case "request-active": // ft-float float|dock active: like the shortcut case "report-all": // a profile's capture: every window as it is now, then a marker
workspace.windowList().forEach(w => report("window", w));
send({ev: "reported", token: c.token});
break;
case "request-float": // ft-float float ID: float it, if it isn't floating
if (w && !isSpare(w.output)) requestFloat(w);
break;
case "request-active": // ft-float float|dock active
requestFloat(workspace.activeWindow); requestFloat(workspace.activeWindow);
break; break;
case "request-pointer": // the float key: the window under the pointer, else the active one
requestFloat(underPointer() || workspace.activeWindow);
break;
} }
} }
+9 -1
View File
@@ -2,11 +2,18 @@
"""ft-float: talk to ft-floatd (floating windows in the Frametop desktop). """ft-float: talk to ft-floatd (floating windows in the Frametop desktop).
ft-float float [ID|active] float a window (default: the active one; for it, a toggle) ft-float float [ID|active] float a window (default: the active one; for it, a toggle)
ft-float float pointer the float key: float the window under the pointer (else the
active one), or put it back if it floats
ft-float dock [ID|active] put a floating window back on the desktop ft-float dock [ID|active] put a floating window back on the desktop
ft-float dock all put every floating window back
ft-float launch APP start an app (its desktop file name, e.g. org.kde.dolphin) and
float its first window where that app last floated
ft-float run COMMAND [ARG...] the same for a command
ft-float close ID close a window ft-float close ID close a window
ft-float list the spare outputs and what floats on them ft-float list the spare outputs and what floats on them
FT_FLOAT_SOCKET names ft-floatd's socket (default frametop_float). FT_FLOAT_SOCKET names ft-floatd's socket (default frametop_float).
""" """
import json
import os import os
import socket import socket
import sys import sys
@@ -17,7 +24,8 @@ s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
s.bind("") s.bind("")
s.settimeout(5) s.settimeout(5)
try: try:
s.sendto(" ".join(sys.argv[1:]).encode(), "\0" + os.environ.get("FT_FLOAT_SOCKET", "frametop_float")) text = "run " + json.dumps(sys.argv[2:]) if sys.argv[1] == "run" else " ".join(sys.argv[1:])
s.sendto(text.encode(), "\0" + os.environ.get("FT_FLOAT_SOCKET", "frametop_float"))
reply = s.recv(8192).decode() reply = s.recv(8192).decode()
except OSError as e: except OSError as e:
sys.exit(f"ft-floatd didn't answer ({e}); is the Frametop desktop running?") sys.exit(f"ft-floatd didn't answer ({e}); is the Frametop desktop running?")
+3
View File
@@ -0,0 +1,3 @@
#!/bin/sh
# Launch as Standalone: write the apps' desktop file copies (see ft_apps.py).
exec python3 "$(dirname "$(readlink -f "$0")")/ft_apps.py" "$@"
+123
View File
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""Launch as Standalone: an action on every app in the Frametop desktop's menus.
Plasma's Application Launcher has no way to add an entry to every app's right-click menu,
but its menu (and the taskbar's) shows each app's own desktop actions. So the Frametop
desktop reads copies of the apps' desktop files with one more action, "Launch as
Standalone", which runs `ft-float launch <desktop file name>`: the app starts and its first
window floats in VR (docs/floating-windows.md, decision 26).
The copies go in OUT (~/.local/share/frametop/apps/applications), and the session puts
OUT_ROOT first in XDG_DATA_DIRS, so a copy wins over the app's own file. Plasma's app cache
is keyed by those directories, so Desktop Mode never sees the copies. Files in
XDG_DATA_HOME/applications come before every data dir, so an app customized there keeps
its own file and has no Launch as Standalone. A copy drops DBusActivatable: a D-Bus
activated app would be asked to run the action itself, and it doesn't know ours.
ft-float-apps write the copies (the session script runs it before Plasma starts;
ft-floatd again whenever an app's desktop file changes)
"""
import os
import sys
from gi.repository import GLib
ACTION = "frametop-standalone"
OUT_ROOT = os.path.expanduser("~/.local/share/frametop/apps")
OUT = os.path.join(OUT_ROOT, "applications")
FT_FLOAT = os.path.join(os.path.dirname(os.path.realpath(__file__)), "ft-float")
GROUP = "Desktop Entry"
def data_dirs():
"""(XDG_DATA_HOME, the XDG_DATA_DIRS other than ours)."""
home = os.environ.get("XDG_DATA_HOME") or os.path.expanduser("~/.local/share")
ours = os.path.realpath(OUT_ROOT)
dirs = [d for d in os.environ.get("XDG_DATA_DIRS", "/usr/local/share:/usr/share").split(":")
if d and os.path.realpath(d) != ours]
return home, dirs
def app_dirs():
"""Every applications folder that holds the apps' own desktop files, home first."""
home, dirs = data_dirs()
return [os.path.join(d, "applications") for d in [home] + dirs]
def scan():
"""Desktop file name -> (its path, whether it's in XDG_DATA_HOME), the first one found
winning, as the desktop spec has it."""
found = {}
home_apps = app_dirs()[0]
for root in app_dirs():
for dirpath, _dirs, files in os.walk(root, followlinks=True):
for name in files:
if name.endswith(".desktop"):
path = os.path.join(dirpath, name)
found.setdefault(os.path.relpath(path, root).replace("/", "-"), (path, root == home_apps))
return found
def standalone(path, desktop_id):
"""The desktop file with Launch as Standalone added, or None for one that isn't a
visible app."""
kf = GLib.KeyFile()
try:
kf.load_from_file(path, GLib.KeyFileFlags.KEEP_TRANSLATIONS | GLib.KeyFileFlags.KEEP_COMMENTS)
except GLib.Error:
return None
def get(key):
try:
return kf.get_string(GROUP, key)
except GLib.Error:
return None
if get("Type") != "Application" or not get("Exec"):
return None
if (get("NoDisplay") or "").lower() == "true" or (get("Hidden") or "").lower() == "true":
return None
actions = [a for a in (get("Actions") or "").split(";") if a and a != ACTION]
kf.set_string(GROUP, "Actions", ";".join(actions + [ACTION]) + ";")
try:
kf.remove_key(GROUP, "DBusActivatable")
except GLib.Error:
pass
group = "Desktop Action " + ACTION
kf.set_string(group, "Name", "Launch as Standalone")
kf.set_string(group, "Icon", "window-new")
exe = FT_FLOAT if " " not in FT_FLOAT else '"' + FT_FLOAT + '"'
kf.set_string(group, "Exec", f"{exe} launch {desktop_id}")
return kf.to_data()[0]
def write_all():
"""Write the copies, and remove the ones whose app is gone. Returns how many there are."""
os.makedirs(OUT, exist_ok=True)
keep = set()
for desktop_id, (path, in_home) in sorted(scan().items()):
if in_home:
continue # XDG_DATA_HOME's own file wins over a copy anyway
data = standalone(path, desktop_id)
if data is None:
continue
keep.add(desktop_id)
out = os.path.join(OUT, desktop_id)
try:
with open(out) as f:
if f.read() == data:
continue
except OSError:
pass
with open(out + ".tmp", "w") as f:
f.write(data)
os.replace(out + ".tmp", out)
for name in os.listdir(OUT):
if name.endswith(".desktop") and name not in keep:
os.remove(os.path.join(OUT, name))
return len(keep)
if __name__ == "__main__":
if len(sys.argv) > 1:
sys.exit(__doc__)
print(f"{write_all()} apps with Launch as Standalone in {OUT}")
+576 -25
View File
@@ -11,7 +11,12 @@ which window floats on which spare output and panel, and connects three parts:
- kscreen-doctor, to turn spare outputs on and off and place them in KWin's layout. - kscreen-doctor, to turn spare outputs on and off and place them in KWin's layout.
Commands and ft-screens' events arrive as datagrams on @frametop_float (ft-float is the Commands and ft-screens' events arrive as datagrams on @frametop_float (ft-float is the
command-line side). Replies go to the sender: command-line side). Replies go to the sender:
float [ID|active] dock [ID|active] close ID list quit (ft-float) float [ID|active|pointer] dock [ID|active|all] close ID list quit (ft-float)
launch APP.desktop run ["argv", ...] (start an app and float its first window)
windows (-> JSON: the open apps' windows and where they are, for a profile; ft-layout save)
profile NAME (open a profile's apps: move the matching windows, launch the missing ones)
("float pointer" is the float key: the window under the pointer, else the active one,
floated or docked; the input relay sends it for float_toggle, and "dock all" for dock_all)
dock N | close N | resize N W H | scale N STEPS (ft-screens, N = its screen) dock N | close N | resize N W H | scale N STEPS (ft-screens, N = its screen)
Spare outputs are WL-<screens> .. WL-<screens + slots - 1>. A floating window's output is its Spare outputs are WL-<screens> .. WL-<screens + slots - 1>. A floating window's output is its
@@ -22,6 +27,25 @@ layout, all within Xwayland's 32767-pixel limit.
Usage: ft-floatd [--screens N] [--slots N] [--margin PX] [--control NAME] [--socket NAME] Usage: ft-floatd [--screens N] [--slots N] [--margin PX] [--control NAME] [--socket NAME]
Defaults: FT_SCREEN_COUNT (from the session) or the layout's count, FLOAT_SLOTS and Defaults: FT_SCREEN_COUNT (from the session) or the layout's count, FLOAT_SLOTS and
FLOAT_MARGIN from ~/.config/frametop.conf (8 and 300), @ft_screens, @frametop_float. FLOAT_MARGIN from ~/.config/frametop.conf (8 and 300), @ft_screens, @frametop_float.
Launching floating: "launch" starts a desktop file's app (Gio), "run" a command. The next
normal window from that process (or a child), or with that desktop file name, within
LAUNCH_SECONDS floats: where that app last floated, or in front of you. Single-instance and
D-Bus-activated apps open their window from a process that was already running, hence the
name. Each app's last floating place (its pose relative to the primary screen, so it moves
with the screens' layout), size in pixels, and scale are kept in PLACES_PATH, by desktop
file name, whenever one of its windows stops floating.
Profiles (docs/profiles.md): "windows" lists every app window with where it is: on a screen
(its rectangle on that output, maximized or not) or floating (its pose relative to the primary
screen, size, scale). "profile NAME" takes a profile's windows from the layout file: the
windows of each app already open (oldest first) move to its entries, and for the entries
left the app is launched once, and again for each window still missing 3 seconds after its
first one shows up (a browser restores its own windows; a terminal opens one each time).
Nothing is ever closed.
Launch as Standalone (float/ft_apps.py): copies of the apps' desktop files with that action,
which only the Frametop desktop reads. ft-floatd rewrites them when apps change.
""" """
import argparse import argparse
import json import json
@@ -36,11 +60,12 @@ import time
import dbus import dbus
import dbus.mainloop.glib import dbus.mainloop.glib
import dbus.service import dbus.service
from gi.repository import GLib from gi.repository import Gio, GLib
HERE = os.path.dirname(os.path.realpath(__file__)) HERE = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "layout")) sys.path.insert(0, os.path.join(HERE, "..", "layout"))
import ft_layout # noqa: E402 (config and layout) import ft_layout # noqa: E402 (config and layout)
import ft_apps # noqa: E402 (Launch as Standalone)
SERVICE = IFACE = "org.frametop.Float" SERVICE = IFACE = "org.frametop.Float"
PATH = "/Float" PATH = "/Float"
@@ -49,6 +74,9 @@ POLL_SECONDS = 20 # NextCommand answers empty after this (KWin's D-Bus ti
SPARE_X, SPARE_CELL = 12000, 5000 # spares in KWin's layout: a grid from here, 4 across SPARE_X, SPARE_CELL = 12000, 5000 # spares in KWin's layout: a grid from here, 4 across
PULL_OUT = 0.3 # a floated window starts this far in front of its screen (metres), clear of it PULL_OUT = 0.3 # a floated window starts this far in front of its screen (metres), clear of it
DEFAULT_MPP = 1.6 / 1920 # metres per pixel when ft-screens can't say (no SteamVR) DEFAULT_MPP = 1.6 / 1920 # metres per pixel when ft-screens can't say (no SteamVR)
PLACES_PATH = os.path.expanduser("~/.config/frametop-float.json") # each app's last floating place
LAUNCH_SECONDS = 30 # a launched app's window must show up within this
IN_FRONT = 2.0 # at most this far in front of you, for an app with no remembered place
DEBUG = os.environ.get("FT_FLOAT_DEBUG") == "1" # log every event from the script DEBUG = os.environ.get("FT_FLOAT_DEBUG") == "1" # log every event from the script
@@ -108,6 +136,20 @@ def kscreen(*args):
return "" return ""
def output_rects():
"""Each output's place and size in KWin's layout (logical)."""
try:
data = json.loads(kscreen("-j") or "{}")
except ValueError:
return {}
out = {}
for o in data.get("outputs", []):
pos, size, scale = o.get("pos") or {}, o.get("size") or {}, float(o.get("scale", 1)) or 1.0
if o.get("name") and size:
out[o["name"]] = (pos.get("x", 0), pos.get("y", 0), size.get("width", 0) / scale, size.get("height", 0) / scale)
return out
def output_scales(): def output_scales():
try: try:
data = json.loads(kscreen("-j") or "{}") data = json.loads(kscreen("-j") or "{}")
@@ -116,6 +158,90 @@ def output_scales():
return {o["name"]: float(o.get("scale", 1)) for o in data.get("outputs", []) if o.get("name")} return {o["name"]: float(o.get("scale", 1)) for o in data.get("outputs", []) if o.get("name")}
def load_places():
try:
with open(PLACES_PATH) as f:
places = json.load(f)
return places if isinstance(places, dict) else {}
except (OSError, ValueError):
return {}
def save_places(places):
tmp = PLACES_PATH + ".tmp"
with open(tmp, "w") as f:
json.dump(places, f, indent=1)
os.replace(tmp, PLACES_PATH)
def to_local(ref, g):
"""A panel's pose (ft_layout.parse_get) in the frame of another panel (ref): its centre
and axes, 12 numbers."""
axes = (ref["x"], ref["y"], ref["z"])
d = [g["center"][k] - ref["center"][k] for k in range(3)]
out = [ft_layout.dot(d, a) for a in axes]
for v in (g["x"], g["y"], g["z"]):
out += [ft_layout.dot(v, a) for a in axes]
return [round(v, 5) for v in out]
def to_world(ref, local):
"""to_local's inverse: (centre, x, y, z) in the world."""
axes = (ref["x"], ref["y"], ref["z"])
def turn(v):
return [sum(v[i] * axes[i][k] for i in range(3)) for k in range(3)]
c = turn(local[0:3])
c = [c[k] + ref["center"][k] for k in range(3)]
return c, turn(local[3:6]), turn(local[6:9]), turn(local[9:12])
def pid_chain(pid):
"""A process and its ancestors, nearest first."""
out = []
while pid > 1 and len(out) < 32:
out.append(pid)
try:
with open(f"/proc/{pid}/stat") as f:
pid = int(f.read().rsplit(")", 1)[1].split()[1])
except (OSError, ValueError, IndexError):
break
return out
# Windows a profile doesn't keep: Plasma's own, the Frametop settings apps, the login splash.
SKIP_APPS = ("org.kde.plasmashell", "org.kde.krunner", "org.kde.ksplashqml", "org.kde.polkit-kde-authentication-agent-1")
def recordable(ev):
"""An app's top-level window, the kind a profile keeps."""
if not ev.get("normal") or ev.get("popup") or ev.get("transient"):
return False
app, cls = ev.get("app", ""), ev.get("cls", "")
return app not in SKIP_APPS and not any(n.startswith(("ft-", "frametop", "ksplash")) for n in (app, cls))
def cmdline(pid):
try:
with open(f"/proc/{int(pid)}/cmdline", "rb") as f:
return [a.decode(errors="replace") for a in f.read().split(b"\0") if a]
except (OSError, ValueError, TypeError):
return []
class Launch:
"""An app we started: its first window floats, or (for a profile) its windows go to the
profile's entries for it, in order."""
def __init__(self, app, pid, entries=None, argv=None):
self.app = app # its desktop file name, without .desktop ("" for a command)
self.argv = argv # the command, when it isn't a desktop file's app
self.pids = [pid] if pid else [] # the processes we started (none when D-Bus started it)
self.entries = entries # a profile's entries still waiting for a window, or None
self.relaunched = False
self.until = time.monotonic() + LAUNCH_SECONDS
class Float: class Float:
"""A floating window.""" """A floating window."""
@@ -131,7 +257,9 @@ class Float:
self.normal = None # its size in pixels when last not full screen self.normal = None # its size in pixels when last not full screen
self.unfull_until = 0.0 # left full screen just now (see follow) self.unfull_until = 0.0 # left full screen just now (see follow)
self.move_from = None # frame when a title-bar move started (put back after) self.move_from = None # frame when a title-bar move started (put back after)
self.resizing = False # KWin's resize by the window's edge is going on (see follow)
self.subs = {} # popup or dialog id -> number on the panel self.subs = {} # popup or dialog id -> number on the panel
self.app = "" # its desktop file name (for its remembered place)
class Slot: class Slot:
@@ -153,11 +281,17 @@ class Daemon:
self.slots = [Slot(k, args.screens) for k in range(args.slots)] self.slots = [Slot(k, args.screens) for k in range(args.slots)]
self.floats = {} # window id -> Float self.floats = {} # window id -> Float
self.windows = {} # window id -> last info from the script self.windows = {} # window id -> last info from the script
self.gone = set() # ids of windows that closed (KWin's ids aren't reused)
self.pending = [] # commands for the script self.pending = [] # commands for the script
self.waiter = None # (reply callback, timeout source) while the script waits self.waiter = None # (reply callback, timeout source) while the script waits
self.screens = Screens(args.control) self.screens = Screens(args.control)
self.sub_numbers = {} # popup/dialog id -> (window id, number) self.sub_numbers = {} # popup/dialog id -> (window id, number)
self.next_sub = 1 self.next_sub = 1
self.launches = [] # Launch: apps we started, waiting for their window
self.captures = {} # token -> sender: "windows" requests waiting for the script's report
self.next_token = 1
self.sock = None # our socket (set by main), for deferred replies
self.places = load_places() # app -> {"rel": pose by the primary screen, "pixels", "scale"}
# ------------------------------------------------------------ the script # ------------------------------------------------------------ the script
@@ -195,8 +329,19 @@ class Daemon:
sid = int(kwin.loadScript(os.path.join(HERE, "frametop-float.js"), SCRIPT, signature="ss")) sid = int(kwin.loadScript(os.path.join(HERE, "frametop-float.js"), SCRIPT, signature="ss"))
if sid < 0: if sid < 0:
raise RuntimeError("KWin didn't load the script") raise RuntimeError("KWin didn't load the script")
bus.get_object("org.kde.KWin", f"/Scripting/Script{sid}").run(dbus_interface="org.kde.kwin.Script") # Scripting.start runs every loaded script that isn't running. Not /Scripting/Script<id>'s
# run: a reloaded script gets the old one's id while the old one is still being deleted,
# so that path is still the old script's, and the new one would never run.
kwin.start()
log(f"script loaded ({sid})") log(f"script loaded ({sid})")
# Older scripts registered the float key with KWin; the input relay owns it now. Drop
# that shortcut, so System Settings doesn't list one that does nothing.
try:
accel = dbus.Interface(bus.get_object("org.kde.kglobalaccel", "/kglobalaccel"), "org.kde.KGlobalAccel")
if accel.unregister("kwin", "Frametop Float Window"):
log("dropped KWin's old float shortcut")
except dbus.DBusException:
pass
# ------------------------------------------------------------ events from the script # ------------------------------------------------------------ events from the script
@@ -211,7 +356,10 @@ class Daemon:
# The script reports every window after "config"; spares nothing floats on are off. # The script reports every window after "config"; spares nothing floats on are off.
GLib.timeout_add(1500, self.disable_unused) GLib.timeout_add(1500, self.disable_unused)
return return
if wid in self.gone:
return # an event that came after the window closed (it would bring it back)
if kind == "removed": if kind == "removed":
self.gone.add(wid)
self.windows.pop(wid, None) self.windows.pop(wid, None)
if wid in self.floats: if wid in self.floats:
log(f"{wid[:9]} closed") log(f"{wid[:9]} closed")
@@ -221,6 +369,24 @@ class Daemon:
if wid: if wid:
self.windows[wid] = ev self.windows[wid] = ev
f = self.floats.get(wid) f = self.floats.get(wid)
if kind == "reported":
self.captured(ev.get("token"))
return
if kind == "added" and self.launches and not f:
launch = self.launched(ev)
if launch and launch.entries is None:
self.float_launched(ev, launch)
return
if launch:
self.place_entry(ev, launch.entries.pop(0))
if launch.entries and not launch.relaunched:
GLib.timeout_add_seconds(3, lambda: self.relaunch(launch) and False)
return
if kind == "added" and not f and self.on_hidden_screen(ev):
# Nobody would see it there (a profile can hide every screen): it floats instead.
log(f"{wid[:9]} ({ev.get('cls')}) opened on a hidden screen")
self.float_launched(ev, None)
return
if kind == "float-request": if kind == "float-request":
self.float_window(ev) self.float_window(ev)
elif kind == "dock-request": elif kind == "dock-request":
@@ -236,6 +402,11 @@ class Daemon:
elif kind == "move-start" and f and ev.get("move"): elif kind == "move-start" and f and ev.get("move"):
f.move_from = ev["frame"] f.move_from = ev["frame"]
self.screens.ask(f"carry {f.slot.index}", quiet=True) self.screens.ask(f"carry {f.slot.index}", quiet=True)
elif kind == "move-start" and f and ev.get("resize"):
f.resizing = True
elif kind == "move-end" and f and f.resizing:
f.resizing = False
self.follow(f, ev)
elif kind == "move-end" and f and f.move_from: elif kind == "move-end" and f and f.move_from:
# The panel carried the window; KWin may have slipped it a few pixels first. # The panel carried the window; KWin may have slipped it a few pixels first.
m, f.move_from = f.move_from, None m, f.move_from = f.move_from, None
@@ -278,9 +449,14 @@ class Daemon:
return return
# Floating when ft-floatd (re)started: take it over where it is. # Floating when ft-floatd (re)started: take it over where it is.
f = Float(wid, slot, None) f = Float(wid, slot, None)
f.app = ev.get("app", "")
slot.window = f slot.window = f
self.floats[wid] = f self.floats[wid] = f
f.scale = slot.kscale = output_scales().get(slot.output, 1.0) f.scale = slot.kscale = output_scales().get(slot.output, 1.0)
# Its panel's density as it is (follow below sets the panel again).
g, px = self.panel_get(slot.index), round(ev["frame"]["w"] * f.scale)
if g and g["metres"] > 0 and px > 0:
f.mpp = g["metres"] / px
log(f"{wid[:9]} ({ev.get('cls')}) already floats on {slot.output}") log(f"{wid[:9]} ({ev.get('cls')}) already floats on {slot.output}")
self.follow(f, ev) self.follow(f, ev)
return return
@@ -290,8 +466,16 @@ class Daemon:
for o in self.floats.values()): for o in self.floats.values()):
self.float_window(ev) self.float_window(ev)
else: else:
self.command(cmd="place", id=wid, output="WL-0", x=ev["frame"]["x"] % 400 + 100, # Onto the first screen that shows (or floating, if none does).
y=ev["frame"]["y"] % 300 + 100, w=ev["frame"]["w"], h=ev["frame"]["h"]) hidden = self.concealed()
shown = [i for i in range(self.screens_n) if i + 1 not in hidden]
if not shown:
self.float_launched(ev, None)
return
name = f"WL-{shown[0]}"
x0, y0, _, _ = output_rects().get(name, (0, 0, 0, 0))
self.command(cmd="place", id=wid, output=name, x=x0 + ev["frame"]["x"] % 400 + 100,
y=y0 + ev["frame"]["y"] % 300 + 100, w=ev["frame"]["w"], h=ev["frame"]["h"])
# ------------------------------------------------------------ floating and docking # ------------------------------------------------------------ floating and docking
@@ -339,22 +523,27 @@ class Daemon:
return float(metres) / max(1, int(size.split("x")[0])) return float(metres) / max(1, int(size.split("x")[0]))
return DEFAULT_MPP return DEFAULT_MPP
def float_window(self, ev): def float_window(self, ev, place=None, mpp=None, scale=None):
"""Float a window: its panel in front of where it was on its screen, or at place (centre
and axes in the world), at the density of its screen, or mpp, and at its screen's scale,
or scale (then ev's frame is the window's size at that scale). Returns its Float."""
wid = ev["id"] wid = ev["id"]
if wid in self.floats: if wid in self.floats:
return return self.floats[wid]
slot = self.free_slot() slot = self.free_slot()
if slot is None: if slot is None:
self.command(cmd="mark", id=wid) # the title bar button set keep-below for nothing
self.notify(f"All {len(self.slots)} floating windows are in use. Put one back on the desktop " self.notify(f"All {len(self.slots)} floating windows are in use. Put one back on the desktop "
"to float another.") "to float another.")
return return None
f = Float(wid, slot, {"output": ev["output"], "frame": ev["frame"], "onAllDesktops": ev.get("onAllDesktops")}) f = Float(wid, slot, {"output": ev["output"], "frame": ev["frame"], "onAllDesktops": ev.get("onAllDesktops")})
f.app = ev.get("app", "")
slot.window = f slot.window = f
self.floats[wid] = f self.floats[wid] = f
scales = output_scales() scales = output_scales()
f.scale = scales.get(ev["output"], 1.0) f.scale = scale or scales.get(ev["output"], 1.0)
slot.kscale = scales.get(slot.output, slot.kscale) slot.kscale = scales.get(slot.output, slot.kscale)
f.mpp = self.screen_mpp(ev["output"]) f.mpp = mpp or self.screen_mpp(ev["output"])
fr, s, m = ev["frame"], f.scale, self.margin fr, s, m = ev["frame"], f.scale, self.margin
w, h = round(fr["w"] * s), round(fr["h"] * s) w, h = round(fr["w"] * s), round(fr["h"] * s)
log(f"{wid[:9]} ({ev.get('cls')}) floats on {slot.output}: {w}x{h} px, scale {s:g}") log(f"{wid[:9]} ({ev.get('cls')}) floats on {slot.output}: {w}x{h} px, scale {s:g}")
@@ -365,12 +554,20 @@ class Daemon:
self.set_size(slot, (w + 2 * m, h + 2 * m), whole(slot.kscale, s)) self.set_size(slot, (w + 2 * m, h + 2 * m), whole(slot.kscale, s))
self.screens.ask(f"scale {slot.index} {s:g}") # for pointer positions (KWin's units) self.screens.ask(f"scale {slot.index} {s:g}") # for pointer positions (KWin's units)
self.set_panel(f, (m, m, w, h), title=round((ev["client"]["y"] - fr["y"]) * s)) self.set_panel(f, (m, m, w, h), title=round((ev["client"]["y"] - fr["y"]) * s))
if place:
self.pose(f, *place)
else:
self.place_panel(f, ev) self.place_panel(f, ev)
kscreen(f"output.{slot.output}.enable", f"output.{slot.output}.scale.{s:g}", kscreen(f"output.{slot.output}.enable", f"output.{slot.output}.scale.{s:g}",
f"output.{slot.output}.position.{slot.pos[0]},{slot.pos[1]}") f"output.{slot.output}.position.{slot.pos[0]},{slot.pos[1]}")
self.rescaled(slot, s) self.rescaled(slot, s)
self.command(cmd="place", id=wid, output=slot.output, x=slot.pos[0] + m / s, y=slot.pos[1] + m / s, self.command(cmd="place", id=wid, output=slot.output, x=slot.pos[0] + m / s, y=slot.pos[1] + m / s,
w=fr["w"], h=fr["h"], onAllDesktops=True) w=fr["w"], h=fr["h"], onAllDesktops=True)
return f
def pose(self, f, c, xa, ya, za):
rows = [f"{xa[k]:.5f} {ya[k]:.5f} {za[k]:.5f} {c[k]:.4f}" for k in range(3)]
self.screens.ask(f"pose {f.slot.index} {' '.join(rows)}")
def set_panel(self, f, crop, title=0): def set_panel(self, f, crop, title=0):
x, y, w, h = crop x, y, w, h = crop
@@ -389,8 +586,7 @@ class Daemon:
dx = ((fr["x"] - out["x"]) + fr["w"] / 2 - out["w"] / 2) * s * f.mpp dx = ((fr["x"] - out["x"]) + fr["w"] / 2 - out["w"] / 2) * s * f.mpp
dy = ((fr["y"] - out["y"]) + fr["h"] / 2 - out["h"] / 2) * s * f.mpp dy = ((fr["y"] - out["y"]) + fr["h"] / 2 - out["h"] / 2) * s * f.mpp
p = [c[k] + xa[k] * dx - ya[k] * dy + za[k] * PULL_OUT for k in range(3)] p = [c[k] + xa[k] * dx - ya[k] * dy + za[k] * PULL_OUT for k in range(3)]
rows = [f"{xa[k]:.5f} {ya[k]:.5f} {za[k]:.5f} {p[k]:.4f}" for k in range(3)] self.pose(f, p, xa, ya, za)
self.screens.ask(f"pose {f.slot.index} {' '.join(rows)}")
def follow(self, f, ev): def follow(self, f, ev):
"""The window moved or resized on its output: crop the panel to it, and keep the """The window moved or resized on its output: crop the panel to it, and keep the
@@ -406,9 +602,17 @@ class Daemon:
full = bool(ev.get("fullScreen")) or (fills and time.monotonic() > f.unfull_until) full = bool(ev.get("fullScreen")) or (fills and time.monotonic() > f.unfull_until)
f.full = full f.full = full
w, h = round(fr["w"] * s), round(fr["h"] * s) w, h = round(fr["w"] * s), round(fr["h"] * s)
if not full: if not full and not (f.normal and abs(f.normal[0] - w) <= math.ceil(s) and abs(f.normal[1] - h) <= math.ceil(s)):
# (Within a logical pixel it's the same size: a size asked for in pixels comes out
# rounded to whole logical pixels. Keeping it stops scale changes from creeping.)
f.normal = (w, h) f.normal = (w, h)
m = 0 if full else self.margin m = 0 if full else self.margin
if f.resizing:
# KWin ends a resize by the window's edge whenever an output changes: the output
# follows when it's done (the margin is room to grow until then).
x, y = round((fr["x"] - out["x"]) * s), round((fr["y"] - out["y"]) * s)
self.set_panel(f, (x, y, w, h), title=round((cl["y"] - fr["y"]) * s))
return
self.set_size(slot, f.normal if full and f.normal else (w + 2 * m, h + 2 * m)) self.set_size(slot, f.normal if full and f.normal else (w + 2 * m, h + 2 * m))
if not full: if not full:
x0, y0 = slot.pos[0] + m / s, slot.pos[1] + m / s x0, y0 = slot.pos[0] + m / s, slot.pos[1] + m / s
@@ -423,10 +627,14 @@ class Daemon:
panel stays the same size (KWin's output scale, in steps of 10%).""" panel stays the same size (KWin's output scale, in steps of 10%)."""
if not f.frame or f.full or steps == 0: if not f.frame or f.full or steps == 0:
return return
s = min(3.0, max(0.5, round(f.scale * 1.1 ** steps * 20) / 20)) self.set_scale(f, round(f.scale * 1.1 ** steps * 20) / 20)
if s == f.scale:
def set_scale(self, f, s):
"""The window's scale (KWin's output scale), at the same size in pixels."""
s = min(3.0, max(0.5, s))
if not f.frame or f.full or s == f.scale or self.floats.get(f.id) is not f:
return return
w, h = round(f.frame["w"] * f.scale), round(f.frame["h"] * f.scale) w, h = f.normal or (round(f.frame["w"] * f.scale), round(f.frame["h"] * f.scale))
slot, m = f.slot, self.margin slot, m = f.slot, self.margin
log(f"{f.id[:9]} scale {f.scale:g} -> {s:g}") log(f"{f.id[:9]} scale {f.scale:g} -> {s:g}")
f.scale = s f.scale = s
@@ -450,17 +658,45 @@ class Daemon:
want, slot.want = slot.size, None want, slot.want = slot.size, None
self.set_size(slot, want) self.set_size(slot, want)
def dock(self, f, frame=None): def dock(self, f, frame=None, output=None, maximized=False):
"""Back where it came from (or onto screen 1 if we don't know).""" """Back where it came from (or onto screen 1 if we don't know), or onto output at frame."""
saved = f.saved or {"output": "WL-0", "frame": dict(f.frame or {"x": 100, "y": 100, "w": 800, "h": 600}), saved = f.saved or {"output": "WL-0", "frame": dict(f.frame or {"x": 100, "y": 100, "w": 800, "h": 600}),
"onAllDesktops": False} "onAllDesktops": False}
fr = frame or saved["frame"] fr = frame or saved["frame"]
log(f"{f.id[:9]} back to {saved['output']}") if not output:
self.command(cmd="place", id=f.id, output=saved["output"], x=fr["x"], y=fr["y"], w=fr["w"], h=fr["h"], output = saved["output"]
onAllDesktops=bool(saved.get("onAllDesktops"))) m = re.match(r"WL-(\d+)$", output)
hidden = self.concealed()
if m and int(m.group(1)) + 1 in hidden:
# It came from a screen that's hidden now: onto the first one that shows.
shown = [i for i in range(self.screens_n) if i + 1 not in hidden]
if shown:
x0, y0, ow, oh = output_rects().get(f"WL-{shown[0]}", (0, 0, 0, 0))
output = f"WL-{shown[0]}"
w, h = min(fr["w"], ow or fr["w"]), min(fr["h"], oh or fr["h"])
fr = {"x": x0 + max(0, (ow - w) / 2), "y": y0 + max(0, (oh - h) / 2), "w": w, "h": h}
log(f"{f.id[:9]} back to {output}")
self.command(cmd="place", id=f.id, output=output, x=fr["x"], y=fr["y"], w=fr["w"], h=fr["h"],
onAllDesktops=bool(saved.get("onAllDesktops")), maximized=maximized)
def remember(self, f):
"""Keep where an app's window floated (before its panel goes)."""
if not f.app or not f.frame:
return
g, ref = self.panel_get(f.slot.index), self.reference()
if not g or not ref:
return
pixels = list(f.normal or (round(f.frame["w"] * f.scale), round(f.frame["h"] * f.scale)))
self.places[f.app] = {"rel": to_local(ref, g), "pixels": pixels, "scale": f.scale,
"mpp": round(f.mpp, 8)}
try:
save_places(self.places)
except OSError as e:
log(f"couldn't save {PLACES_PATH}: {e}")
def release(self, f): def release(self, f):
"""Its window left: hide the panel and turn the spare off.""" """Its window left: hide the panel and turn the spare off."""
self.remember(f)
slot = f.slot slot = f.slot
if slot.window is f: if slot.window is f:
slot.window = None slot.window = None
@@ -470,6 +706,269 @@ class Daemon:
self.screens.ask(f"unfloat {slot.index}", quiet=True) self.screens.ask(f"unfloat {slot.index}", quiet=True)
kscreen(f"output.{slot.output}.disable") kscreen(f"output.{slot.output}.disable")
# ------------------------------------------------------------ launching floating
def panel_get(self, index):
reply = self.screens.ask(f"get {index}", quiet=True)
return ft_layout.parse_get(reply) if reply.startswith("ok") else None
def primary(self):
"""The primary screen (0-based): the one with the taskbar."""
try:
return min(ft_layout.primary_screen(ft_layout.load_layout()), self.screens_n - 1)
except (OSError, ValueError, KeyError):
return 0
def reference(self):
"""The primary screen's panel: remembered places are relative to it, so they move with
the screens' layout."""
return self.panel_get(self.primary() + 1)
def in_front(self):
"""A place in front of you, facing you, a little nearer than the primary screen (so text
looks as big as on the screens, at their density)."""
reply = self.screens.ask("head", quiet=True)
ref = self.reference()
if not reply.startswith("ok"):
return None
h = reply.split()
eye, heading = [float(v) for v in h[1:4]], float(h[4])
# (Within reach: you may have walked away from the screens since they were arranged.)
d = min(IN_FRONT, max(0.8, math.dist(eye, ref["center"]) - PULL_OUT)) if ref else IN_FRONT
fwd = ft_layout.turn_yaw((0.0, 0.0, -1.0), heading)
c = [eye[k] + fwd[k] * d for k in range(3)]
c[1] -= 0.1 * d # a little below eye level, like a screen
return c, ft_layout.turn_yaw((1.0, 0.0, 0.0), heading), (0.0, 1.0, 0.0), ft_layout.turn_yaw((0.0, 0.0, 1.0), heading)
def launch(self, app=None, argv=None):
"""Start an app (a desktop file name) or a command; its first window will float.
Returns (reply, pid or None)."""
if app:
app = app.removesuffix(".desktop")
info = Gio.DesktopAppInfo.new(app + ".desktop")
if info is None:
return f"error no app {app}", None
pids = []
try:
info.launch_uris_as_manager([], Gio.AppLaunchContext(), GLib.SpawnFlags.SEARCH_PATH,
None, None, lambda _info, pid, *_: pids.append(pid), None)
except GLib.Error as e:
return f"error {app}: {e.message}", None
self.launches.append(Launch(app, pids[0] if pids else None))
log(f"launched {app} (pid {pids[0] if pids else 'by D-Bus'})")
return "ok", (pids[0] if pids else None)
else:
try:
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)
except OSError as e:
return f"error {argv[0]}: {e}", None
GLib.child_watch_add(GLib.PRIORITY_DEFAULT, proc.pid, lambda *_: None) # reap it
self.launches.append(Launch("", proc.pid))
log(f"started {argv[0]} (pid {proc.pid})")
return "ok", proc.pid
def launched(self, ev):
"""Is this new window the one an app we started was to open? Takes it off the list."""
now = time.monotonic()
self.launches = [la for la in self.launches if la.until > now]
if not ev.get("normal") or ev.get("popup") or ev.get("transient"):
return None
chain = pid_chain(int(ev.get("pid") or 0))
app = ev.get("app", "")
for la in self.launches:
if any(p in chain for p in la.pids) or (la.app and app and la.app == app):
if not la.entries or len(la.entries) <= 1:
self.launches.remove(la)
return la
return None
def concealed(self):
"""The screens (1-based) hidden on their own (ft-layout hide N)."""
reply = self.screens.ask("concealed", quiet=True)
return {int(w) for w in reply.split()[1:] if w.isdigit()} if reply.startswith("ok") else set()
def on_hidden_screen(self, ev):
"""A new top-level window on a screen that's hidden on its own."""
m = re.match(r"WL-(\d+)$", ev.get("output", ""))
if not m or int(m.group(1)) >= self.screens_n:
return False
if not ev.get("normal") or ev.get("popup") or ev.get("transient") or ev.get("cls") == "ksplashqml":
return False
return int(m.group(1)) + 1 in self.concealed()
def float_launched(self, ev, launch):
"""Float a launched app's window (or one that opened on a hidden screen) where that app
last floated (its size too), or in front of you, at the primary screen's density."""
app = ev.get("app") or (launch.app if launch else "")
known = self.places.get(app) if app else None
ref = self.reference()
place = to_world(ref, known["rel"]) if known and ref and len(known.get("rel", [])) == 12 else self.in_front()
mpp = (known or {}).get("mpp") or self.screen_mpp(f"WL-{self.primary()}")
scale = None
if known and known.get("pixels"):
scale = known.get("scale") or output_scales().get(ev["output"], 1.0)
ev = dict(ev, frame=dict(ev["frame"], w=known["pixels"][0] / scale, h=known["pixels"][1] / scale))
self.float_window(dict(ev, app=app), place=place, mpp=mpp, scale=scale)
# ------------------------------------------------------------ profiles (docs/profiles.md)
def capture(self, sender):
""""windows": have the script report every window as it is now, and answer when the
report is in (or after 3 seconds with what we know)."""
if not sender:
return "error windows needs a reply address"
token = self.next_token
self.next_token += 1
self.captures[token] = sender
self.command(cmd="report-all", token=token)
GLib.timeout_add(3000, lambda: self.captured(token) and False)
return None # answered by captured
def captured(self, token):
sender = self.captures.pop(token, None)
if sender and self.sock:
try:
self.sock.sendto(("ok " + json.dumps(self.window_entries())).encode(), sender)
except OSError as e:
log(f"windows: {e}")
def window_entries(self):
"""Every app window and where it is, as a profile keeps it."""
ref = self.reference()
rects = None
out = []
for wid, ev in self.windows.items():
if not recordable(ev):
continue
entry = {"app": ev["app"]} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
if not entry.get("app") and not entry.get("cmd"):
continue
f = self.floats.get(wid)
if f:
g = self.panel_get(f.slot.index)
if not g or not ref or not f.frame:
continue
pixels = list(f.normal or (round(f.frame["w"] * f.scale), round(f.frame["h"] * f.scale)))
entry["float"] = {"rel": to_local(ref, g), "pixels": pixels, "scale": f.scale, "mpp": round(f.mpp, 8)}
else:
m = re.match(r"WL-(\d+)$", ev.get("output", ""))
if not m or int(m.group(1)) >= self.screens_n:
continue
fr, o = ev["frame"], ev.get("outputRect")
if not o:
rects = rects if rects is not None else output_rects()
x0, y0, _, _ = rects.get(ev["output"], (0, 0, 0, 0))
o = {"x": x0, "y": y0}
entry["screen"] = int(m.group(1)) + 1
entry["rect"] = [round(fr["x"] - o["x"]), round(fr["y"] - o["y"]), round(fr["w"]), round(fr["h"])]
if ev.get("maximized"):
entry["maximized"] = True
out.append(entry)
return out
def window_key(self, ev):
return ev.get("app") or json.dumps(cmdline(ev.get("pid")))
def open_profile(self, name):
"""Open a profile's apps (additive: nothing closes)."""
try:
profile = ft_layout.load_layout().get("profiles", {}).get(name)
except (OSError, ValueError):
profile = None
if profile is None:
return f"error no profile {name!r}"
groups = {}
for e in profile.get("windows", []):
key = e.get("app") or json.dumps(e.get("cmd") or [])
if key != "[]":
groups.setdefault(key, []).append(e)
claimed = set()
for key, entries in groups.items():
have = [ev for wid, ev in self.windows.items()
if wid not in claimed and recordable(ev) and self.window_key(ev) == key]
for e, ev in zip(entries, have):
claimed.add(ev["id"])
self.place_entry(ev, e)
rest = entries[len(have):]
if rest:
self.launch_entries(rest)
log(f"profile {name!r}: {sum(len(v) for v in groups.values())} windows, {len(claimed)} already open")
return "ok"
def launch_entries(self, entries):
"""Launch an app for a profile's entries that have no window yet."""
e = entries[0]
if e.get("app"):
reply, pid = self.launch(app=e["app"])
app = e["app"]
else:
reply, pid = self.launch(argv=e["cmd"])
app = ""
if not reply.startswith("ok"):
log(f"profile: {reply}")
return
la = self.launches[-1] # the one launch() just added
la.entries, la.argv = list(entries), (None if app else e["cmd"])
def relaunch(self, la):
"""3 seconds after a profile's app showed its first window: start it again for each
entry still waiting (an app that restores its own windows has shown them by now)."""
if la.relaunched or not la.entries or la not in self.launches:
return
la.relaunched = True
la.until = time.monotonic() + LAUNCH_SECONDS
for _ in la.entries:
if la.app:
reply, pid = self.launch(app=la.app)
else:
reply, pid = self.launch(argv=la.argv)
if reply.startswith("ok"):
self.launches.pop() # launch() added one of its own; this one waits for them all
if pid:
la.pids.append(pid)
def place_entry(self, ev, e):
"""Move a window to a profile entry's place: on a screen, or floating."""
f = self.floats.get(ev["id"])
if "float" in e:
fl, ref = e["float"], self.reference()
if not ref or len(fl.get("rel", [])) != 12:
return
place = to_world(ref, fl["rel"])
pixels, scale = fl.get("pixels"), fl.get("scale")
if f:
self.pose(f, *place)
if scale:
self.set_scale(f, scale)
if pixels and f.frame:
m = self.margin
self.command(cmd="geometry", id=f.id, x=f.slot.pos[0] + m / f.scale,
y=f.slot.pos[1] + m / f.scale, w=pixels[0] / f.scale, h=pixels[1] / f.scale)
return
if pixels:
scale = scale or output_scales().get(ev["output"], 1.0)
ev = dict(ev, frame=dict(ev["frame"], w=pixels[0] / scale, h=pixels[1] / scale))
else:
scale = None
self.float_window(ev, place=place, mpp=fl.get("mpp"), scale=scale)
return
n = int(e.get("screen", 1)) - 1
if not 0 <= n < self.screens_n:
n = 0
name = f"WL-{n}"
x0, y0, ow, oh = output_rects().get(name, (0, 0, 0, 0))
rx, ry, rw, rh = (e.get("rect") or [100, 100, ev["frame"]["w"], ev["frame"]["h"]])[:4]
if ow and oh: # keep it on the screen if the screen got smaller
rw, rh = min(rw, ow), min(rh, oh)
rx, ry = max(0, min(rx, ow - rw)), max(0, min(ry, oh - rh))
frame = {"x": x0 + rx, "y": y0 + ry, "w": rw, "h": rh}
if f:
self.dock(f, frame=frame, output=name, maximized=bool(e.get("maximized")))
else:
self.command(cmd="place", id=ev["id"], output=name, x=frame["x"], y=frame["y"], w=rw, h=rh,
maximized=bool(e.get("maximized")))
# ------------------------------------------------------------ popups and dialogs # ------------------------------------------------------------ popups and dialogs
def sub(self, ev): def sub(self, ev):
@@ -507,11 +1006,25 @@ class Daemon:
return s.window return s.window
return None return None
def request(self, text): def request(self, text, sender=None):
words = text.split() words = text.split()
if not words: if not words:
return "error empty" return "error empty"
cmd, rest = words[0], words[1:] cmd, rest = words[0], words[1:]
if cmd == "launch" and len(rest) == 1:
return self.launch(app=rest[0])[0]
if cmd == "run" and rest:
try:
argv = json.loads(text.split(None, 1)[1])
except ValueError:
return "error run takes a JSON list"
if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv):
return "error run takes a JSON list"
return self.launch(argv=argv)[0]
if cmd == "windows":
return self.capture(sender)
if cmd == "profile" and rest:
return self.open_profile(text.split(None, 1)[1])
if cmd == "list": if cmd == "list":
return "ok " + " ".join(f"{s.output}:{s.window.id if s.window else '-'}" for s in self.slots) return "ok " + " ".join(f"{s.output}:{s.window.id if s.window else '-'}" for s in self.slots)
if cmd == "quit": if cmd == "quit":
@@ -520,6 +1033,13 @@ class Daemon:
if cmd in ("float", "dock") and (not rest or rest[0] == "active"): if cmd in ("float", "dock") and (not rest or rest[0] == "active"):
self.command(cmd="request-active") self.command(cmd="request-active")
return "ok" return "ok"
if cmd == "float" and rest == ["pointer"]:
self.command(cmd="request-pointer")
return "ok"
if cmd == "dock" and rest == ["all"]:
for f in list(self.floats.values()):
self.dock(f)
return "ok"
if cmd in ("dock", "close", "resize", "scale") and rest and rest[0].isdigit(): if cmd in ("dock", "close", "resize", "scale") and rest and rest[0].isdigit():
f = self.by_panel(int(rest[0])) f = self.by_panel(int(rest[0]))
if not f: if not f:
@@ -539,7 +1059,8 @@ class Daemon:
if not ev: if not ev:
return f"error no window {rest[0]}" return f"error no window {rest[0]}"
if cmd == "float": if cmd == "float":
self.float_window(ev) # The script's word for where it is now: a window on a screen isn't followed here.
self.command(cmd="request-float", id=rest[0])
elif cmd == "dock" and rest[0] in self.floats: elif cmd == "dock" and rest[0] in self.floats:
self.dock(self.floats[rest[0]]) self.dock(self.floats[rest[0]])
elif cmd == "close": elif cmd == "close":
@@ -575,6 +1096,34 @@ class Service(dbus.service.Object):
self.daemon.wait(reply) self.daemon.wait(reply)
def watch_apps(daemon):
"""Keep Launch as Standalone's desktop file copies up to date (float/ft_apps.py), in a
session that reads them (the session script puts them in XDG_DATA_DIRS)."""
ours = os.path.realpath(ft_apps.OUT_ROOT)
if ours not in (os.path.realpath(d) for d in os.environ.get("XDG_DATA_DIRS", "").split(":") if d):
return
later = []
def write():
later.clear()
try:
log(f"Launch as Standalone: {ft_apps.write_all()} apps")
except OSError as e:
log(f"Launch as Standalone: {e}")
return False
def changed(*_):
if not later: # wait for an install or update to settle
later.append(GLib.timeout_add_seconds(3, write))
write()
daemon.app_monitors = []
for d in ft_apps.app_dirs():
if os.path.isdir(d):
m = Gio.File.new_for_path(d).monitor_directory(Gio.FileMonitorFlags.NONE, None)
m.connect("changed", changed)
daemon.app_monitors.append(m)
def main(): def main():
conf = ft_layout.read_conf() conf = ft_layout.read_conf()
p = argparse.ArgumentParser(description="Floating windows for the Frametop desktop") p = argparse.ArgumentParser(description="Floating windows for the Frametop desktop")
@@ -598,6 +1147,7 @@ def main():
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind("\0" + args.socket) sock.bind("\0" + args.socket)
sock.setblocking(False) sock.setblocking(False)
daemon.sock = sock
def readable(*_): def readable(*_):
while True: while True:
@@ -605,8 +1155,8 @@ def main():
data, sender = sock.recvfrom(4096) data, sender = sock.recvfrom(4096)
except BlockingIOError: except BlockingIOError:
return True return True
reply = daemon.request(data.decode(errors="replace").strip()) reply = daemon.request(data.decode(errors="replace").strip(), sender)
if sender: if sender and reply is not None:
try: try:
sock.sendto(reply.encode(), sender) sock.sendto(reply.encode(), sender)
except OSError: except OSError:
@@ -615,6 +1165,7 @@ def main():
log(f"{args.screens} screens, {args.slots} floating slots (WL-{args.screens} and up), margin {args.margin} px") log(f"{args.screens} screens, {args.slots} floating slots (WL-{args.screens} and up), margin {args.margin} px")
daemon.load_script(bus) daemon.load_script(bus)
watch_apps(daemon)
daemon.loop.run() daemon.loop.run()
+18 -11
View File
@@ -5,15 +5,17 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on. - `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log. - `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below. - `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments. - `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
``` ```
gaze/build.sh # build ft-gaze gaze/run.sh install # the gaze service: builds ft-gaze and the panel, starts with SteamVR
gaze/probe/install.sh # build, and add Frametop Gaze Probe to the app menu
gaze/probe/ft-gazeprobe --screen 1
gaze/run.sh install # the gaze service, with SteamVR
gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse alone) gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse alone)
gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for sudo) gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for sudo)
gaze/build.sh # build ft-gaze and the panel by hand
gaze/probe/install.sh # development: build, and add Frametop Gaze Probe to the app menu
gaze/probe/ft-gazeprobe --screen 1
``` ```
## Gaze pointer ## Gaze pointer
@@ -21,14 +23,17 @@ gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for su
Gaze as an input method for the whole desktop, without replacing anything of SteamVR's: Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes: - `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
- **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: calibrate it in the probe with the tracker toggle on Own tracker. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either. - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
- **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost. - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery. With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset. Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click. - The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look, and the mouse does the last bit. By default the mouse moves it only while a button is held (see "The mouse only corrects" below). With `POINTER_GAZE_MOUSE_MOVE=free`, moving the mouse takes the pointer, from where the gaze put it, and looking well away (5 degrees) gives it back to the gaze. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse and the keyboard, not the controllers ([docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why).
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do the probe's check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was. Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
@@ -40,7 +45,7 @@ Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction,
| SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. | | SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. | | mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. |
| Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. | | Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
| Own tracker | Our own tracker (`tracker/`, experimental; see "Our own eye tracker"). It keeps its own calibration, not SteamVR's: the probe's calibration with the tracker toggle on Own tracker fits it (its dots go out to the Calibration ring angle each way, on an oval, since the fit goes wrong past its dots), and practice clicks teach it how far the headset has moved on your face since. After the headset was off, the probe first asks for one look at a centre dot, which resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it, and asks the gaze service to keep the tracker running. The gaze pointer can use it too (Eye tracker: Own tracker, on the Gaze page of Frametop Input Settings). ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. | | Own tracker | Our own tracker (`tracker/`, experimental; see "Our own eye tracker"). It keeps its own calibration, not SteamVR's: Calibrate on the Gaze page fits it while Eye tracker is Own tracker (eight dots to a ring instead of six, on a slight oval, since the fit goes wrong past its dots; the probe's calibration with its tracker toggle on Own tracker does the same), and clicks teach it how far the headset has moved on your face since. After the headset was off, one look at a centre dot (the quick check, or the probe's first dot) resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it, and asks the gaze service to keep the tracker running. The gaze pointer can use it too (Eye tracker: Own tracker, on the Gaze page of Frametop Input Settings). ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over. The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over.
@@ -50,7 +55,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). - `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too).
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`. - `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones run on a PC through `frame-job` (`gaze/tracker/.frame-job`). `lab/py` runs them with that Python (in the dev container on the Frame; frame-job's setup makes the same venv on the PC). - `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
Ground rules, for anyone changing it: Ground rules, for anyone changing it:
@@ -61,12 +66,14 @@ Ground rules, for anyone changing it:
## Headset fit ## Headset fit
The probe's Headset fit mode (Check headset fit on the Gaze page, or `ft-gazeprobe --mode fit`) shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it. Check headset fit on the Gaze page opens it in the headset panel: a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and the hints, live while you adjust the headset. A left click or Meta+J runs the guided check (dots, then looks down, up, left and right), and a right click or Meta+K closes it. The probe's Headset fit mode (`ft-gazeprobe --mode fit`, for development) has the same check with maps: it shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
Losing an eye is usually about where you look, not the tracker. On this Frame the left eye was lost 57 to 64 % of the time looking 30 to 50 degrees down (at the keyboard) and the right eye never; at screen height both were seen over 98 % of the time. Looking down, the lids come down over the eyes. That's harmless, since the gaze service ignores looks down past the screens: they show on the maps, but not in the counts or as a problem. Losing an eye is usually about where you look, not the tracker. On this Frame the left eye was lost 57 to 64 % of the time looking 30 to 50 degrees down (at the keyboard) and the right eye never; at screen height both were seen over 98 % of the time. Looking down, the lids come down over the eyes. That's harmless, since the gaze service ignores looks down past the screens: they show on the maps, but not in the counts or as a problem.
## Probe ## Probe
The probe is a development tool (in the Gaze page's overflow menu): calibration experiments, accuracy tests, and practice modes. Users calibrate and check in the headset panel instead.
The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The arrow in the panel's title bar collapses it to just that bar, so the dot and targets behind it stay visible; the collapsed bar stays through tests. The keys do the same (Tab, C, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place. The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The arrow in the panel's title bar collapses it to just that bar, so the dot and targets behind it stay visible; the collapsed bar stays through tests. The keys do the same (Tab, C, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
- **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots. - **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
+9 -2
View File
@@ -1,5 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Build ft-gaze in the dev container on the Frame (gaze/build/ft-gaze; it also runs there). # Build ft-gaze and the calibration panel ft-gazepanel in the dev container on the Frame
# (gaze/build/; they also run there). The panel draws its text with stb_truetype (public
# domain, one header, pinned as in screens/build.sh).
# The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header # The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header
# shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches. # shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches.
set -euo pipefail set -euo pipefail
@@ -10,4 +12,9 @@ openvr=v2.15.6
[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; } [ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \ g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \
-o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread -o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
echo "built build/ft-gaze"' stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
-o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
$(pkg-config --libs gbm libdrm) -lpthread
echo "built build/ft-gaze build/ft-gazepanel"'
+2 -2
View File
@@ -1,5 +1,5 @@
"""fitcheck: how well the eye tracker sees each eye, for fitting the headset (ft-gazeprobe's """fitcheck: how well the eye tracker sees each eye, for fitting the headset (the headset
Headset fit mode). panel's fit check, gazecheck.py, and ft-gazeprobe's Headset fit mode).
From each ft-gaze sample it takes, per eye, whether the tracker has that eye (its variance From each ft-gaze sample it takes, per eye, whether the tracker has that eye (its variance
for the eye's direction, "unc", under EYE_LOST; see gazecal), how open the eye is, and the for the eye's direction, "unc", under EYE_LOST; see gazecal), how open the eye is, and the
+21 -1
View File
@@ -1,5 +1,5 @@
// ft-gaze: the headset's eye tracking as rays and Frametop screen pixels (OpenVR overlay // ft-gaze: the headset's eye tracking as rays and Frametop screen pixels (OpenVR overlay
// client, runs in the dev container). An experiment for gaze input; ft-gazeprobe reads it. // client, runs in the dev container). The gaze service (ft-gazed) and the gaze probe run it.
// //
// Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source // Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source
// hit-tested against the Frametop screens: // hit-tested against the Frametop screens:
@@ -461,6 +461,9 @@ int main(int argc, char **argv) {
vr::VRActionSetHandle_t set = vr::k_ulInvalidActionSetHandle; vr::VRActionSetHandle_t set = vr::k_ulInvalidActionSetHandle;
input->GetActionHandle("/actions/gaze/in/gaze", &gaze); input->GetActionHandle("/actions/gaze/in/gaze", &gaze);
input->GetActionSetHandle("/actions/gaze", &set); input->GetActionSetHandle("/actions/gaze", &set);
if (me == vr::VRInputError_None)
std::fprintf(stderr, "ft-gaze: action manifest %s: ok\n", manifest.c_str());
else
std::fprintf(stderr, "ft-gaze: action manifest %s: error %d\n", manifest.c_str(), int(me)); std::fprintf(stderr, "ft-gaze: action manifest %s: error %d\n", manifest.c_str(), int(me));
EyeFile eyes; EyeFile eyes;
@@ -476,9 +479,24 @@ int main(int argc, char **argv) {
double lastEmit = 0; double lastEmit = 0;
int actionErrors = 0; int actionErrors = 0;
vr::EVRInputError lastActionError = vr::VRInputError_None; vr::EVRInputError lastActionError = vr::VRInputError_None;
// During a VR game, SteamVR's gaze action is left alone. With ft-gaze reading the eyes, SteamVR
// restarted its eye tracker every 10 s or so in a game, as if the headset came off, and each
// restart took input focus from the game: Beat Saber paused (PR #13). Of what ft-gaze reads,
// only the action reaches SteamVR (the mmap and our tracker are files), so gaze still works
// over the dashboard. Games are told apart the way ft-screens does it, by the scene app.
bool inGame = false;
double nextGameCheck = 0;
while (true) { while (true) {
const double now = NowRaw(); const double now = NowRaw();
if (now >= nextGameCheck) {
nextGameCheck = now + 0.5;
const bool game = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
if (game != inGame)
std::fprintf(stderr, "ft-gaze: %s\n",
game ? "a VR game is running: SteamVR's gaze action left alone" : "the VR game ended");
inGame = game;
}
vr::TrackedDevicePose_t hp; vr::TrackedDevicePose_t hp;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1); sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1);
if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking); if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking);
@@ -499,6 +517,7 @@ int main(int argc, char **argv) {
// SteamVR's action: a room-space origin and fixation point, turned into the head // SteamVR's action: a room-space origin and fixation point, turned into the head
// frame so every source reports the same kind of angles. // frame so every source reports the same kind of angles.
std::string action = "{\"ok\":0}"; std::string action = "{\"ok\":0}";
if (!inGame) {
vr::VRActiveActionSet_t active{}; vr::VRActiveActionSet_t active{};
active.ulActionSet = set; active.ulActionSet = set;
active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin; active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
@@ -518,6 +537,7 @@ int main(int argc, char **argv) {
int(e.bValid)); int(e.bValid));
lastActionError = ae; lastActionError = ae;
} }
}
std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null"; std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null";
if (haveMmap) { if (haveMmap) {
+10 -1
View File
@@ -36,8 +36,17 @@ def main():
if r is None: if r is None:
sys.exit("the pointer helper isn't running") sys.exit("the pointer helper isn't running")
print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)") print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
if ask("ft_gazed", "status", 0.3) is None: st = ask("ft_gazed", "status", 0.3)
if st is None:
print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow") print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
elif r == "ok on":
try:
calibrated = (json.loads(st).get("checks") or {}).get("calibrated")
except ValueError:
calibrated = None
if calibrated is False:
print("note: no calibration for this eye tracker yet; it opens in the headset "
"(if it can't, ft-gazectl status says why: checks.problem)")
elif cmd in ("status", "forget", "reload"): elif cmd in ("status", "forget", "reload"):
r = ask("ft_gazed", cmd) r = ask("ft_gazed", cmd)
if r is None: if r is None:
+76 -25
View File
@@ -3,8 +3,12 @@
Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
again when the file changes: again when the file changes:
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes, GAZE_TRACKER=auto|own|steam
ft-gaze's source "own"; this service runs it, see below) our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own";
this service runs it, see below) or SteamVR's. auto (the default)
is ours when it's installed (gaze/tracker/install.sh: the frame
grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
switches when ours is installed or removed.
GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
off it was at your recent nudges; left or right counts that eye twice off it was at your recent nudges; left or right counts that eye twice
as much as the other. Either eye alone carries the gaze when the as much as the other. Either eye alone carries the gaze when the
@@ -44,9 +48,11 @@ click, it sends "lesson <raw yaw> <raw pitch> <true yaw> <true pitch>": where th
was when the mouse took over, and where the pointer was when you clicked (you were looking was when the mouse took over, and where the pointer was when you clicked (you were looking
there). The gap is the tracker's error there. The raw gaze is the one sent, so it also says there). The gap is the tracker's error there. The raw gaze is the one sent, so it also says
when that look was (the history of what was sent), and so what each eye read then: when that look was (the history of what was sent), and so what each eye read then:
- SteamVR: each eye learns its own error, unless the gaze was more than LESSON_MAX degrees - SteamVR: each eye learns its own error, unless the gaze was more than
POINTER_GAZE_NUDGE_MAX degrees (frametop.conf, 55 by default: the helper's limit too)
past the correction (then it wasn't a nudge onto what you looked at); past the correction (then it wasn't a nudge onto what you looked at);
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @ft_eyes), as the - our tracker: unless it was more than POINTER_GAZE_NUDGE_MAX off, the look goes to it as
a click ("click T YAW PITCH" on @ft_eyes), as the
probe's clicks do, and it learns how far the headset has moved on your face. That's probe's clicks do, and it learns how far the headset has moved on your face. That's
what it gets wrong, and after the headset was off, the first click resets it; what it gets wrong, and after the headset was off, the first click resets it;
- either way, how far off each eye was (before the lesson taught it anything) goes to the - either way, how far off each eye was (before the lesson taught it anything) goes to the
@@ -61,16 +67,27 @@ GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the prob
reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh), reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
which copies them only while ft-eyes runs. which copies them only while ft-eyes runs.
Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
our tracker asks for a click, and the full calibration when gaze mode comes on without one,
both in a panel fixed to the headset (gaze/panel/ft-gazepanel, which this service runs too).
Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
tracker's shared memory read-only. tracker's shared memory read-only.
Control socket: abstract unix datagram "@ft_gazed": Control socket: abstract unix datagram "@ft_gazed":
lesson <rhy> <rhp> <thy> <thp> from the pointer helper (see above) lesson <rhy> <rhp> <thy> <thp> from the pointer helper (see above)
recheck <deg> from the pointer helper: a click's correction was past
POINTER_GAZE_NUDGE_MAX, so the quick check (gazecheck.py)
status reply: one JSON object status reply: one JSON object
forget drop what the lessons taught (the calibration stays) forget drop what the lessons taught (the calibration stays)
reload read calibration.json and the settings again reload read calibration.json and the settings again
eyes <seconds> keep our own tracker running that much longer (at most 120), eyes <seconds> keep our own tracker running that much longer (at most 120),
whatever GAZE_TRACKER says: the probe's lease. Reply: "ok" whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
quickcal the one-dot check now (the calibration if there's none)
calibrate the full calibration in the panel
fitcheck the headset fit check in the panel
calaccept | calquit from the pointer helper while the panel is up: take this dot
now (a left click, Meta+J) | close it (a right click, Meta+K)
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -96,6 +113,7 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent)) sys.path.insert(0, str(Path(__file__).resolve().parent))
from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402 from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402
EyeWeights, Fixation, LiveCorrection, SteamEyeLog) EyeWeights, Fixation, LiveCorrection, SteamEyeLog)
from gazecheck import Checks # noqa: E402
REPO = Path(__file__).resolve().parents[1] REPO = Path(__file__).resolve().parents[1]
HELPER = REPO / "gaze" / "build" / "ft-gaze" HELPER = REPO / "gaze" / "build" / "ft-gaze"
@@ -105,6 +123,7 @@ EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker
EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh) EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh)
EYES_SOCKET = "\0ft_eyes" # its control socket EYES_SOCKET = "\0ft_eyes" # its control socket
EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service) EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service)
EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service")) # tracker/install.sh
CONF = Path.home() / ".config" / "frametop.conf" CONF = Path.home() / ".config" / "frametop.conf"
CALIBRATION = STATE / "calibration.json" CALIBRATION = STATE / "calibration.json"
LESSONS = STATE / "pointer-lessons.json" LESSONS = STATE / "pointer-lessons.json"
@@ -112,9 +131,9 @@ LESSON_LOG = STATE / "pointer-lessons.jsonl"
SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here
SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
TRACKERS = ("steam", "own") TRACKERS = ("steam", "own")
TRACKER_SETTINGS = ("auto",) + TRACKERS
BIASES = ("auto", "left", "right") BIASES = ("auto", "left", "right")
LESSON_MAX = 8.0 # degrees past the correction NUDGE_MAX = 55.0 # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
OWN_LESSON_MAX = 25.0 # our tracker: after the headset was off, its first clicks can be 10-17 off
HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later) HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
LOOK = 0.3 # seconds of samples before that moment make the look (the probe's fixation) LOOK = 0.3 # seconds of samples before that moment make the look (the probe's fixation)
RETRY = 3.0 # seconds before starting ft-gaze again RETRY = 3.0 # seconds before starting ft-gaze again
@@ -140,8 +159,14 @@ def log(msg):
print(f"ft-gazed: {msg}", file=sys.stderr, flush=True) print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
def own_installed():
"""Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
def read_settings(): def read_settings():
"""(tracker, eye bias) from frametop.conf, defaults for anything missing or unknown.""" """(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything
missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed."""
conf = {} conf = {}
try: try:
for line in CONF.read_text().splitlines(): for line in CONF.read_text().splitlines():
@@ -151,9 +176,15 @@ def read_settings():
conf[k.strip()] = v.strip().lower() conf[k.strip()] = v.strip().lower()
except OSError: except OSError:
pass pass
tracker = conf.get("GAZE_TRACKER", "steam") setting = conf.get("GAZE_TRACKER", "auto")
setting = setting if setting in TRACKER_SETTINGS else "auto"
tracker = setting if setting != "auto" else "own" if own_installed() else "steam"
bias = conf.get("GAZE_EYE", "auto") bias = conf.get("GAZE_EYE", "auto")
return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto" try:
nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0) # the helper's range
except ValueError:
nudge = NUDGE_MAX
return tracker, setting, bias if bias in BIASES else "auto", nudge
def mtime(path): def mtime(path):
@@ -168,7 +199,7 @@ class Service:
self.override, self.verbose, self.to = source, verbose, to self.override, self.verbose, self.to = source, verbose, to
self.source = source or "mmap1" # the older path's source self.source = source or "mmap1" # the older path's source
STATE.mkdir(parents=True, exist_ok=True) STATE.mkdir(parents=True, exist_ok=True)
self.tracker, self.bias = read_settings() self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings()
self.conf_mtime = mtime(CONF) self.conf_mtime = mtime(CONF)
self.models = {name: Correction() for name in SOURCES} self.models = {name: Correction() for name in SOURCES}
self.mode = DEFAULT_MODEL self.mode = DEFAULT_MODEL
@@ -209,6 +240,7 @@ class Service:
self.sel = selectors.DefaultSelector() self.sel = selectors.DefaultSelector()
self.sel.register(self.sock, selectors.EVENT_READ, "control") self.sel.register(self.sock, selectors.EVENT_READ, "control")
self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own") self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own")
self.checks = Checks(self, self.sel)
self.proc = None self.proc = None
self.buf = b"" self.buf = b""
self.restart_at = 0.0 self.restart_at = 0.0
@@ -228,9 +260,12 @@ class Service:
def load_settings(self): def load_settings(self):
self.conf_mtime = mtime(CONF) self.conf_mtime = mtime(CONF)
tracker, bias = read_settings() tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
if (tracker, bias) != (self.tracker, self.bias): if (tracker, bias) != (self.tracker, self.bias):
log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else "")) auto = ""
if self.tracker_setting == "auto":
auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
self.tracker, self.bias = tracker, bias self.tracker, self.bias = tracker, bias
for w in self.weights.values(): for w in self.weights.values():
w.bias = bias w.bias = bias
@@ -273,6 +308,13 @@ class Service:
tmp.replace(LESSONS) tmp.replace(LESSONS)
self.dirty = False self.dirty = False
def forget_lessons(self):
"""Drop what the lessons taught (the calibration stays)."""
self.lives = {name: PointerLessons() for name in SOURCES}
self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
self.refit()
self.save_lessons()
def refit(self): def refit(self):
for name, live in self.lives.items(): for name, live in self.lives.items():
live.wear_time = self.steam.worn() live.wear_time = self.steam.worn()
@@ -304,8 +346,8 @@ class Service:
cy, cp = self.correction(self.source, rhy, rhp) cy, cp = self.correction(self.source, rhy, rhp)
left = math.hypot(dy - cy, dp - cp) left = math.hypot(dy - cy, dp - cp)
rec.update(source=self.source, model=self.mode, correction=[cy, cp], lesson_deg=left) rec.update(source=self.source, model=self.mode, correction=[cy, cp], lesson_deg=left)
if left > LESSON_MAX: if left > self.nudge_max:
rec["refused"] = f"more than {LESSON_MAX} deg past the correction" rec["refused"] = f"more than {self.nudge_max:g} deg past the correction"
else: else:
self.lives[self.source].add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp, self.lives[self.source].add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp,
"wy": 1.0, "wp": 1.0, "how": "pointer"}, self.models[self.source], self.mode) "wy": 1.0, "wp": 1.0, "how": "pointer"}, self.models[self.source], self.mode)
@@ -316,11 +358,10 @@ class Service:
weights = self.weights[self.tracker if kind == "own" else "steam"] weights = self.weights[self.tracker if kind == "own" else "steam"]
rec.update(tracker=self.tracker if kind == "own" else "steam", bias=self.bias, lesson_deg=left, look_t=t, rec.update(tracker=self.tracker if kind == "own" else "steam", bias=self.bias, lesson_deg=left, look_t=t,
eyes=eyes, weights=[round(w, 3) for w in weights.weights()]) eyes=eyes, weights=[round(w, 3) for w in weights.weights()])
limit = OWN_LESSON_MAX if kind == "own" else LESSON_MAX
if t is None: if t is None:
rec["refused"] = "that gaze isn't in the last few seconds sent" rec["refused"] = "that gaze isn't in the last few seconds sent"
elif left > limit: elif left > self.nudge_max:
rec["refused"] = f"more than {limit} deg off" rec["refused"] = f"more than {self.nudge_max:g} deg off"
if "refused" in rec: if "refused" in rec:
return self.taken(rec) return self.taken(rec)
if kind == "own": if kind == "own":
@@ -505,6 +546,7 @@ class Service:
return low return low
def on_sample(self, s): def on_sample(self, s):
self.checks.on_sample(s)
kind = self.kind kind = self.kind
if kind != self.last_kind: if kind != self.last_kind:
log({"own": "our own tracker", "eyes": "SteamVR's eyes, each calibrated", log({"own": "our own tracker", "eyes": "SteamVR's eyes, each calibrated",
@@ -636,6 +678,7 @@ class Service:
if words[:1] == ["lesson"] and len(words) == 5: if words[:1] == ["lesson"] and len(words) == 5:
try: try:
rec = self.lesson(*map(float, words[1:])) rec = self.lesson(*map(float, words[1:]))
self.checks.after_lesson(rec)
reply = "refused" if "refused" in rec else f"ok {rec['lesson_deg']:.2f}" reply = "refused" if "refused" in rec else f"ok {rec['lesson_deg']:.2f}"
log(f"lesson {rec['lesson_deg']:.2f} deg at {rec['raw'][0]:+.1f},{rec['raw'][1]:+.1f}" log(f"lesson {rec['lesson_deg']:.2f} deg at {rec['raw'][0]:+.1f},{rec['raw'][1]:+.1f}"
+ (f", eyes off {', '.join('-' if m is None else f'{m:.2f}' for m in rec['miss'])}" + (f", eyes off {', '.join('-' if m is None else f'{m:.2f}' for m in rec['miss'])}"
@@ -646,11 +689,10 @@ class Service:
elif words[:1] == ["status"]: elif words[:1] == ["status"]:
reply = json.dumps(self.status()) reply = json.dumps(self.status())
elif words[:1] == ["forget"]: elif words[:1] == ["forget"]:
self.lives = {name: PointerLessons() for name in SOURCES} self.forget_lessons()
self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
self.refit()
self.save_lessons()
reply = "ok" reply = "ok"
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "calaccept", "calquit", "recheck"):
reply = self.checks.command(words)
elif words[:1] == ["eyes"] and len(words) == 2: elif words[:1] == ["eyes"] and len(words) == 2:
try: try:
secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX) secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX)
@@ -708,12 +750,13 @@ class Service:
st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"), st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
lessons=max(len(m) for m in w.misses), own_running=bool(own), lessons=max(len(m) for m in w.misses), own_running=bool(own),
own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values())) own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists()) st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
tracker_setting=self.tracker_setting, own_installed=own_installed())
st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2) st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
if self.last_sample else None, "headset_on": self.steam.wearing(), if self.last_sample else None, "headset_on": self.steam.wearing(),
"headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None, "headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
"eyes_lost": self.lost, "fallback_ready": [self.fallback.ready(0), self.fallback.ready(1)], "eyes_lost": self.lost, "fallback_ready": [self.fallback.ready(0), self.fallback.ready(1)],
**self.counts}) "checks": self.checks.status(), **self.counts})
return st return st
def periodic(self): def periodic(self):
@@ -724,7 +767,8 @@ class Service:
if mtime(CALIBRATION) != self.cal_mtime: if mtime(CALIBRATION) != self.cal_mtime:
self.load_calibration() self.load_calibration()
self.refit() self.refit()
if mtime(CONF) != self.conf_mtime: if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
and own_installed() != (self.tracker == "own")):
self.load_settings() self.load_settings()
want = self.eyes_wanted() want = self.eyes_wanted()
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at: if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
@@ -737,6 +781,7 @@ class Service:
self.eyes_sock.sendto(b"status", EYES_SOCKET) self.eyes_sock.sendto(b"status", EYES_SOCKET)
except OSError: except OSError:
pass # not up yet: status() says so once the last answer is old pass # not up yet: status() says so once the last answer is old
self.checks.periodic()
if self.dirty: if self.dirty:
self.save_lessons() self.save_lessons()
@@ -747,9 +792,13 @@ class Service:
now = time.monotonic() now = time.monotonic()
if not self.proc and now >= self.restart_at: if not self.proc and now >= self.restart_at:
self.start_helper() self.start_helper()
for key, _ in self.sel.select(timeout=0.5): for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5):
if key.data == "control": if key.data == "control":
self.on_control() self.on_control()
elif key.data == "checks":
self.checks.on_readable()
elif key.data == "panel" and self.checks.panel_proc:
self.checks.read_panel()
elif key.data == "own": elif key.data == "own":
self.on_own() self.on_own()
elif key.data == "eyes" and self.eyes_proc: elif key.data == "eyes" and self.eyes_proc:
@@ -758,12 +807,14 @@ class Service:
self.read_stdout() self.read_stdout()
elif key.data == "stderr" and self.proc: elif key.data == "stderr" and self.proc:
self.read_stderr() self.read_stderr()
self.checks.tick()
if now >= next_periodic: if now >= next_periodic:
self.periodic() self.periodic()
next_periodic = now + 1.0 next_periodic = now + 1.0
if self.verbose and now >= next_verbose: if self.verbose and now >= next_verbose:
log(json.dumps(self.status())) log(json.dumps(self.status()))
next_verbose = now + 5 next_verbose = now + 5
self.checks.stop()
self.stop_helper() self.stop_helper()
self.stop_eyes() self.stop_eyes()
if self.dirty: if self.dirty:
+31 -10
View File
@@ -544,6 +544,9 @@ class SteamEyeLog:
each time the headset goes on ("HMD on"): the eye model starts over then too.""" each time the headset goes on ("HMD on"): the eye model starts over then too."""
PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt" PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt"
# It writes "HMD on" again every minute or so while on, and flickers off for 0.01-0.3 s:
# only an on after an off of BLIP or longer counts.
BLIP = 1.5
def __init__(self): def __init__(self):
self.pos = 0 self.pos = 0
@@ -595,8 +598,13 @@ class SteamEyeLog:
self.starts.append(t) self.starts.append(t)
restarted = not first restarted = not first
elif "HMD on" in line: elif "HMD on" in line:
off = bool(self.offs) and (not self.wears or self.offs[-1] > self.wears[-1])
if off and self.wears and t - self.offs[-1] < self.BLIP:
self.offs.pop() # the sensor flickering: it never came off
elif off or not self.wears:
self.wears.append(t) self.wears.append(t)
elif "HMD off" in line: elif "HMD off" in line:
if not self.offs or (self.wears and self.wears[-1] > self.offs[-1]):
self.offs.append(t) self.offs.append(t)
elif "Accept usercal" in line: elif "Accept usercal" in line:
self.accepts.append(t) self.accepts.append(t)
@@ -632,31 +640,44 @@ def cross_validate(points, mode):
return errs return errs
def steady_samples(samples, vergence_jump=1.5): def steady_samples(samples, vergence_jump=1.5, why=None):
"""The samples of one look at one spot where the tracker had both eyes: none in a blink """The samples of one look at one spot where the tracker had both eyes: none in a blink
(openness under half its median over the samples), none where it had lost an eye (its (openness under half its median over the samples), none where it had lost an eye (its
variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
vergence) is more than `vergence_jump` degrees from its median over the samples. The vergence) is more than `vergence_jump` degrees from its median over the samples. The
vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
fraction of one far off), so only a jump away from what it was during this look means fraction of one far off), so only a jump away from what it was during this look means
the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.""" the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
`why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
"lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
if why is None:
why = {}
# Openness: a blink is a sharp drop from what it was during this look. Not a fixed # Openness: a blink is a sharp drop from what it was during this look. Not a fixed
# level: looking down, the upper lids come down with the eyes, and in bright light you # level: looking down, the upper lids come down with the eyes, and in bright light you
# squint, so the reading can stay under 0.5 for the whole look while the tracker follows # squint, so the reading can stay under 0.5 for the whole look while the tracker follows
# the eyes fine (a calibration dot at the bottom of the bright round failed that way). # the eyes fine (a calibration dot at the bottom of the bright round failed that way).
opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o] opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12 floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
opened = [] seen = []
for smp in samples: for smp in samples:
o = (smp["src"].get("mmap1") or {}).get("open") m1 = smp["src"].get("mmap1") or {}
if not o or min(o) >= floor: o = m1.get("open")
opened.append(smp) lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
# A lost eye's openness reads 0 too, so a lost eye is named before a blink.
key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
"blink" if o and min(o) < floor else None
if key:
why[key] = why.get(key, 0) + 1
else:
seen.append(smp)
def vergence(smp): def vergence(smp):
return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr")) return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST] have = [v for v in map(vergence, seen) if v is not None]
have = [v for v in map(vergence, opened) if v is not None]
if len(have) < 5: if len(have) < 5:
return opened return seen
med = statistics.median(have) med = statistics.median(have)
return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump] kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
if len(kept) < len(seen):
why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept)
return kept
+858
View File
@@ -0,0 +1,858 @@
"""gazecheck: the gaze service's checks and calibration, in the panel fixed to the headset
(gaze/panel/ft-gazepanel; ft-gazed runs it). Every kind is made of dots shown at head-relative
directions: look at each one.
quick one dot in the middle of your view. It opens when the headset goes on: eyes seen
for DON_DELAY after none for AWAY_MIN (SteamVR's tracker's variance for an eye under
EYE_LOST). SteamVR's "HMD on" can't say: it repeats every minute or so, and it can
stay on for hours with nobody in the headset. It also opens when our own tracker asks
for a click (its "reseat": the headset may sit differently on your face now), at most
once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse
button or key combination mapped to Gaze quick check), and when a click's correction
was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is
far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing.
five the middle and four around it, when the first FIVE_COUNT lessons after a quick check
were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
size, and the tracker's error with it, changes with brightness), each the middle and
a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
whenever gaze mode is on without a calibration for the tracker in use and someone's
in the headset, and on "calibrate". One that closes unfinished (ignored, too few
dots) opens again only once the headset comes off and on, or gaze mode off and on.
Frametop's screens hide while it runs. Quitting it while there's still no
calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
Why gaze mode can't work yet goes in the status ("problem"), for Input Settings.
fit the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
seen) and hints, from the gaze probe's Headset fit (gaze/fitcheck.py), while you
adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
works however far off the tracker is; a left click or Meta+J (the pointer helper's
"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
renewed every second; the helper shows it again by itself when that stops).
What a capture teaches:
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
calib-start, a calib-point for each dot, calib-fit (its calibration)
SteamVR's quick and five: a lesson for each eye, like the pointer's; full: each source's
calibration fitted from the dots (gazecal.Correction, the probe's way), saved
to calibration.json, and the lessons start over on top of it
Either way, how far off each eye was goes to the eye bias, and each capture to the lesson log.
"""
import json
import math
import os
import selectors
import signal
import socket
import statistics
import subprocess
import sys
import time
from pathlib import Path
from fitcheck import MIN_REGION, FitCheck, wrap
from gazecal import DEFAULT_MODEL, EYE_LOST, STATE, steady_samples
REPO = Path(__file__).resolve().parents[1]
PANEL_PROG = REPO / "gaze" / "build" / "ft-gazepanel"
PANEL = "\0ft_gazepanel"
POINTER = "\0ft_pointer_helper"
SCREENS = "\0ft_screens"
EYES = "\0ft_eyes"
CONF = Path.home() / ".config" / "frametop.conf"
CALIBRATION = STATE / "calibration.json"
POINTS = STATE / "points.jsonl" # the probe's record of calibration dots, for its refine
CHECK_LOG = STATE / "checks.jsonl"
CHECK_SETTLE = 0.45
CHECK_WINDOW = 0.6
CHECK_SPREAD = 1.0 # degrees
ACCEPT_SPREAD = 2.5 # degrees: a capture asked for (calaccept) takes this much
CLICK_IDLE = 120.0 # seconds a dot of five or full waits for its click; then the check closes
QUICK_TIMEOUT = 6.0
QUICK_COOLDOWN = 120.0
DON_DELAY = 3.0 # seconds of eyes after AWAY_MIN without: the headset went on
AWAY_MIN = 3.0
EYES_GONE = 2.0 # seconds without eyes that close a check: the headset came off
FIVE_LIMIT = 2.0
FIVE_COUNT = 3
DONE_PAUSE = 0.35 # seconds the filled dot shows before the next
RING = 20.0
ROUND_BG = (0.03, 0.33, 0.8)
ROUND_NAMES = ("dark", "medium", "bright")
RING_SCALE = (1.0, 0.5, 1.0)
PANEL_RETRY = 10.0
FULL_RETRY = 10.0 # seconds before an automatic calibration that failed to start tries again
FIT_TIMEOUT = 300.0 # seconds the fit check stays up
FIT_EVERY = 0.5 # seconds between its cards' updates (each is a new picture for the panel)
FIT_HINT_WIDTH = 95 # characters a hint line holds in the panel
def log(msg):
print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
def check_dots(kind, own):
"""(yaw, pitch, round): head-relative degrees, yaw +left, pitch +up."""
if kind == "quick":
return [(0.0, 0.0, 0)]
if kind == "five":
return [(0.0, 0.0, 0), (12.0, 0.0, 0), (-12.0, 0.0, 0), (0.0, 9.0, 0), (0.0, -9.0, 0)]
out = []
n = 8 if own else 6
for rnd in range(3):
out.append((0.0, 0.0, rnd))
r = RING * RING_SCALE[rnd]
for i in range(n):
a = math.radians(-90 + rnd * 20 + i * 360 / n) # as the probe: x right, y down
x, y = r * math.cos(a), r * math.sin(a) * (0.9 if own else 1.0)
out.append((-x, -y, rnd))
return out
# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long):
# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the
# headset's fit causes, in the calibration's panel.
STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
"lost_right": ("right eye lost", "SteamVR lost your right eye"),
"lost_both": ("both eyes lost", "SteamVR lost both eyes"),
"blink": ("blinked", "you blinked"),
"vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
FIT_HINT = "check the headset fit"
ACCEPT_WAIT = 1.5 # seconds after a click with no capture before the panel says what it waits for
def reject_reason(reply, why):
"""Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's
drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause."""
if reply is None:
if not why:
return "no reading", "SteamVR sent no reading for that look", False
key = max(why, key=why.get)
return *STEAM_REASONS[key], key.startswith("lost")
words = reply.removeprefix("fail ").split()
# ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
eye = words[1]
if "seen" in words:
return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
if "moved" in words:
return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
if not reply:
return "no answer", "our tracker didn't answer", False
if reply.startswith("our tracker isn't running"):
return "tracker not running", "our tracker isn't running", False
text = reply.removeprefix("fail ")
return text[:24], f"our tracker said: {text}", False
def spread(points):
"""The median point and the spread around it (1.4826 x the median distance: a standard
deviation that one stray sample can't move far)."""
mx = statistics.median(p[0] for p in points)
my = statistics.median(p[1] for p in points)
return 1.4826 * statistics.median(math.hypot(p[0] - mx, p[1] - my) for p in points), (mx, my)
def write_gaze(on, path=CONF):
"""POINTER_GAZE=1|0 in the config, every other line kept."""
try:
lines = path.read_text().splitlines()
except OSError:
lines = []
value = f"POINTER_GAZE={1 if on else 0}"
for i, line in enumerate(lines):
if line.split("#", 1)[0].split("=", 1)[0].strip() == "POINTER_GAZE":
lines[i] = value
break
else:
lines.append(value)
tmp = path.with_suffix(".tmp")
tmp.write_text("\n".join(lines) + "\n")
tmp.replace(path)
def ask(addr, command, timeout=1.0):
"""A command to a local datagram socket, and its reply ("" without one)."""
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
try:
s.bind("")
s.settimeout(timeout)
s.sendto(command.encode(), addr)
return s.recv(4096).decode("utf-8", "replace")
except OSError:
return ""
finally:
s.close()
class Checks:
def __init__(self, svc, sel):
self.svc = svc
self.sel = sel
self.out = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
self.out.bind("") # the panel's and the helper's replies come back here
sel.register(self.out, selectors.EVENT_READ, "checks")
self.check = None
self.gaze_on = None
self.gaze_heard = 0.0
self.full_armed = True # gaze mode on without a calibration opens the full one (need_full)
self.full_blocked = None # why it can't open now
self.full_retry_at = 0.0
self.full_failed = None # why the last calibration failed (too few dots), for problem()
self.last_quick = 0.0
self.sample_at = 0.0 # the tracker last sent anything
self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc")
self.away = True # no eyes for AWAY_MIN: their coming back is the headset going on
self.back_since = None
self.reseat_seen = False
self.after_quick = None
self.panel_proc = None
self.panel_restart_at = 0.0
self.screens_shown = None
self.last_progress = 0.0
@property
def active(self):
return self.check is not None
# --- The panel process ---
def start_panel(self):
if not PANEL_PROG.exists():
log(f"ft-gazepanel isn't built: run {REPO}/gaze/build.sh")
self.panel_restart_at = time.monotonic() + 60
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
distrobox = Path.home() / ".local" / "bin" / "distrobox"
self.panel_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(PANEL_PROG), "--watch-stdin"],
env=env, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE, start_new_session=True)
os.set_blocking(self.panel_proc.stderr.fileno(), False)
self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
log("ft-gazepanel started")
def read_panel(self):
try:
data = os.read(self.panel_proc.stderr.fileno(), 65536)
except BlockingIOError:
return
if not data:
log(f"ft-gazepanel stopped (exit {self.panel_proc.poll()}); again in {PANEL_RETRY:.0f} s")
self.stop_panel()
self.panel_restart_at = time.monotonic() + PANEL_RETRY
if self.check:
self.close("the panel stopped")
return
for line in data.decode("utf-8", "replace").splitlines():
if line.strip():
log(line)
def stop_panel(self):
if not self.panel_proc:
return
try:
self.sel.unregister(self.panel_proc.stderr)
except (KeyError, ValueError):
pass
if self.panel_proc.stdin and not self.panel_proc.stdin.closed:
self.panel_proc.stdin.close()
try:
self.panel_proc.wait(timeout=2)
except subprocess.TimeoutExpired:
try:
os.killpg(self.panel_proc.pid, signal.SIGTERM)
except ProcessLookupError:
pass
self.panel_proc = None
def to_panel(self, command):
try:
self.out.sendto(command.encode(), PANEL)
except OSError:
pass
def to_helper(self, command):
try:
self.out.sendto(command.encode(), POINTER)
except OSError:
pass
def on_readable(self):
"""Replies on our socket: the helper's "ok on|off" to "gaze ?"; the panel's are dropped."""
while True:
try:
data = self.out.recv(4096).decode("utf-8", "replace")
except (BlockingIOError, OSError):
return
if data in ("ok on", "ok off"):
on = data == "ok on"
was, self.gaze_on, self.gaze_heard = self.gaze_on, on, time.monotonic()
if on and was is False:
self.on_gaze_on()
# --- State ---
def calibrated(self):
"""Does the tracker in use have a calibration? None: our tracker hasn't said yet."""
svc = self.svc
kind = svc.kind
if kind == "own":
if time.monotonic() - svc.own_at > 5 or not svc.own:
return None
return bool((svc.own.get("calibration") or {}).get("dots"))
if kind == "eyes":
return True
return svc.models[svc.source].samples > 0
def eyes_seen(self, within=1.0):
return time.monotonic() - self.seen_at < within
def can_run(self):
"""Someone's in the headset and the tracker is sending."""
return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
def on_gaze_on(self):
self.full_armed = True
self.need_full("gaze mode came on without a calibration")
def need_full(self, reason):
"""Gaze mode is on without a calibration: open the full one, once per arming (see the top),
or note why it can't open."""
cal = self.calibrated()
if cal:
self.full_armed = True # missing one later (the other tracker picked) is news again
if not self.gaze_on or self.check or not self.full_armed or cal is not False:
self.full_blocked = None
return
now = time.monotonic()
if not self.can_run():
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
else "no eyes seen (is the headset on?)")
elif now < self.full_retry_at:
return
else:
reply = self.start("full", reason)
why = None if reply == "ok" else reply.removeprefix("error ")
if why:
self.full_retry_at = now + FULL_RETRY
if why and why != self.full_blocked:
log(f"the calibration can't open: {why}")
self.full_blocked = why
if not why:
self.full_armed = False
def problem(self):
"""Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said
yet is None: Input Settings has its own line for our tracker."""
if not self.gaze_on or self.calibrated() is not False:
return None
if self.check and self.check["kind"] == "full":
return "Not calibrated yet: the calibration is open in the headset"
if self.full_blocked:
return f"Not calibrated, and the calibration can't open: {self.full_blocked}"
if not self.full_armed:
if self.full_failed:
return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate"
return "Not calibrated: the calibration closed unfinished. Use Calibrate"
return "Not calibrated: the calibration opens in the headset"
def auto_quick(self, reason):
now = time.monotonic()
if self.check or not self.gaze_on or not self.can_run() or self.calibrated() is not True:
return
if now - self.last_quick < QUICK_COOLDOWN:
log(f"quick check skipped ({reason}): one ran {now - self.last_quick:.0f} s ago")
return
self.start("quick", reason)
# --- Running a check ---
def start(self, kind, reason):
svc = self.svc
if self.check:
return "error a check is running"
if not self.panel_proc:
return "error the panel isn't running (gaze/build.sh builds it)"
if kind == "fit":
return self.start_fit(reason)
if not self.can_run():
return "error the headset is off or the tracker isn't sending"
own = svc.kind == "own"
if kind == "full" and own:
reply = ask(EYES, "calib-start", 3.0)
if not reply.startswith("ok"):
log(f"calibration not started: our tracker says {reply or 'nothing'}")
return f"error our tracker: {reply or 'no reply'}"
now = time.monotonic()
self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
"started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
log(f"{kind} check: {reason}")
if kind == "full":
st = ask(SCREENS, "state", 0.5).split()
if len(st) >= 3 and st[0] == "ok":
self.screens_shown = (st[2] == "0") if st[1] == "always" else (st[2] == "1")
ask(SCREENS, "hide", 0.5)
self.to_helper("calpanel 1")
self.to_panel(f"show {'full' if kind == 'full' else 'quick'}")
self.show_dot()
if kind == "quick":
self.last_quick = now
return "ok"
def start_fit(self, reason):
# Eyes lost are what it's there to show, so it needs only the tracker sending.
if time.monotonic() - self.sample_at > 2:
return "error the headset is off or the eye tracker isn't sending"
now = time.monotonic()
self.check = {"kind": "fit", "reason": reason, "own": False, "dots": [], "i": 0, "started": now, "shown": now,
"run": [], "accept": False, "done_at": None, "tries": 0, "skipped": 0, "captured": 0,
"points": {}, "fit": FitCheck(), "drawn": {}, "drawn_at": 0.0, "step": None}
log(f"fit check: {reason}")
self.to_helper("calpanel 1")
self.to_panel("show fit")
self.to_panel("title Headset fit: adjust the headset while you watch")
self.to_panel("text Left click or Meta+J: guided check · Right click or Meta+K: done")
return "ok"
def fit_tick(self, now):
c = self.check
fit = c["fit"]
# The guided check: its dots at head-relative directions in the panel (the probe's
# screen fractions, spread over the middle of the panel), and its looks as the title.
step = fit.guide_step(now)
key = None if step is None else (step[0], step[1])
if key != c["step"]:
c["step"] = key
if step is None:
self.to_panel("dot 0 0 off")
self.to_panel("title Headset fit: adjust the headset while you watch")
elif step[0] == "dot":
fx, fy = step[1]
self.to_panel(f"dot {(0.5 - fx) * 32:.2f} {(0.5 - fy) * 24:.2f} look")
self.to_panel("title Look at the dot")
else:
self.to_panel("dot 0 0 off")
self.to_panel(f"title {step[1]}")
if now - c["drawn_at"] < FIT_EVERY:
return
c["drawn_at"] = now
drawn = c["drawn"]
for k in (0, 1):
word, (r, g, b) = fit.status(k)
sig, seen = fit.signal(k), fit.tracked_share(k, now)
cmd = (f"eye {k} {r:.2f} {g:.2f} {b:.2f} {-1 if sig is None else round(sig, 1):g} "
f"{-1 if seen is None else round(seen * 20) / 20:g} {word.capitalize()}")
if drawn.get(k) != cmd:
drawn[k] = cmd
self.to_panel(cmd)
if fit.have_eye_data and fit.samples < 3 * MIN_REGION:
hints = ["Look around slowly: up, down, left and right. Or left click (Meta+J) for a guided check."]
else:
hints = fit.hints()
lines = [line for h in hints for line in wrap(h, FIT_HINT_WIDTH)][:8]
cmd = "hints " + "|".join(lines)
if drawn.get("hints") != cmd:
drawn["hints"] = cmd
self.to_panel(cmd)
def show_dot(self):
c = self.check
yaw, pitch, rnd = c["dots"][c["i"]]
if c["kind"] == "full":
self.to_panel(f"bg {ROUND_BG[rnd]}")
self.to_panel(f"title Gaze calibration: {ROUND_NAMES[rnd]} round, {rnd + 1} of 3")
self.to_panel("text Look at the dot and click (left click or Meta+J). Right click or Meta+K: stop")
elif c["kind"] == "five":
self.to_panel(f"text Look at the dot and click ({c['i'] + 1} of {len(c['dots'])})")
else:
self.to_panel("text Look at the dot")
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
self.last_progress = None
c["shown"] = time.monotonic()
c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None
def on_sample(self, s):
self.sample_at = time.monotonic()
unc = (s["src"].get("mmap1") or {}).get("unc")
if unc and min(unc) <= EYE_LOST:
self.seen_at = time.monotonic()
if self.away and self.back_since is None:
self.back_since = self.seen_at
c = self.check
if c and c["kind"] == "fit":
c["fit"].feed(s, self.sample_at)
return
if not c or c["done_at"]:
return
if c["own"]:
src = s["src"].get("own") or {}
else:
src = s["src"].get("mmap1") or {}
if "hy" not in src:
per = [s["src"].get(n) or {} for n in ("left", "right")]
per = [p for p in per if "hy" in p]
src = {"hy": statistics.fmean(p["hy"] for p in per), "hp": statistics.fmean(p["hp"] for p in per)} if per else {}
if "hy" not in src:
return
now = time.monotonic()
c["gaze_at"] = now
if now < c["shown"] + CHECK_SETTLE:
return
g = (src["hy"], src["hp"])
run = c["run"]
if run:
recent = run[-30:]
my, mp = statistics.median(p[2] for p in recent), statistics.median(p[3] for p in recent)
if math.hypot(g[0] - my, g[1] - mp) > 2.5 * CHECK_SPREAD:
run.clear() # the eyes moved on (a saccade, a blink): start over
run.append((now, s, g[0], g[1]))
window = [p for p in run if p[0] >= now - CHECK_WINDOW]
held = now - run[0][0]
# The quick check's ring fills in quarters: each step is a new picture for the panel, so
# few of them keep it solid. The others wait for the click (see the top): no ring.
progress = math.floor(min(1.0, held / CHECK_WINDOW) * 4) / 4
if c["kind"] == "quick" and progress != self.last_progress:
yaw, pitch, _ = c["dots"][c["i"]]
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture {progress:.2f}")
self.last_progress = progress
if c["accept"] and held >= 0.3 and len(window) >= 10:
sd, _ = spread([(p[2], p[3]) for p in window])
if sd <= ACCEPT_SPREAD:
self.capture(window)
return
if c["kind"] == "quick" and held >= CHECK_WINDOW and len(window) >= 20:
sd, _ = spread([(p[2], p[3]) for p in window])
if sd <= CHECK_SPREAD:
self.capture(window)
else:
del run[:len(run) // 2] # not steady enough yet: keep trying with the newer half
def capture(self, window):
svc = self.svc
c = self.check
yaw, pitch, rnd = c["dots"][c["i"]]
samples = [p[1] for p in window]
rec = {"time": time.time(), "check": c["kind"], "dot": c["i"], "round": rnd, "true": [yaw, pitch],
"samples": len(samples), "own": c["own"]}
ok = True
if c["own"]:
eyes = []
for k in (0, 1):
seen = [smp["src"]["own"]["eyes"][k] for smp in samples
if (smp["src"].get("own") or {}).get("eyes") and smp["src"]["own"]["eyes"][k]]
eyes.append((statistics.median(e[0] for e in seen), statistics.median(e[1] for e in seen)) if seen else None)
miss = [math.hypot(yaw - e[0], pitch - e[1]) if e else None for e in eyes]
rec.update(eyes=eyes, miss=miss)
t0, t1 = samples[0]["t"], samples[-1]["t"]
if c["kind"] == "full":
reply = ask(EYES, f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0)
rec["reply"] = reply
ok = reply.startswith("ok")
else:
try:
svc.eyes_sock.sendto(f"click {t1:.6f} {yaw:.4f} {pitch:.4f}".encode(), EYES)
except OSError as e:
rec["reply"], ok = f"our tracker isn't running ({e})", False
if ok:
svc.weights["own"].add(miss)
else:
why = {}
steady = steady_samples(samples, why=why)
rec["dropped"] = why
reads = {}
for name in ("action", "mmap1", "mmap2", "left", "right"):
pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
if "hy" in (smp["src"].get(name) or {})]
if len(pts) >= 15:
reads[name] = (statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
rec["reads"] = reads
main = ("left", "right") if svc.kind == "eyes" else (svc.source,)
if not all(n in reads for n in main):
ok = False
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had both eyes"
elif c["kind"] == "full":
for name, (hy, hp) in reads.items():
c["points"].setdefault(name, []).append((hy, hp, yaw - hy, pitch - hp))
try:
with open(POINTS, "a") as f:
for name, (hy, hp) in reads.items():
f.write(json.dumps({"time": time.time(), "source": name, "hy": hy, "hp": hp,
"off": [yaw - hy, pitch - hp], "layout": None, "how": "panel"}) + "\n")
except OSError:
pass
else:
miss = []
for name in main:
hy, hp = reads[name]
cy, cp = svc.correction(name, hy, hp)
miss.append(math.hypot(yaw - hy - cy, pitch - hp - cp))
svc.lives[name].add({"time": time.time(), "hy": hy, "hp": hp, "dy": yaw - hy, "dp": pitch - hp,
"wy": 1.0, "wp": 1.0, "how": "check"}, svc.models[name], svc.mode)
rec["miss"] = miss
if svc.kind == "eyes":
svc.weights["steam"].add(miss)
svc.dirty = True
if not ok:
short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
rec["reason"] = long
c["reasons"][long] = c["reasons"].get(long, 0) + 1
if fit:
c["fit_reasons"].add(long)
rec["accepted"] = ok
self.log_check(rec)
if not ok:
c["tries"] += 1
log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
full = c["kind"] == "full"
if c["tries"] >= 2 or not full:
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
self.skip()
else:
self.note(f"Not taken: {long}. Look at the dot and click again")
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
c["run"], c["accept"], c["accept_at"] = [], False, None
c["shown"] = time.monotonic() # settle again, then retry
return
c["captured"] += 1
c["tries"] = 0
self.note("")
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
c["done_at"] = time.monotonic() + DONE_PAUSE
def note(self, text):
"""The panel's warning line, over the instructions (empty: none). It stays until the
next dot is taken, so a skipped dot's reason is still there at the one after it."""
c = self.check
if c.get("note") != text:
c["note"] = text
self.to_panel(f"note {text}".rstrip())
def skip(self):
c = self.check
yaw, pitch, _ = c["dots"][c["i"]]
c["skipped"] += 1
c["tries"] = 0
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
c["done_at"] = time.monotonic() + DONE_PAUSE
def advance(self):
c = self.check
c["i"] += 1
if c["i"] < len(c["dots"]):
self.show_dot()
return
self.finish()
def finish(self):
svc = self.svc
c = self.check
kind = c["kind"]
if kind in ("quick", "five"):
if c["captured"]:
log(f"{kind} check done ({c['captured']} of {len(c['dots'])} dots)")
self.after_quick = [] if kind == "quick" else None
self.close()
return
n = len(c["dots"])
if c["captured"] < n * 2 / 3:
reasons = c["reasons"]
main = max(reasons, key=reasons.get) if reasons else None
self.full_failed = main
log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays"
+ (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else ""))
self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
if main:
fit = main in c["fit_reasons"]
self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else ""))
self.to_panel("dot 0 0 off")
c["done_at"] = time.monotonic() + (8.0 if main else 3.0) # time to read why
c["closing"] = True
return
self.full_failed = None
if c["own"]:
reply = ask(EYES, "calib-fit", 10.0)
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
else:
mode = svc.mode if svc.mode != "none" else DEFAULT_MODEL
for name, pts in c["points"].items():
if name in svc.models and pts:
svc.models[name].fit(pts, mode)
d = {name: m.to_json() for name, m in svc.models.items()}
d["_meta"] = {"calibrated_at": time.time(), "model": mode, "how": "panel"}
d["_live"] = {}
tmp = CALIBRATION.with_suffix(".tmp")
tmp.write_text(json.dumps(d, indent=1))
tmp.replace(CALIBRATION)
svc.forget_lessons() # a new calibration: the pointer's lessons start over on top of it
svc.load_calibration()
svc.refit()
log(f"calibration ({c['captured']} of {n} dots): {mode}, saved")
self.close()
def close(self, why=None):
if not self.check:
return
if why:
log(f"{self.check['kind']} check closed: {why}")
self.to_panel("hide")
self.to_helper("calpanel 0")
if self.check["kind"] == "full" and self.screens_shown:
ask(SCREENS, "show", 0.5)
self.screens_shown = None
self.check = None
def quit(self):
c = self.check
if not c:
return
self.close("quit")
if c["kind"] == "full" and self.calibrated() is False:
# No calibration still: gaze mode can't work, so it goes off until it's turned on again.
try:
write_gaze(False)
except OSError as e:
log(f"can't write {CONF}: {e}")
self.to_helper("gaze off")
self.gaze_on = False
log("gaze mode off: no calibration")
def log_check(self, rec):
try:
with open(CHECK_LOG, "a") as f:
f.write(json.dumps(rec) + "\n")
except OSError:
pass
# --- From the service ---
def command(self, words):
"""quickcal, calibrate, calaccept, calquit -> a reply."""
cmd = words[0]
if cmd == "quickcal":
return self.start("full" if self.calibrated() is False else "quick", "asked for")
if cmd == "calibrate":
return self.start("full", "asked for")
if cmd == "fitcheck":
return self.start("fit", "asked for")
if cmd == "calaccept":
if self.check and self.check["kind"] == "fit":
self.check["fit"].toggle_guide(time.monotonic())
elif self.check:
if not self.check["accept"]:
self.check["accept_at"] = time.monotonic()
self.check["accept"] = True
return "ok"
if cmd == "calquit":
self.quit()
return "ok"
if cmd == "recheck" and len(words) == 2:
self.auto_quick(f"a click was corrected {words[1]} deg")
return "ok"
return "error unknown command"
def after_lesson(self, rec):
if self.after_quick is None or "refused" in rec:
return
self.after_quick.append(rec.get("lesson_deg", 0.0))
if len(self.after_quick) < FIVE_COUNT:
return
off = self.after_quick
self.after_quick = None
if all(d > FIVE_LIMIT for d in off):
log(f"the {FIVE_COUNT} lessons after the quick check were {', '.join(f'{d:.1f}' for d in off)} deg off")
if self.gaze_on and self.can_run() and not self.check:
self.start("five", "the quick check didn't fix it")
def tick(self):
c = self.check
if not c:
return
now = time.monotonic()
if c["kind"] == "fit":
# Not closed when the eyes go: adjusting the headset loses them.
if now - c["started"] > FIT_TIMEOUT:
self.close("timed out")
else:
self.fit_tick(now)
return
if c["done_at"] and now >= c["done_at"]:
if c.get("closing"):
self.close()
else:
self.advance()
return
if not self.eyes_seen(EYES_GONE):
self.close("the headset came off")
return
if c["done_at"]:
return
if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
# Clicked, but no capture yet (see on_sample): say what it's waiting for.
full = c["kind"] == "full"
if now - c.get("gaze_at", 0.0) > 0.5:
self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
else:
self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
self.close("ignored")
elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
self.close(f"no click on dot {c['i'] + 1} for {CLICK_IDLE:.0f} s")
def periodic(self):
svc = self.svc
now = time.monotonic()
if not self.panel_proc and now >= self.panel_restart_at:
self.start_panel()
self.to_helper("gaze ?")
if now - self.gaze_heard > 5:
self.gaze_on = None # the helper isn't answering
if self.check:
self.to_helper("calpanel 1")
if not self.eyes_seen(AWAY_MIN):
self.away, self.back_since = True, None
elif self.back_since is not None and not self.eyes_seen():
self.back_since = None # gone again before DON_DELAY
elif self.back_since is not None and now - self.back_since >= DON_DELAY:
self.away, self.back_since = False, None
self.full_armed = True # a calibration that closed unfinished opens again
self.auto_quick("the headset went on")
if svc.kind == "own" and now - svc.own_at < 5:
reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
if not reseat:
self.reseat_seen = False
elif not self.reseat_seen and self.can_run():
self.reseat_seen = True
self.auto_quick("our tracker asked for a click")
self.need_full("gaze mode is on without a calibration")
def status(self):
c = self.check
st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
"problem": self.problem(),
"panel": self.panel_proc is not None,
"last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
if c:
st["check"] = {"kind": c["kind"], "dot": c["i"] + 1, "dots": len(c["dots"]), "captured": c["captured"],
"skipped": c["skipped"], "reason": c["reason"]}
return st
def stop(self):
self.close("the gaze service is stopping")
self.stop_panel()
+533
View File
@@ -0,0 +1,533 @@
// ft-gazepanel: the gaze calibration panel (docs/design.md, gaze/README.md). A SteamVR overlay
// fixed to the headset, so wherever you turn your head it stays in the same place in your
// view: a dot drawn at a head-relative direction is exactly that direction from the headset,
// which is what the gaze service needs to know where you were looking. It's drawn on the CPU
// and takes no input: the gaze service (gaze/ft-gazed) drives it, and the pointer helper
// passes it your presses (calaccept, calquit).
//
// The panel sits POINTER-like at --distance (1.5 m, about where Frametop's screens are, so
// the eyes converge as they do in use). "quick" is a small square, QUICK_DEG across, for the
// one-dot check; "full" is FULL_DEG across (4:3), with a solid background whose brightness the
// service sets per round (pupil size changes with it, and the tracker's error with it); "fit"
// is FIT_DEG across (4:3), see-through like quick, for the headset fit check: a card per eye
// (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and hints.
//
// Control socket: abstract unix datagram "@ft_gazepanel" (--socket NAME); a sender with an
// address gets "ok" or "error ...":
// show quick|full|fit the panel, empty, in front of you
// hide
// bg <0..1> the background's brightness (full)
// dot <yaw> <pitch> <state> [<progress 0..1>]
// the dot, head-relative degrees (yaw +left, pitch +up); state:
// look, capture (a ring filling to progress), done,
// fail, off
// title <text> / text <text> a line at the top / at the bottom (empty to clear)
// note <text> a warning line just above the bottom one, in orange (red on
// the bright round): why a dot wasn't taken (empty to clear)
// eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
// fit: an eye's card (0 left): its state in that colour, the
// tracker's signal, the share of the last 10 s it was seen
// hints <line>|<line>|... fit: lines under the cards (empty to clear)
// ping
//
// Each picture goes into the next of three shared buffers (linear DMA-BUFs SteamVR imported
// once, the size of the biggest panel; the texture bounds show the part in use), and the panel
// switches to it, as screens/keyboard.cpp does. SetOverlayRaw, which uploads a new texture each
// time, flickered on every change of the full calibration's 1024x768 picture, and in a live
// test the headset kept showing an old picture after the panel had drawn new ones (2026-10-01).
// It's only the fallback. A "show" makes the panel visible once its first picture is in.
//
// Options: --watch-stdin (quit when stdin closes: the service runs it), --socket NAME,
// --distance METRES. Runs in the dev container (gaze/build.sh builds it into gaze/build).
#include <openvr.h>
#define STB_TRUETYPE_IMPLEMENTATION
#include "stb_truetype.h"
#include <drm_fourcc.h>
#include <fcntl.h>
#include <gbm.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <csignal>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <map>
#include <string>
#include <thread>
#include <vector>
namespace {
using Clock = std::chrono::steady_clock;
constexpr double kQuickDeg = 16; // QUICK_DEG: the one-dot check's square
constexpr double kFullDeg = 64; // FULL_DEG: the full calibration's width (4:3)
constexpr double kFitDeg = 40; // FIT_DEG: the headset fit check's width (4:3)
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600;
std::atomic<bool> g_stop{false};
// ---------------------------------------------------------------- text (as screens/keyboard.cpp)
stbtt_fontinfo g_font;
std::vector<unsigned char> g_fontData;
bool g_fontOk = false;
constexpr int kMaxW = kFullW, kMaxH = kFullH; // the buffers' size: the biggest panel
struct Glyph {
std::vector<unsigned char> bitmap;
int w = 0, h = 0, xoff = 0, yoff = 0, advance = 0;
};
std::map<std::pair<uint32_t, int>, Glyph> g_glyphs;
void LoadFont() {
std::string path;
if (FILE *p = popen("fc-match -f '%{file}' 'Noto Sans' 2>/dev/null", "r")) {
char buf[512];
if (std::fgets(buf, sizeof buf, p)) path = buf;
pclose(p);
}
if (path.empty()) path = "/usr/share/fonts/google-noto-vf/NotoSans[wght].ttf";
if (FILE *f = std::fopen(path.c_str(), "rb")) {
std::fseek(f, 0, SEEK_END);
g_fontData.resize(size_t(std::ftell(f)));
std::fseek(f, 0, SEEK_SET);
g_fontOk = std::fread(g_fontData.data(), 1, g_fontData.size(), f) == g_fontData.size() &&
stbtt_InitFont(&g_font, g_fontData.data(), stbtt_GetFontOffsetForIndex(g_fontData.data(), 0));
std::fclose(f);
}
if (!g_fontOk) std::fprintf(stderr, "ft-gazepanel: no font (%s); no text\n", path.c_str());
}
const Glyph &GetGlyph(uint32_t cp, int size) {
auto [it, fresh] = g_glyphs.try_emplace({cp, size});
Glyph &g = it->second;
if (fresh) {
const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
unsigned char *b = stbtt_GetCodepointBitmap(&g_font, 0, scale, int(cp), &g.w, &g.h, &g.xoff, &g.yoff);
if (b) g.bitmap.assign(b, b + size_t(g.w) * g.h), stbtt_FreeBitmap(b, nullptr);
int adv, lsb;
stbtt_GetCodepointHMetrics(&g_font, int(cp), &adv, &lsb);
g.advance = int(std::lround(adv * scale));
}
return g;
}
std::vector<uint32_t> Codepoints(const std::string &s) {
std::vector<uint32_t> out;
for (const unsigned char *p = (const unsigned char *)s.c_str(); *p;) {
uint32_t c = *p++;
int more = c >= 0xF0 ? 3 : c >= 0xE0 ? 2 : c >= 0xC0 ? 1 : 0;
if (more) c &= 0x3Fu >> more;
for (; more && (*p & 0xC0) == 0x80; --more) c = c << 6 | (*p++ & 0x3F);
out.push_back(c);
}
return out;
}
// ---------------------------------------------------------------- the picture
struct EyeCard {
std::string word = "no data";
double r = 0.6, g = 0.6, b = 0.6, signal = -1, seen = -1;
};
struct Panel {
bool full = false, fit = false;
EyeCard eyes[2];
std::vector<std::string> hints;
int w = kQuickPx, h = kQuickPx;
double wDeg = kQuickDeg; // across
std::vector<uint8_t> px;
double bg = 0.05;
std::string title, text, note;
bool dotOn = false;
double dotYaw = 0, dotPitch = 0, progress = 0;
std::string state = "off";
};
void Blend(Panel &p, int x, int y, double r, double g, double b, double a) {
if (x < 0 || y < 0 || x >= p.w || y >= p.h || a <= 0) return;
uint8_t *q = &p.px[(size_t(y) * p.w + x) * 4];
a = std::min(a, 1.0);
q[0] = uint8_t(std::lround(q[0] + (r * 255 - q[0]) * a));
q[1] = uint8_t(std::lround(q[1] + (g * 255 - q[1]) * a));
q[2] = uint8_t(std::lround(q[2] + (b * 255 - q[2]) * a));
q[3] = uint8_t(std::lround(q[3] + (255 - q[3]) * a));
}
// A filled disc, or (inner > 0) a ring from inner to outer radius, anti-aliased; with sweep < 1,
// only that share of the ring, clockwise from the top.
void Disc(Panel &p, double cx, double cy, double outer, double inner, double r, double g, double b, double a,
double sweep = 1) {
const int x0 = int(cx - outer - 1), x1 = int(cx + outer + 1), y0 = int(cy - outer - 1), y1 = int(cy + outer + 1);
for (int y = y0; y <= y1; ++y)
for (int x = x0; x <= x1; ++x) {
const double dx = x + 0.5 - cx, dy = y + 0.5 - cy, d = std::hypot(dx, dy);
double cover = std::clamp(outer - d + 0.5, 0.0, 1.0);
if (inner > 0) cover = std::min(cover, std::clamp(d - inner + 0.5, 0.0, 1.0));
if (sweep < 1) {
double ang = std::atan2(dx, -dy) / (2 * M_PI); // 0 at the top, clockwise
if (ang < 0) ang += 1;
if (ang > sweep) continue;
}
Blend(p, x, y, r, g, b, a * cover);
}
}
void Rect(Panel &p, int x0, int y0, int x1, int y1, double r, double g, double b, double a) {
for (int y = std::max(y0, 0); y < std::min(y1, p.h); ++y)
for (int x = std::max(x0, 0); x < std::min(x1, p.w); ++x) Blend(p, x, y, r, g, b, a);
}
// Text centred on (x, cy), or starting at x (left).
void Text(Panel &p, const std::string &s, int size, int x, int cy, double r, double g, double b, bool left = false) {
if (!g_fontOk || s.empty()) return;
const auto cps = Codepoints(s);
int width = 0;
for (uint32_t cp : cps) width += GetGlyph(cp, size).advance;
int ascent, descent, gap;
stbtt_GetFontVMetrics(&g_font, &ascent, &descent, &gap);
const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
if (!left) x -= width / 2;
const int baseline = cy + int(std::lround((ascent + descent) * scale / 2));
for (uint32_t cp : cps) {
const Glyph &gl = GetGlyph(cp, size);
for (int gy = 0; gy < gl.h; ++gy)
for (int gx = 0; gx < gl.w; ++gx)
Blend(p, x + gl.xoff + gx, baseline + gl.yoff + gy, r, g, b, gl.bitmap[size_t(gy) * gl.w + gx] / 255.0);
x += gl.advance;
}
}
void Text(Panel &p, const std::string &s, int size, int x, int cy, double lum, bool left = false) {
Text(p, s, size, x, cy, lum, lum, lum, left);
}
// The headset fit check (see the top): a card per eye, then the hints.
void DrawFit(Panel &p, double pxPerDeg, int textSize, double faint) {
const int margin = int(p.w * 0.06), cw = int(p.w * 0.41), ch = int(p.h * 0.32), top = int(p.h * 0.12);
const int pad = int(pxPerDeg * 0.9), big = int(pxPerDeg * 1.6), small = int(pxPerDeg * 0.8);
for (int k = 0; k < 2; ++k) {
const EyeCard &e = p.eyes[k];
const int x0 = k == 0 ? margin : p.w - margin - cw;
Rect(p, x0, top, x0 + cw, top + ch, 1, 1, 1, 0.07);
Text(p, k ? "Right eye" : "Left eye", textSize, x0 + pad, top + pad + textSize / 2, faint, true);
Text(p, e.word, big, x0 + pad, top + int(ch * 0.40), e.r, e.g, e.b, true);
// The tracker's signal: a bar, red to green.
const int by = top + int(ch * 0.62), bh = std::max(4, int(pxPerDeg * 0.35)), bw = cw - 2 * pad;
Text(p, "Signal", small, x0 + pad, by - small, faint, true);
Rect(p, x0 + pad, by, x0 + pad + bw, by + bh, 1, 1, 1, 0.15);
if (e.signal >= 0) {
const double v = std::clamp(e.signal, 0.0, 1.0);
const double r = v < 0.5 ? 1.0 : 1.0 - 1.3 * (v - 0.5), g = v < 0.5 ? 0.3 + 0.9 * v : 0.75 + 0.5 * (v - 0.5);
Rect(p, x0 + pad, by, x0 + pad + int(bw * v), by + bh, r, g, 0.35, 0.95);
}
char seen[64] = "Seen: not yet";
if (e.seen >= 0) std::snprintf(seen, sizeof seen, "Seen %d%% of the last 10 s", int(std::lround(e.seen * 100)));
Text(p, seen, small, x0 + pad, top + ch - pad, faint, true);
}
int y = top + ch + pad * 2;
for (const std::string &line : p.hints) {
Text(p, line, textSize, margin, y, faint, true);
y += int(textSize * 1.4);
}
}
// Head-relative direction -> panel pixel: the panel is a plane `d` in front of the headset.
void ToPixel(const Panel &p, double yaw, double pitch, double &x, double &y) {
const double yr = yaw * M_PI / 180, pr = pitch * M_PI / 180;
const double half = std::tan(p.wDeg * M_PI / 360); // half the width, per unit of distance
const double X = -std::tan(yr), Y = std::tan(pr) / std::cos(yr);
x = (X / (2 * half) + 0.5) * p.w;
y = (0.5 - Y / (2 * half) * p.w / p.h) * p.h;
}
void Draw(Panel &p) {
p.px.assign(size_t(p.w) * p.h * 4, 0);
const double pxPerDeg = p.w / p.wDeg;
if (p.full) {
for (size_t i = 0; i < p.px.size(); i += 4)
p.px[i] = p.px[i + 1] = p.px[i + 2] = uint8_t(std::lround(p.bg * 255)), p.px[i + 3] = 255;
} else {
// The quick check and the fit check: a dim rounded panel, see-through, so it's clear of what's behind.
const double r = std::min(p.w, p.h) * 0.12;
for (int y = 0; y < p.h; ++y)
for (int x = 0; x < p.w; ++x) {
const double dx = std::max({r - x - 0.5, x + 0.5 - (p.w - r), 0.0});
const double dy = std::max({r - y - 0.5, y + 0.5 - (p.h - r), 0.0});
const double cover = std::clamp(r - std::hypot(dx, dy) + 0.5, 0.0, 1.0);
Blend(p, x, y, 0.06, 0.06, 0.07, 0.82 * cover);
}
}
const bool light = p.full && p.bg > 0.5; // a dark dot on the bright round
const double ink = light ? 0.0 : 1.0, faint = light ? 0.2 : 0.75;
const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
// Under the full calibration's lowest dots (RING degrees down) and over the text.
if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05);
else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3);
if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
if (!p.dotOn || p.state == "off") return;
double x, y;
ToPixel(p, p.dotYaw, p.dotPitch, x, y);
const double core = 0.22 * pxPerDeg;
if (p.state == "look") {
// Still, so the panel looks solid and is drawn again only when something changes.
const double rr = 0.75 * pxPerDeg;
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.6);
Disc(p, x, y, core, 0, ink, ink, ink, 1);
} else if (p.state == "capture") {
const double rr = 0.75 * pxPerDeg;
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.25);
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, 0.3, 0.85, 1.0, 1, std::clamp(p.progress, 0.0, 1.0));
Disc(p, x, y, core, 0, ink, ink, ink, 1);
} else if (p.state == "done") {
Disc(p, x, y, 0.75 * pxPerDeg, 0, 0.25, 0.85, 0.4, 0.9);
Disc(p, x, y, core, 0, 1, 1, 1, 1);
} else if (p.state == "fail") {
Disc(p, x, y, 0.75 * pxPerDeg, 0.6 * pxPerDeg, 0.95, 0.35, 0.3, 0.9);
Disc(p, x, y, core, 0, ink, ink, ink, 1);
}
}
// ---------------------------------------------------------------- the buffers (see the top)
struct Buffer {
gbm_bo *bo = nullptr;
int fd = -1;
vr::SharedTextureHandle_t handle = 0;
};
struct Buffers {
int drm = -1;
gbm_device *gbm = nullptr;
Buffer b[3];
int next = 0;
bool ok = false;
bool Make() {
drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
if (drm >= 0) gbm = gbm_create_device(drm);
for (Buffer &x : b) {
// ABGR8888 is R, G, B, A in memory, like Panel::px.
if (gbm) x.bo = gbm_bo_create(gbm, kMaxW, kMaxH, GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
if (!x.bo || (x.fd = gbm_bo_get_fd(x.bo)) < 0) break;
vr::DmabufAttributes_t a{};
a.unWidth = kMaxW, a.unHeight = kMaxH;
a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
a.unFormat = DRM_FORMAT_ABGR8888;
a.ulModifier = DRM_FORMAT_MOD_LINEAR;
a.unPlaneCount = 1;
a.plane[0].unOffset = gbm_bo_get_offset(x.bo, 0);
a.plane[0].unStride = gbm_bo_get_stride(x.bo);
a.plane[0].nFd = x.fd;
if (!vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &x.handle)) x.handle = 0;
if (!x.handle) break;
}
ok = b[2].handle != 0;
if (!ok) {
std::fprintf(stderr, "ft-gazepanel: no shared buffers; falling back to SetOverlayRaw (it flickers)\n");
Drop();
}
return ok;
}
void Drop() {
for (Buffer &x : b) {
if (x.handle) vr::VRIPCResourceManager()->UnrefResource(x.handle);
if (x.fd >= 0) close(x.fd);
if (x.bo) gbm_bo_destroy(x.bo);
x = Buffer{};
}
if (gbm) gbm_device_destroy(gbm);
if (drm >= 0) close(drm);
gbm = nullptr, drm = -1, ok = false;
}
// p's picture to the overlay: into the next buffer, premultiplied (the overlay's flag says
// so), then the overlay switches to it.
void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t h, const Panel &p) {
vr::EVROverlayError e;
if (!ok) {
e = ov->SetOverlayRaw(h, const_cast<uint8_t *>(p.px.data()), uint32_t(p.w), uint32_t(p.h), 4);
} else {
Buffer &x = b[next];
next = (next + 1) % 3;
uint32_t stride = 0;
void *mapping = nullptr;
auto *dst = static_cast<uint8_t *>(gbm_bo_map(x.bo, 0, 0, p.w, p.h, GBM_BO_TRANSFER_WRITE, &stride, &mapping));
if (!dst) {
std::fprintf(stderr, "ft-gazepanel: can't map a buffer\n");
return;
}
for (int y = 0; y < p.h; ++y) {
const uint8_t *src = &p.px[size_t(y) * p.w * 4];
uint8_t *row = dst + size_t(y) * stride;
for (int i = 0; i < p.w * 4; i += 4) {
const unsigned a = src[i + 3];
row[i] = uint8_t(src[i] * a / 255), row[i + 1] = uint8_t(src[i + 1] * a / 255);
row[i + 2] = uint8_t(src[i + 2] * a / 255), row[i + 3] = uint8_t(a);
}
}
gbm_bo_unmap(x.bo, mapping);
const vr::VRTextureBounds_t bounds{0, 0, float(p.w) / kMaxW, float(p.h) / kMaxH};
ov->SetOverlayTextureBounds(h, &bounds);
vr::Texture_t tex = {&x.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
e = ov->SetOverlayTexture(h, &tex);
}
if (e != vr::VROverlayError_None)
std::fprintf(stderr, "ft-gazepanel: the picture didn't go to SteamVR: %s\n", ov->GetOverlayErrorNameFromEnum(e));
}
};
} // namespace
int main(int argc, char **argv) {
bool watchStdin = false;
std::string sockName = "ft_gazepanel";
double distance = 1.5;
for (int i = 1; i < argc; ++i) {
if (!std::strcmp(argv[i], "--watch-stdin")) watchStdin = true;
else if (!std::strcmp(argv[i], "--socket") && i + 1 < argc) sockName = argv[++i];
else if (!std::strcmp(argv[i], "--distance") && i + 1 < argc) distance = std::clamp(std::atof(argv[++i]), 0.5, 5.0);
else {
std::fprintf(stderr, "usage: %s [--watch-stdin] [--socket NAME] [--distance METRES]\n", argv[0]);
return 2;
}
}
std::signal(SIGINT, [](int) { g_stop = true; });
std::signal(SIGTERM, [](int) { g_stop = true; });
if (watchStdin)
std::thread([] {
char c[256];
while (read(0, c, sizeof c) > 0) {
}
g_stop = true;
}).detach();
int sock = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
sockaddr_un addr{};
addr.sun_family = AF_UNIX;
std::memcpy(addr.sun_path + 1, sockName.data(), std::min(sockName.size(), sizeof addr.sun_path - 2));
if (bind(sock, reinterpret_cast<sockaddr *>(&addr), socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sockName.size())) != 0) {
std::fprintf(stderr, "ft-gazepanel: @%s is taken (another copy running?)\n", sockName.c_str());
return 1;
}
// As Frametop's other SteamVR clients: background first, so we never start vrserver.
vr::EVRInitError err = vr::VRInitError_None;
vr::VR_Init(&err, vr::VRApplication_Background);
if (err == vr::VRInitError_None) {
vr::VR_Shutdown();
vr::VR_Init(&err, vr::VRApplication_Overlay);
}
if (err != vr::VRInitError_None) {
std::fprintf(stderr, "ft-gazepanel: SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
return 1;
}
vr::IVROverlay *ov = vr::VROverlay();
vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
if (ov->CreateOverlay("frametop.gazepanel", "Frametop gaze calibration", &h) != vr::VROverlayError_None) {
std::fprintf(stderr, "ft-gazepanel: can't create the overlay (another copy running?)\n");
return 1;
}
ov->SetOverlaySortOrder(h, 250); // in front of Frametop's screens and the pointer's dot
LoadFont();
Buffers buffers;
if (buffers.Make()) ov->SetOverlayFlag(h, vr::VROverlayFlags_IsPremultiplied, true);
Panel p;
bool visible = false, dirty = false, shown = false; // shown: SteamVR shows it (after its first picture)
auto place = [&] {
const double half = std::tan(p.wDeg * M_PI / 360);
vr::HmdMatrix34_t m{};
m.m[0][0] = m.m[1][1] = m.m[2][2] = 1;
m.m[2][3] = float(-distance);
ov->SetOverlayTransformTrackedDeviceRelative(h, vr::k_unTrackedDeviceIndex_Hmd, &m);
ov->SetOverlayWidthInMeters(h, float(2 * distance * half));
};
std::fprintf(stderr, "ft-gazepanel running: @%s, %.2f m\n", sockName.c_str(), distance);
while (!g_stop) {
char buf[512];
sockaddr_un from{};
socklen_t fromLen = sizeof from;
ssize_t n;
while ((n = recvfrom(sock, buf, sizeof buf - 1, 0, reinterpret_cast<sockaddr *>(&from), &fromLen)) > 0) {
buf[n] = 0;
std::string reply = "ok";
char word[16] = "", state[16] = "";
double a = 0, b = 0, c = 0, d = 0, e = 0;
int rest = 0;
if (!std::strncmp(buf, "show ", 5)) {
p.full = !std::strcmp(buf + 5, "full");
p.fit = !std::strcmp(buf + 5, "fit");
p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
place();
visible = dirty = true; // shown with its first picture
} else if (!std::strcmp(buf, "hide")) {
ov->HideOverlay(h);
visible = shown = false;
} else if (std::sscanf(buf, "bg %lf", &a) == 1) {
p.bg = std::clamp(a, 0.0, 1.0), dirty = true;
} else if (std::sscanf(buf, "dot %lf %lf %15s %lf", &a, &b, state, &c) >= 3) {
p.dotYaw = a, p.dotPitch = b, p.state = state, p.progress = c;
p.dotOn = std::strcmp(state, "off") != 0, dirty = true;
} else if (int k; std::sscanf(buf, "eye %d %lf %lf %lf %lf %lf %n", &k, &a, &b, &c, &d, &e, &rest) >= 6 &&
rest > 0 && (k == 0 || k == 1)) {
p.eyes[k] = EyeCard{buf + rest, a, b, c, d, e}, dirty = true;
} else if (!std::strncmp(buf, "hints", 5)) {
p.hints.clear();
std::string s = buf[5] == ' ' ? buf + 6 : "";
for (size_t at = 0; !s.empty() && at <= s.size();) {
const size_t bar = std::min(s.find('|', at), s.size());
p.hints.push_back(s.substr(at, bar - at));
at = bar + 1;
}
dirty = true;
} else if (!std::strncmp(buf, "title", 5)) {
p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
} else if (!std::strncmp(buf, "text", 4)) {
p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
} else if (!std::strncmp(buf, "note", 4)) {
p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true;
} else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
reply = visible ? "ok shown" : "ok hidden";
} else {
reply = "error unknown command";
}
if (fromLen > offsetof(sockaddr_un, sun_path))
sendto(sock, reply.data(), reply.size(), 0, reinterpret_cast<sockaddr *>(&from), fromLen);
fromLen = sizeof from;
}
vr::VREvent_t ev;
while (vr::VRSystem()->PollNextEvent(&ev, sizeof ev))
if (ev.eventType == vr::VREvent_Quit) {
vr::VRSystem()->AcknowledgeQuit_Exiting();
g_stop = true;
}
if (visible && dirty) {
Draw(p);
buffers.Present(ov, h, p);
if (!shown) ov->ShowOverlay(h), shown = true;
dirty = false;
}
std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
}
ov->DestroyOverlay(h);
buffers.Drop();
vr::VR_Shutdown();
return 0;
}
+4 -4
View File
@@ -1,9 +1,9 @@
[Desktop Entry] [Desktop Entry]
Type=Application Type=Application
Name=Frametop Gaze Probe Name=Frametop Gaze Probe (development)
GenericName=Eye tracking playground GenericName=Eye tracking development tool
Comment=How accurate the headset's eye tracking is on your screens, and gaze clicking with calibration Comment=For developing Frametop's gaze tracking. Day to day, the calibration and checks are on the Gaze page of Frametop Input Settings
Exec=@REPO@/gaze/probe/ft-gazeprobe Exec=@REPO@/gaze/probe/ft-gazeprobe
Icon=view-visible Icon=view-visible
Categories=Utility;Development; Categories=Development;
Keywords=eye;gaze;tracking;calibration;pointer;steamvr;frametop; Keywords=eye;gaze;tracking;calibration;pointer;steamvr;frametop;
+2 -2
View File
@@ -12,8 +12,8 @@ case ${1:-status} in
"$root/gaze/build.sh" "$root/gaze/build.sh"
fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit" fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl" on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl"
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable --now $unit "$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $unit
sleep 2; echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 5" ;; $(start_with_steamvr $unit)" ;;
uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;; uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;; start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "gaze/ft-gazectl status" || true ;; status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "gaze/ft-gazectl status" || true ;;
+7 -21
View File
@@ -2,12 +2,12 @@
# Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system # Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
# service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of # service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
# SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while # SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker # ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
# is Own tracker or the gaze probe uses it. # tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit. On the # probe uses it. install.sh offers this after gaze mode.
# Frame, sudo asks for the password in the terminal, or runs SUDO_ASKPASS when that's set. # Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
# From a PC (or with no terminal), the password comes from steamos_root_pwd in the repo's .env # the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
# and is sent to sudo -S on stdin, never on a command line. # (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
# Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]] # Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]]
set -euo pipefail set -euo pipefail
@@ -16,21 +16,7 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
src=$FRAME_REPO/gaze/tracker src=$FRAME_REPO/gaze/tracker
unit=frametop-eyegrab.service unit=frametop-eyegrab.service
sudo_run() { sudo_run() { frame_sudo "$1"; }
if [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; then
sudo -A bash -c "$1" # SUDO_ASKPASS supplies the password
return
fi
if [ "$FRAME_LOCAL" = 1 ] && [ -t 0 ]; then
sudo bash -c "$1" # asks for the password here
return
fi
local pw
pw=$(sed -n 's/^steamos_root_pwd=//p' "$root/.env" 2>/dev/null)
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
[ -n "$pw" ] || { echo "no terminal for sudo, and steamos_root_pwd is missing from $root/.env" >&2; exit 1; }
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
}
case ${1:-install} in case ${1:-install} in
install) install)
-41
View File
@@ -1,41 +0,0 @@
"""Per-frame pupil ellipses through a recording, cached in the capture (numbers only).
ellipses(cap) -> {camera: array of rows (t, x, y, a, b, major, fill, glint mid x, y)}, every
EVERY-th frame of each camera; the glint midpoint is NaN when the pair isn't seen.
"""
import numpy as np
import eyes_lab # noqa: F401 (puts gaze/tracker on the path)
import eyes_pupil
EVERY = 2
VERSION = 2
def load_index(cap):
return np.array([[float(v) for v in l.split()]
for l in (cap / "index.txt").read_text().splitlines() if len(l.split()) == 4])
def ellipses(cap):
cache = cap / f"ellipses-v{VERSION}.npz"
if cache.exists():
z = np.load(cache)
return {0: z["cam0"], 1: z["cam1"]}
idx = load_index(cap)
frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, 400, 512)
idx = idx[:len(frames)]
out = {}
for c in (0, 1):
rows, prev = [], None
for i in np.where(idx[:, 2] == c)[0][::EVERY]:
p = eyes_pupil.find_pupil(frames[i], prev)
prev = p
if p is None:
continue
pair = eyes_pupil.glint_pair(p)
mx, my = ((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2) if pair else (np.nan, np.nan)
rows.append((idx[i, 3], p["x"], p["y"], p["a"], p["b"], p["major"], p["fill"], mx, my))
out[c] = np.array(rows).reshape(-1, 9)
np.savez(cache, cam0=out[0], cam1=out[1])
return out
Executable
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env bash
# Frametop's one-line installer. In a terminal on the Steam Frame (Konsole in the desktop, or
# over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
#
# It asks which version to install, clones the repo into ~/frametop (or updates the clone
# that's there), and runs its install.sh. Run it again to update, or to switch versions.
# Options (piped, they go after "bash -s --"):
# --stable the main branch: tested releases (the default for a new install)
# --experimental the experimental branch: the newest features, less tested
# --dir DIR where the repo goes (default ~/frametop)
# --clone-only get or update the repo, but don't run install.sh
# --yes, --no-bluetooth passed to install.sh (--yes also answers this script's question:
# the version already there, or stable)
set -euo pipefail
usage() {
cat <<'EOF'
usage: get.sh [--stable | --experimental] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
EOF
}
# Everything happens in main, called on the last line, so a download cut short runs nothing.
main() {
local repo=https://github.com/DeeJanuz/frametop.git dir=$HOME/frametop branch= clone_only=0
local yes=0 tty=0 current= def answer
local pass=()
while [ $# -gt 0 ]; do
case $1 in
--stable) branch=main ;;
--experimental) branch=experimental ;;
--dir) dir=${2:?--dir needs a folder}; shift ;;
--clone-only) clone_only=1 ;;
--yes) yes=1; pass+=("$1") ;;
--no-bluetooth) pass+=("$1") ;;
-h|--help) usage; return 0 ;;
*) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
esac
shift
done
if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
echo "Frametop installs on a Steam Frame (SteamOS, VR variant). Run this in a terminal on the headset." >&2
return 1
fi
# Piped into bash, stdin is this script: the questions (here and install.sh's) read the terminal.
{ : </dev/tty; } 2>/dev/null && tty=1
if [ "$tty" = 0 ] && [ "$yes" = 0 ]; then
echo "This asks questions, and there's no terminal to ask in: run it in one, or add --yes." >&2
return 1
fi
if [ -e "$dir/.git" ]; then
git -C "$dir" remote get-url origin 2>/dev/null | grep -qi 'frametop' ||
{ echo "$dir is a git repo, but not Frametop's. Pick another folder with --dir." >&2; return 1; }
current=$(git -C "$dir" branch --show-current)
elif [ -e "$dir" ]; then
echo "$dir is there and isn't Frametop's repo. Move it, or pick another folder with --dir." >&2
return 1
fi
if [ -z "$branch" ]; then
def=main
[ "$current" = experimental ] && def=experimental
if [ "$yes" = 1 ]; then
branch=$def
else
echo "Which version of Frametop?"
echo " 1) stable: the main branch, tested releases"
echo " 2) experimental: the newest features, less tested"
[ -n "$current" ] && echo "(installed now: $current)"
read -r -p "Choose 1 or 2 [$([ "$def" = main ] && echo 1 || echo 2)]: " answer </dev/tty || answer=
case ${answer:-$def} in
1|main|s*) branch=main ;;
2|experimental|e*) branch=experimental ;;
*) echo "not 1 or 2: $answer" >&2; return 2 ;;
esac
fi
fi
if [ ! -e "$dir" ]; then
echo "Cloning Frametop ($branch) into $dir"
git clone --branch "$branch" "$repo" "$dir"
else
if ! git -C "$dir" diff --quiet || ! git -C "$dir" diff --cached --quiet; then
echo "$dir has changes of its own. Commit or stash them first (git -C $dir status)." >&2
return 1
fi
echo "Updating $dir to the latest $branch"
git -C "$dir" fetch --quiet origin
if [ "$current" != "$branch" ]; then
if git -C "$dir" show-ref --verify --quiet "refs/heads/$branch"; then
git -C "$dir" switch --quiet "$branch"
else
git -C "$dir" switch --quiet --track -c "$branch" "origin/$branch"
fi
fi
git -C "$dir" merge --ff-only --quiet "origin/$branch" ||
{ echo "$dir has commits of its own on $branch, so it can't just move to the latest. Update it by hand." >&2; return 1; }
fi
echo "Frametop $branch: $(git -C "$dir" log -1 --format='%h %s')"
if [ "$clone_only" = 1 ]; then
echo "Install with: cd $dir && ./install.sh"
return 0
fi
cd "$dir"
if [ "$tty" = 1 ]; then
./install.sh "${pass[@]}" </dev/tty
else
./install.sh "${pass[@]}" </dev/null
fi
}
main "$@"
+2
View File
@@ -1,3 +1,5 @@
# Model sources that tools/convert_models.py downloads; the converted ncnn models are kept # Model sources that tools/convert_models.py downloads; the converted ncnn models are kept
models/onnx/ models/onnx/
models/*.task models/*.task
# frame-job settings (where replays run, and the lab's capture folder): personal, not shipped
.frame-job
+76 -24
View File
@@ -1,28 +1,47 @@
# Hands (experimental) # Hands (experimental, deferred)
Hand tracking from the headset's own cameras. It serves two things in Frametop: Hand tracking from the headset's own cameras. It's deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it and the README doesn't list it. It still builds and runs, installed by hand (below), for working on it.
It serves two things in Frametop:
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro. - **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro.
- **Pinches:** look at something and pinch to click it, pinch and move to drag, with the eye tracker doing the looking (`gaze/`). The tracker publishes the pinches. The pointer helper doesn't read them yet. - **Pinches and grips:** with `POINTER_HANDS=1`, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (`gaze/`), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.
Two programs, each a user service that starts and stops with SteamVR: Two programs, each a user service that stops when SteamVR does:
- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host. - `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container. - `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
``` ```
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), enable ft-handsctl on | off # on the Frame: start or stop hand tracking (SteamVR must be running)
hands/run.sh status # the services, and ft-hands' last status lines ft-handsctl status # the services, and ft-hands' last status lines
hands/run.sh log [lines] ft-handsctl log [lines]
ft-handsctl cutouts on|off|state # ft-screens' hand cutouts, without stopping tracking
ft-handsctl gestures # pinches and grips, live (tools/watch_gestures.py --distance)
hands/ft-cutouts on | off | status # the cutouts only: no pinches or grips
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), install disabled
hands/run.sh start|stop # start or stop the services
hands/run.sh restart # after changing a setting hands/run.sh restart # after changing a setting
hands/run.sh status
hands/run.sh log [lines]
hands/run.sh caps # after rebuilding ft-camd (a rebuild clears its capabilities) hands/run.sh caps # after rebuilding ft-camd (a rebuild clears its capabilities)
hands/run.sh uninstall hands/run.sh uninstall
``` ```
Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them): Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them), read when ft-camd and ft-hands start:
- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`. - `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below). - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
The pointer helper's `POINTER_HANDS` and `POINTER_PINCH_*`/`POINTER_GRIP_*` settings are in "Pinches and grips in the pointer" below.
Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/user/UID/frametop/`, which the desktop session deletes whenever it starts): Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/user/UID/frametop/`, which the desktop session deletes whenever it starts):
@@ -30,9 +49,9 @@ Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/us
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` | | `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` |
| `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) | | `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) |
| `gestures` | ft-hands | `include/fh_gestures.h` | `tools/watch_gestures.py`; the pointer helper, later | | `gestures` | ft-hands | `include/fh_gestures.h` | the pointer helper (`pointer/helper/pointer_holds.cpp`), `tools/watch_gestures.py` |
The source keeps the `fh_` names and magic strings of frame-hands, where this was developed (`~/Desktop/Projects/frame-hands` on the developer's Frame, which keeps the recordings, probes and Python prototype). So its recordings and tools still work. The source keeps the `fh_` names and magic strings of frame-hands, the project it started as, so recordings made with it still work.
## ft-camd ## ft-camd
@@ -45,7 +64,7 @@ Polling buffers for changes can catch a frame while the camera is still writing
- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart. - The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart.
- It also re-maps an index on the fly when its buffer holds no new frame. - It also re-maps an index on the fly when its buffer holds no new frame.
**Privileges.** Setting up needs three things. `pidfd_getfd` on XRService needs `CAP_SYS_PTRACE`, because the Frame has `ptrace_scope=1`. The system-wide tracepoint needs `CAP_PERFMON`, because `perf_event_paranoid` is 2. Its format files are root-only, which needs `CAP_DAC_READ_SEARCH`. `hands/run.sh install` gives the binary those capabilities with `sudo setcap`. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under `sudo`, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers. **Privileges.** Setting up needs three things. `pidfd_getfd` on XRService needs `CAP_SYS_PTRACE`, because the Frame has `ptrace_scope=1`. The system-wide tracepoint needs `CAP_PERFMON`, because `perf_event_paranoid` is 2. Its format files are root-only, which needs `CAP_DAC_READ_SEARCH`. `hands/run.sh install` gives the binary those capabilities with `sudo setcap`. File capabilities need a filesystem mounted without `nosuid`. The Frame's `/home` (ext4) has no `nosuid`. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under `sudo`, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers.
The ring is mode 0600, in a folder only you can write. Frame handling: The ring is mode 0600, in a folder only you can write. Frame handling:
@@ -55,11 +74,13 @@ The ring is mode 0600, in a folder only you can write. Frame handling:
Options: Options:
- `--dark R`: a frame dimmer than R times the camera's recent brightest counts as near-black. Default 0.4.
- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands. - `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands.
- `--with-color` (the service uses it): also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232). They run at `--color-idle` (2 fps), enough for ft-hands to tell how bright it is, until a reader asks for more in `/run/user/UID/frametop-hands/color-fps` (ft-hands writes 30 while it tracks or records with them), up to `--color-fps` (30; the cameras run at 60). `HANDS_CAMERAS=mono` leaves them out. Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core. - `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. The service leaves it off and runs the mono cameras only (see "Known issues"). To try the colour cameras, add it to `ExecStart` in `hands/frametop-camd.service` and run `hands/run.sh install` again; ft-hands then picks the cameras by the light. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232). They run at `--color-idle` (2 fps), enough for ft-hands to tell how bright it is, until a reader asks for more in `/run/user/UID/frametop-hands/color-fps` (ft-hands writes 30 while it tracks or records with them), up to `--color-fps` (30; the cameras run at 60). `HANDS_CAMERAS=mono` leaves them out. Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
- Each mono camera's latest near-black frame's mean goes in the ring (`dark_mean`): a short fixed exposure, so it follows the room's IR light, sunlight above all. - Each mono camera's latest near-black frame's mean goes in the ring (`dark_mean`): a short fixed exposure, so it follows the room's IR light, sunlight above all.
- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras. - The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras.
- Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping while the headset was worn: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn probed all 32 colour buffers, a whole-buffer cache sync each, which also made the mono cameras miss frames. Runs with the headset idle had none of this. So the passthrough compositor may be writing into the colour buffers while Room View shows. Since then a colour camera never takes the mono ones down: it probes at most 4 buffers a frame, and one that goes stale twice in a row is paused (10 s, doubling up to 160 s) and learned again, without ft-camd exiting. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room. `FT_CAMD_DEBUG=1` prints, at each colour stale frame, how many sampled words changed in every candidate buffer. - Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping while the headset was worn: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn probed all 32 colour buffers, a whole-buffer cache sync each, which also made the mono cameras miss frames. Runs with the headset idle had none of this. So the passthrough compositor may be writing into the colour buffers while Room View shows. Since then a colour camera never takes the mono ones down: it probes at most 4 buffers a frame, and one that goes stale twice in a row is paused (10 s, doubling up to 160 s) and learned again, without ft-camd exiting. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room.
- `FT_CAMD_DEBUG=1` in the environment: at each stale colour frame, ft-camd logs to stderr the camera, the frame's time, V4L2 index and sequence number, and, for every candidate buffer, how many of its sampled words changed, in all and in the last eighth of the samples.
- `--sensor S`: only the mono cameras whose sensor name contains S. - `--sensor S`: only the mono cameras whose sensor name contains S.
- `--status S`: a status line every S seconds (0: never). - `--status S`: a status line every S seconds (0: never).
@@ -87,18 +108,20 @@ Options:
- `--models DIR`: where the models are. - `--models DIR`: where the models are.
- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs. - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
- `--no-publish`: don't write the hands and gestures files. - `--no-publish`: don't write the hands and gestures files.
- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them. - `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording. - `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence. - `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
- `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s. - `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s.
- `--cams auto|mono|color|all` (`HANDS_CAMERAS`, default `auto`): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. `auto` goes by the colour frames' mean brightness: at `--bright-on` (`HANDS_BRIGHT_ON`, 40) or over for 2 s it tracks with `--bright` (`HANDS_BRIGHT`: `all`, every camera, the default, or `color`), and under `--bright-off` (`HANDS_BRIGHT_OFF`, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (`cameras: mono -> all (...)`), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame. - `--cams auto|mono|color|all` (`HANDS_CAMERAS`, default `auto`): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. `auto` goes by the colour frames' mean brightness: at `--bright-on` (`HANDS_BRIGHT_ON`, 40) or over for 2 s it tracks with `--bright` (`HANDS_BRIGHT`: `all`, every camera, the default, or `color`), and under `--bright-off` (`HANDS_BRIGHT_OFF`, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (`cameras: mono -> all (...)`), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame. With no colour cameras in the ring (ft-camd without `--with-color`, as the service runs it), ft-hands tracks with the mono cameras whatever this says.
- `--color-left NODE` (`HANDS_COLOR_LEFT`, `color_video0`) and `--color-crop subtract|none` (`HANDS_COLOR_CROP`, `subtract`): how the colour module's calibration maps onto the images. Not settled yet: `tools/check_color.py` on a recording with a lit, textured view tells. - `--color-left NODE` (`HANDS_COLOR_LEFT`, `color_video0`) and `--color-crop subtract|none` (`HANDS_COLOR_CROP`, `subtract`): how the colour module's calibration maps onto the images. Not settled yet: `tools/check_color.py` on a recording with a lit, textured view tells.
- `--grip-begin R`, `--grip-end R`: the grip detector (below). - `--grip-begin R`, `--grip-end R`: the grip detector (below). Defaults 1.2 and 1.45.
- `--gesture-log`: print what the pinch and grip detectors measure, 10 times a second: each hand's thumb-to-index distance (world and triangulated), its palm-down reading and its finger curl.
**Gestures** (`/run/user/UID/frametop-hands/gestures`, `include/fh_gestures.h`), for the pointer helper: **Gestures** (`/run/user/UID/frametop-hands/gestures`, `include/fh_gestures.h`), for the pointer helper:
- A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm. Not begun with the palm facing down (`--pinch-palm-down`, 0.6), which is how typing looks. - A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm (see "Pinch" below).
- A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes. A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it; the pointer helper ignores grips that begin more than 30 cm below the eyes, which it can tell and ft-hands can't. - A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes, and not with the thumb within 3 cm of the index tip (that's a pinch with the other fingers curled). A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it. The pointer helper ignores grips that begin more than `POINTER_GRIP_BELOW` (0.35 m) below the eyes, which it can tell and ft-hands can't.
- `tools/watch_gestures.py --distance` shows both live; `ft-handreplay --timeline` logs them and each hand's finger curl. - `tools/watch_gestures.py --distance` shows both live; `ft-handreplay --timeline` logs them and each hand's finger curl.
The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten. The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
@@ -125,13 +148,30 @@ How good the depth is, measured from recordings (2026-09-30, `--depth` below): t
ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`. ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.
- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost). - A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute before the palm check below. - The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute (see the next point).
- No pinch begins while the palm faces down (`--pinch-palm-down MAX`: the palm normal's share of the head's up axis, default 0.6; 1 turns it off), and a close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index. In the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. The limit held back every typing pinch and none of the deliberate ones. Looking down tilts the head frame, which lowers the reading for a hand on a keyboard, so the consumer's gaze check stays the other guard. - `--pinch-palm-down MAX` holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and no pinch begins within `POINTER_PINCH_TYPING` of one.
- A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once. - A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once.
- The pinch point is midway between the thumb and index tips. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times. - The pinch point is between the index and middle knuckles, which hold still while the fingers open and close. The tips' midpoint moved 1-2 cm as a pinch opened, which dragged every release off its press. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
- `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance. - `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance.
The pointer helper is the natural consumer. Its gaze mode already treats a press as "stop where the gaze put it, drag onto the target, click on release", and "hold still for half a second, then move" as a drag. A pinch begin would be the press, the end the release, and the pinch point's movement the drag. ## Pinches and grips in the pointer
With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/pointer_holds.cpp`) reads the gestures file every frame. It's off by default.
- **Pinch to click.** In gaze mode a pinch works like the mouse's press: the pointer stops where the gaze put it, and the click comes when the pinch opens, where the pointer is then. A quick tap clicks where you looked. Held, the pinching hand moves the pointer to correct the gaze, and the correction is a lesson for the gaze tracker, as with the mouse.
- **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.
- **Grip to drag.** Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected.
- A pinch ended by losing the hand, or by a grip taking over, doesn't click.
- The hand's movement is taken in the room, from where the eye was when the gesture began, so turning your head doesn't move the pointer. The first gesture while the pointer is off only wakes it. Gestures are ignored in a VR game (unless the dashboard is up), with the headset off, and while the mouse's button is held.
Settings in `~/.config/frametop.conf`:
- `POINTER_HANDS` (0): 1 turns pinches and grips on.
- `POINTER_PINCH_GAIN` (0.5): a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision.
- `POINTER_PINCH_DEADZONE` (1.5): how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it.
- `POINTER_GRIP_GAIN` (1): a grip moves the pointer this many times the hand's angle.
- `POINTER_GRIP_BELOW` (0.35): grips that begin more than this many metres below the eyes are ignored, because hands resting on a desk curl like a loose fist. Pinches have no such limit: deliberate ones sat 0.35-0.45 m below the eyes with the elbow resting.
- `POINTER_PINCH_TYPING` (1): no pinch begins within this many seconds of a key press, because typing touches thumb to index.
## Recordings ## Recordings
@@ -155,15 +195,27 @@ hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --co
## Tools ## Tools
Python, with NumPy and OpenCV (in the dev container: `python3-numpy`, `python3-opencv`, which `setup/dev-container.sh` installs). Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files. Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, because Fedora's `python3-opencv` pulls in over a gigabyte; in the dev container, run `sudo dnf install python3-numpy python3-opencv` once. Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files.
- `tools/check_sides.py --ring` (or a recording): are the side cameras named right? - `tools/check_sides.py --ring` (or a recording): are the side cameras named right?
- `tools/check_color.py REC`: how the colour module's calibration maps onto its images. - `tools/check_color.py REC`: how the colour module's calibration maps onto its images.
- `tools/show_set.py REC`: a recording's frame sets as images. - `tools/show_set.py REC`: a recording's frame sets as images.
- `tools/watch_gestures.py [--distance]`: pinches, live. - `tools/watch_gestures.py [--distance]`: pinches and grips, live.
- `tools/depth_report.py DEPTH`: the depth measures above. - `tools/depth_report.py DEPTH`: the depth measures above.
- `tools/cut_sets.py REC OUT [--sets N | --at I,J,...]`: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
- `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`). - `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens.
## Build ## Build
`hands/build.sh` builds in the dev container on the Frame, into `hands/build/`, with `hands/Makefile`. The first build fetches ncnn at a pinned tag and builds it into `hands/build/ncnn`, which takes a few minutes; `NCNN=DIR` points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container. `hands/build.sh` builds in the dev container on the Frame, into `hands/build/`, with `hands/Makefile`. The first build fetches ncnn at a pinned tag (`NCNN_TAG` in the Makefile) and builds it into `hands/build/ncnn`, which takes a few minutes; `NCNN=DIR` points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container.
## Known issues
- **The side cameras can come out swapped.** ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and some XRService restarts reverse it. For now it's caught by hand: `tools/check_sides.py --ring`, then `HANDS_SWAP_SIDES=1`. It needs a fix in ft-camd, or at least an automatic check when it starts.
- **The colour cameras can't be used while the headset is worn.** The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
- **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
- **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
- **Pinches aren't reliable enough for everyday use yet.** That's why hand tracking stays off until `ft-handsctl on`, and `POINTER_HANDS` is 0 by default.
- **Floating windows don't get hand cutouts.** Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (`screens/vr.cpp`, `UpdateCutouts`).
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# ft-cutouts: the hand cutouts without pinches and grips. Your hands show through Frametop's
# screens, but ft-hands runs with --no-gestures, so nothing clicks or drags. ft-handsctl on is
# the full version; each stops the other. Like it, this doesn't start with SteamVR, and it
# stops when SteamVR does. On the Frame.
#
# ft-cutouts on | off | status
#
# Runs this checkout's hands/build as transient user units, so it doesn't need hands/run.sh
# install: only a build (hands/build.sh) and ft-camd's capabilities (hands/run.sh caps).
set -euo pipefail
here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
camd=frametop-cutouts-camd.service hands=frametop-cutouts-hands.service
die() { echo "$*" >&2; exit 1; }
run_unit() { # unit, the ft-handsctl unit it replaces, description, then the command
local unit=$1 conflicts=$2 what=$3
shift 3
systemctl --user -q is-active "$unit" && return
systemd-run --user --quiet --collect --unit="$unit" --description="Frametop hand cutouts: $what" \
-p PartOf=steamvr.service -p After=steamvr.service -p Conflicts="$conflicts" \
-p Restart=always -p RestartSec=5 -p TimeoutStopSec=5 "$@"
}
case ${1:-status} in
on)
systemctl --user -q is-active steamvr.service || die "SteamVR isn't running"
[ -x "$here/build/ft-camd" ] && [ -x "$here/build/ft-hands" ] || die "not built: hands/build.sh"
grep -qa -- --no-gestures "$here/build/ft-hands" || die "ft-hands predates --no-gestures: hands/build.sh"
getcap "$here/build/ft-camd" | grep -q cap_sys_ptrace ||
die "ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)"
if systemctl --user -q is-active frametop-camd.service frametop-hands.service; then
echo "stopping ft-handsctl's hand tracking (it has pinches and grips)"
fi
run_unit $camd frametop-camd.service "the camera broker" "$here/build/ft-camd" --status 60
if ! systemctl --user -q is-active $hands; then
"$here/../scripts/container-up.sh" # so stopping the unit can't take the container down
pkill -x ft-hands || true
fi
run_unit $hands frametop-hands.service "hand tracking, no gestures" "$HOME/.local/bin/distrobox" enter dev -- \
"$here/build/ft-hands" --status 60 --no-gestures
"$here/ft-handsctl" cutouts on >/dev/null || true
sleep 4
"$0" status ;;
off)
systemctl --user stop $hands $camd 2>/dev/null || true
systemctl --user -q is-active frametop-hands.service || pkill -x ft-hands || true
echo "hand cutouts off" ;;
status)
for u in $camd $hands; do echo "$u: $(systemctl --user is-active $u || true)"; done
echo "ft-screens cutouts: $("$here/ft-handsctl" cutouts state 2>&1)"
inv=$(systemctl --user show -p InvocationID --value $hands) # this run's lines only
[ -n "$inv" ] && journalctl --user _SYSTEMD_INVOCATION_ID="$inv" --no-pager -o cat |
grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
*) sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
esac
+6 -15
View File
@@ -5,9 +5,9 @@
# Usage: hands/run.sh install|uninstall # Usage: hands/run.sh install|uninstall
# hands/run.sh caps # give ft-camd its capabilities again (a rebuild clears them) # hands/run.sh caps # give ft-camd its capabilities again (a rebuild clears them)
# hands/run.sh start|stop|restart|status|log [lines] # hands/run.sh start|stop|restart|status|log [lines]
# install and caps need the password (sudo setcap, once per build of ft-camd). On the Frame, # install and caps need the password (sudo setcap, once per build of ft-camd): it's asked in
# sudo asks in the terminal. From a PC (or with no terminal), the password comes from # the terminal, on the Frame or from a PC (frame_sudo in scripts/_env.sh, which also takes it
# steamos_root_pwd in the repo's .env and is sent to sudo -S on stdin, never on a command line. # from the repo's .env).
set -euo pipefail set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
. "$root/scripts/_env.sh" . "$root/scripts/_env.sh"
@@ -17,17 +17,7 @@ units="frametop-camd.service frametop-hands.service"
# format files. ft-camd drops them all once it has set up. # format files. ft-camd drops them all once it has set up.
caps=cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep caps=cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep
sudo_run() { sudo_run() { frame_sudo "$1"; }
if [ "$FRAME_LOCAL" = 1 ] && [ -t 0 ]; then
sudo bash -c "$1" # asks for the password here
return
fi
local pw
pw=$(sed -n 's/^steamos_root_pwd=//p' "$root/.env" 2>/dev/null)
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
[ -n "$pw" ] || { echo "no terminal for sudo, and steamos_root_pwd is missing from $root/.env" >&2; exit 1; }
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
}
set_caps() { # only when missing: a rebuild clears them, a reinstall doesn't set_caps() { # only when missing: a rebuild clears them, a reinstall doesn't
local bin local bin
@@ -54,7 +44,8 @@ mkdir -p ~/.local/bin && ln -sfn $(printf %q "$FRAME_REPO/hands/ft-handsctl") ~/
$states; echo 'start it with: ft-handsctl on'" ;; $states; echo 'start it with: ft-handsctl on'" ;;
caps) set_caps ;; caps) set_caps ;;
uninstall) "$frame" --host "systemctl --user disable --now $units 2>/dev/null uninstall) "$frame" --host "systemctl --user disable --now $units 2>/dev/null
for u in $units; do rm -f ~/.config/systemd/user/\$u; done; systemctl --user daemon-reload; echo removed" ;; for u in $units; do rm -f ~/.config/systemd/user/\$u; done; systemctl --user daemon-reload
[ -L ~/.local/bin/ft-handsctl ] && rm -f ~/.local/bin/ft-handsctl; echo removed" ;;
start|stop|restart) "$frame" --host "systemctl --user $1 $units; $states" ;; start|stop|restart) "$frame" --host "systemctl --user $1 $units; $states" ;;
status) "$frame" --host "$states; journalctl --user -u frametop-hands.service --no-pager -o cat -n 4" || true ;; status) "$frame" --host "$states; journalctl --user -u frametop-hands.service --no-pager -o cat -n 4" || true ;;
log) "$frame" --host "journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n ${2:-30}" ;; log) "$frame" --host "journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n ${2:-30}" ;;
+20 -8
View File
@@ -3,9 +3,12 @@
// Python prototype: the same scheduling, with the models on a few threads. // Python prototype: the same scheduling, with the models on a few threads.
// //
// ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] // ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
// [--no-publish] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ... // [--no-publish] [--no-gestures] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
// (--help lists them all) // (--help lists them all)
// //
// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
//
// Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their // Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
// own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the // own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
// hands) they expose for the room and the hands come out dark. The two Arcturus colour // hands) they expose for the room and the hands come out dark. The two Arcturus colour
@@ -150,7 +153,7 @@ struct Lighting {
int main(int argc, char **argv) { int main(int argc, char **argv) {
double seconds = 0, status = 5; double seconds = 0, status = 5;
int threads = 3, niceness = 5; int threads = 3, niceness = 5;
bool int8 = false, publish = true, track = true, swap_sides = false; bool int8 = false, publish = true, track = true, swap_sides = false, gestures_on = true;
std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn"; std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME; std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
// SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
@@ -189,6 +192,7 @@ int main(int argc, char **argv) {
else if (a == "--nice" && more) niceness = std::atoi(argv[++i]); else if (a == "--nice" && more) niceness = std::atoi(argv[++i]);
else if (a == "--int8") int8 = true; else if (a == "--int8") int8 = true;
else if (a == "--no-publish") publish = false; else if (a == "--no-publish") publish = false;
else if (a == "--no-gestures") gestures_on = false;
else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]); else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]); else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
else if (a == "--pinch-triangulated") pinch_params.triangulated = true; else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
@@ -217,11 +221,12 @@ int main(int argc, char **argv) {
} }
else { else {
std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n" std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
" [--no-gestures] (hands for the cutouts only: no pinches or grips)\n"
" [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n" " [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n" " [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
" [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n" " [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
" [--color-left color_video0|color_video3] [--color-crop subtract|none]\n" " [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
" [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (0.6)\n" " [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
" [--grip-begin R] (1.2) [--grip-end R] (1.45) [--gesture-log]\n" " [--grip-begin R] (1.2) [--grip-end R] (1.45) [--gesture-log]\n"
" [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n" " [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n"
"Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n" "Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n"
@@ -265,12 +270,14 @@ int main(int argc, char **argv) {
return true; return true;
}; };
if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) || if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
(publish && (!pub.open(err) || !gestures.open(pinch, grip, err))) || (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
(!record.empty() && !start_recording(record, err))) { (!record.empty() && !start_recording(record, err))) {
std::fprintf(stderr, "%s\n", err.c_str()); std::fprintf(stderr, "%s\n", err.c_str());
return 1; return 1;
} }
if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1; if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
// so a reader can't take an earlier run's file for this one's
if (publish && !gestures_on) unlink((run_dir() + "/gestures").c_str());
std::map<std::string, int> index; // mono calibration name -> ring camera std::map<std::string, int> index; // mono calibration name -> ring camera
std::map<std::string, int> color; // colour calibration name (color_video<N>) -> ring camera std::map<std::string, int> color; // colour calibration name (color_video<N>) -> ring camera
@@ -541,9 +548,11 @@ int main(int argc, char **argv) {
const auto hands = tracker.step(images, int64_t(tmin)); const auto hands = tracker.step(images, int64_t(tmin));
const uint64_t capture = uint64_t(int64_t(tmin) - raw_off); // CLOCK_MONOTONIC const uint64_t capture = uint64_t(int64_t(tmin) - raw_off); // CLOCK_MONOTONIC
const std::vector<Seen> views = tracker.views_now(); const std::vector<Seen> views = tracker.views_now();
if (gestures_on) {
grip.update(hands, views, int64_t(capture)); grip.update(hands, views, int64_t(capture));
pinch.update(hands, views, int64_t(capture), grip.gripping()); pinch.update(hands, views, int64_t(capture), grip.gripping());
if (gesture_log && capture - t_glog >= 100'000'000) { }
if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
t_glog = capture; t_glog = capture;
for (int k = 0; k < 2; ++k) for (int k = 0; k < 2; ++k)
if (pinch.world_d[k] >= 0) if (pinch.world_d[k] >= 0)
@@ -554,7 +563,8 @@ int main(int argc, char **argv) {
// a gesture down or closing gets the full rate, even while the palm holds still // a gesture down or closing gets the full rate, even while the palm holds still
next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) - next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
0.005) * 1e9); 0.005) * 1e9);
if (publish) pub.write(hands, capture), gestures.write(pinch, grip, capture); if (publish) pub.write(hands, capture);
if (publish && gestures_on) gestures.write(pinch, grip, capture);
for (const Pinch::Event &e : grip.events) for (const Pinch::Event &e : grip.events)
std::printf("grip %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance, std::printf("grip %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
e.point[0], e.point[1], e.point[2]); e.point[0], e.point[1], e.point[2]);
@@ -593,6 +603,7 @@ int main(int argc, char **argv) {
s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten, s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
!rec ? "" : (" recorded " + std::to_string(rec->written()) + " dropped " + !rec ? "" : (" recorded " + std::to_string(rec->written()) + " dropped " +
std::to_string(rec->dropped())).c_str()); std::to_string(rec->dropped())).c_str());
if (gestures_on) {
std::printf(" pinches: left %u right %u (held back, palm down: %d %d) grips: left %u right %u", std::printf(" pinches: left %u right %u (held back, palm down: %d %d) grips: left %u right %u",
pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1], pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
grip.side(0).begins, grip.side(1).begins); grip.side(0).begins, grip.side(1).begins);
@@ -604,6 +615,7 @@ int main(int argc, char **argv) {
: "open", : "open",
pinch.side(k).distance, grip.curl[k]); pinch.side(k).distance, grip.curl[k]);
std::printf("\n"); std::printf("\n");
}
for (const Hand *h : hands) for (const Hand *h : hands)
std::printf(" hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m scale %.2f speed %.2f m/s\n", h->id, std::printf(" hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m scale %.2f speed %.2f m/s\n", h->id,
h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale, h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale,
@@ -616,8 +628,8 @@ int main(int argc, char **argv) {
} }
} }
if (!color.empty()) unlink(want_file.c_str()); if (!color.empty()) unlink(want_file.c_str());
if (publish) { if (publish) pub.write({}, mono_ns());
pub.write({}, mono_ns()); if (publish && gestures_on) {
pinch.release(int64_t(mono_ns())); // a drag in progress ends, as lost pinch.release(int64_t(mono_ns())); // a drag in progress ends, as lost
grip.release(int64_t(mono_ns())); grip.release(int64_t(mono_ns()));
gestures.write(pinch, grip, mono_ns()); gestures.write(pinch, grip, mono_ns());
+179 -53
View File
@@ -6,9 +6,12 @@ to the input relay over its control socket (@frametop_relay):
- Devices: every USB/Bluetooth mouse and keyboard, a live activity light to - Devices: every USB/Bluetooth mouse and keyboard, a live activity light to
identify them, and a role for each (3D pointer, pass through, ignore). identify them, and a role for each (3D pointer, pass through, ignore).
- Buttons: press a button or key on a pointer device, then pick an action. - Buttons: press a button or key on a pointer device, then pick an action.
- Controllers: the same for the Frame controllers' buttons. They're read by the pointer - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze
helper through SteamVR input (@ft_pointer_helper: vrstatus, vrglobal), and a mapped mode is a mouse feature). They're read by the pointer helper through SteamVR input
button is taken from games. (@ft_pointer_helper: vrstatus, vrglobal), and a mapped button is taken from games.
- Keyboard: when Frametop's keyboard opens, and key combinations for any action, a command
of your own too ("command:CMD"), or a modifier tapped alone (a Meta tap opens Steam's menu
and Meta+Shift+F floats a window, unless the rules have their own list).
- Pointer: speed, dot size, distance and the rest, applied live. - Pointer: speed, dot size, distance and the rest, applied live.
- Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or - Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or
one by one. The helper lists them (@ft_pointer_helper "overlays"). one by one. The helper lists them (@ft_pointer_helper "overlays").
@@ -58,13 +61,18 @@ DEFAULT_BUTTONS = {0x110: "left", 0x111: "right", 0x112: "middle", 0x113: "back"
ACTION_LABELS = { ACTION_LABELS = {
"left": "Left click", "right": "Right click", "middle": "Middle click", "back": "Back", "left": "Left click", "right": "Right click", "middle": "Middle click", "back": "Back",
"scroll_up": "Scroll up", "scroll_down": "Scroll down", "dashboard": "Toggle SteamVR dashboard", "scroll_up": "Scroll up", "scroll_down": "Scroll down", "dashboard": "Toggle SteamVR dashboard",
"steam_menu": "Open Steam menu / close dashboard",
"recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off", "recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off",
"follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)", "follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)",
"gaze_precision": "Gaze precision: hold to steer, release to click", "gaze_precision": "Gaze precision: hold to steer, release to click",
"gaze_drag": "Gaze drag: press where you look, steer, release", "gaze_drag": "Gaze drag: press where you look, steer, release",
"gaze_left": "Gaze left click: tap, or hold and turn your head to aim",
"gaze_right": "Gaze right click: tap, or hold and turn your head to aim",
"gaze_quickcal": "Gaze quick check (one dot)",
"sens_up": "Faster pointer", "sens_up": "Faster pointer",
"sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout", "sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout",
"screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard", "screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
"float_toggle": "Float window in VR / put it back", "dock_all": "Put all floating windows back",
"key": "Pass through as key", "key": "Pass through as key",
"none": "Do nothing", "none": "Do nothing",
} }
@@ -87,27 +95,79 @@ CONTROLLER_BUTTONS = {
"right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip", "right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
"right/thumbstick": "Right stick click", "right/thumbstick": "Right stick click",
} }
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")] # Gaze mode is a mouse and keyboard feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS]
# Key combinations take any action but key and none; the keyboard clicks only work there.
SHORTCUT_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")]
KEYBOARD_ONLY = ("gaze_left", "gaze_right")
# Profiles (docs/profiles.md): "profile:NAME" opens one (the relay runs ft-layout use NAME).
LAYOUT_PATH = os.path.expanduser("~/.config/frametop-layout.json")
PROFILE = "profile:"
# "command:CMD": the relay runs CMD with sh -c (key combinations only, here).
COMMAND = "command:"
def profile_actions():
"""One action per profile (named layout), for buttons, controllers, and key combinations."""
names = sorted(read_json(LAYOUT_PATH).get("layouts", {}), key=str.casefold)
return [PROFILE + n for n in names]
def action_label(a):
if a.startswith(PROFILE):
return f"Open profile {a[len(PROFILE):]}"
if a.startswith(COMMAND):
return f"Run: {a[len(COMMAND):]}"
return ACTION_LABELS.get(a, a)
def is_profile(a):
return a.startswith(PROFILE) and len(a) > len(PROFILE)
def is_command(a):
return a.startswith(COMMAND) and bool(a[len(COMMAND):].strip())
def mappable(a):
"""An action a controller button can have."""
return a in CONTROLLER_ACTIONS or is_profile(a)
def shortcut_mappable(a):
"""An action a key combination can have: the gaze ones and commands too."""
return a in SHORTCUT_ACTIONS or is_profile(a) or is_command(a)
# Gaze mode settings (pointer helper), like POINTER_SETTINGS. # Gaze mode settings (pointer helper), like POINTER_SETTINGS.
GAZE_SETTINGS = [ GAZE_SETTINGS = [
("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"), ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
("POINTER_GAZE_NUDGE_MAX", "Largest nudge to learn", 8, 1, 30, 0.5, "°"), ("POINTER_GAZE_NUDGE_MAX", "Largest correction to learn", 55, 1, 110, 1, "°"),
("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"), ("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"),
("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"), ("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
("POINTER_PRECISION_GAIN", "Precision steering", 0.5, 0.1, 2.0, 0.05, "×"),
("POINTER_PRECISION_DEADZONE", "Precision dead zone", 0.3, 0.0, 3.0, 0.1, "°"),
("POINTER_GAZE_DRAG_GAIN", "Drag steering", 1.0, 0.1, 2.0, 0.05, "×"),
] ]
# In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE). # In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE).
GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click", GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click",
"direct": "Click right away where the pointer is"} "direct": "Click right away where the pointer is"}
# The hand role the pointer's virtual controller takes (POINTER_ROLE).
POINTER_ROLES = {"right": "Right hand", "left": "Left hand", "stylus": "Stylus (no hand)"}
# Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code. # Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code.
MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125} MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"} MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. # What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
def key_bindings(rules):
"""The rules' key combinations, or the defaults if it has none of its own (an empty list
counts as its own)."""
bound = rules.get("key_bindings")
return dict(bound) if isinstance(bound, dict) else dict(DEFAULT_KEY_BINDINGS)
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. GAZE_TRACKER
# can also be auto (the default): ours when it's installed, else SteamVR's.
GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"} GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
# Our tracker's frame grabber (gaze/tracker/install.sh), on the host: this runs in the dev
# container, which has the host's /etc under /run/host.
EYEGRAB = ("/etc/frametop/ft-eyegrab", "/run/host/etc/frametop/ft-eyegrab")
GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"} GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"}
# Pointer settings: key, label, default, min, max, step, unit. # Pointer settings: key, label, default, min, max, step, unit.
POINTER_SETTINGS = [ POINTER_SETTINGS = [
@@ -250,12 +310,14 @@ class Backend(QObject):
self._capture_vr = False self._capture_vr = False
self._capture_combo = "" # the action a key combination is being captured for self._capture_combo = "" # the action a key combination is being captured for
self._combo_mods = set() self._combo_mods = set()
self._combo_tap = None # a modifier pressed alone, nothing since: released, it's a tap
self._vr = {} # the helper's vrstatus, {} when it doesn't answer self._vr = {} # the helper's vrstatus, {} when it doesn't answer
self._vr_at = 0.0 self._vr_at = 0.0
self._gaze = {} # ft-gazed's status, {} when it isn't running self._gaze = {} # ft-gazed's status, {} when it isn't running
self._gaze_prev = None # the status before, for rates self._gaze_prev = None # the status before, for rates
self._gaze_at = 0.0 self._gaze_at = 0.0
self._gaze_mode = None # the helper's gaze mode: True, False, None (no answer) self._gaze_mode = None # the helper's gaze mode: True, False, None (no answer)
self._check_asked = 0.0 # when quickcal or calibrate went to the gaze service (its errors)
self._driver_block = "" # set by _check_driver self._driver_block = "" # set by _check_driver
self._panels = None # SteamVR's overlays, from the helper; None until it answers self._panels = None # SteamVR's overlays, from the helper; None until it answers
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
@@ -319,6 +381,10 @@ class Backend(QObject):
except BlockingIOError: except BlockingIOError:
return return
text = data.decode(errors="replace") text = data.decode(errors="replace")
if text.startswith("error ") and self._check_asked and time.monotonic() - self._check_asked < 3:
self._check_asked = 0.0 # the gaze service's answer to quickcal or calibrate
self.message.emit("Gaze check: " + text[6:], True)
continue
if text in ("ok on", "ok off"): # the helper's answer to "gaze ?" (from an unbound socket) if text in ("ok on", "ok off"): # the helper's answer to "gaze ?" (from an unbound socket)
self._gaze_mode = text == "ok on" self._gaze_mode = text == "ok on"
self._gaze_at = time.monotonic() self._gaze_at = time.monotonic()
@@ -358,13 +424,16 @@ class Backend(QObject):
self.activity.emit(msg["id"]) self.activity.emit(msg["id"])
code, value = int(msg["code"]), msg["value"] code, value = int(msg["code"]), msg["value"]
if code in MODIFIER_CODES: if code in MODIFIER_CODES:
(self._combo_mods.add if value else self._combo_mods.discard)(MODIFIER_CODES[code]) mod = MODIFIER_CODES[code]
if value == 1:
self._combo_tap = None if self._combo_mods else mod
(self._combo_mods.add if value else self._combo_mods.discard)(mod)
if value == 0 and self._combo_tap == mod:
self._end_shortcut_capture(str(mod)) # the relay's modifier tap
elif value == 1 and code < BTN_MISC: elif value == 1 and code < BTN_MISC:
self._save_shortcut("+".join(str(c) for c in sorted(self._combo_mods) + [code]), self._end_shortcut_capture("+".join(str(c) for c in sorted(self._combo_mods) + [code]))
self._capture_combo) elif value == 1:
self._capture_combo = "" self._combo_tap = None # a mouse button: no tap
self._combo_mods = set()
self.shortcutCaptureChanged.emit()
elif t == "event": elif t == "event":
self.activity.emit(msg["id"]) self.activity.emit(msg["id"])
if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key" if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key"
@@ -471,7 +540,8 @@ class Backend(QObject):
@Property("QVariantList", constant=True) @Property("QVariantList", constant=True)
def actions(self): def actions(self):
return [{"value": k, "text": v} for k, v in ACTION_LABELS.items()] return [{"value": k, "text": v} for k, v in ACTION_LABELS.items() if k not in KEYBOARD_ONLY] + \
[{"value": a, "text": action_label(a)} for a in profile_actions()]
@Slot(str, result="QVariantList") @Slot(str, result="QVariantList")
def mappings(self, device_id): def mappings(self, device_id):
@@ -486,7 +556,7 @@ class Backend(QObject):
r = rows[code] r = rows[code]
r["isDefault"] = code in DEFAULT_BUTTONS # a built-in binding (left/right/middle/side/extra) r["isDefault"] = code in DEFAULT_BUTTONS # a built-in binding (left/right/middle/side/extra)
r["name"] = self.codeName(code) r["name"] = self.codeName(code)
r["actionLabel"] = ACTION_LABELS.get(r["action"], r["action"]) r["actionLabel"] = action_label(r["action"])
out.append(r) out.append(r)
return out return out
@@ -505,7 +575,7 @@ class Backend(QObject):
rules.setdefault("buttons", {}).setdefault(device_id, {})[str(code)] = action rules.setdefault("buttons", {}).setdefault(device_id, {})[str(code)] = action
self._remember_name(rules, device_id) self._remember_name(rules, device_id)
self._save_rules(rules) self._save_rules(rules)
self.message.emit(f"{self.codeName(code)} → {ACTION_LABELS.get(action, action)}", False) self.message.emit(f"{self.codeName(code)} → {action_label(action)}", False)
@Slot(str, int) @Slot(str, int)
def removeMapping(self, device_id, code): def removeMapping(self, device_id, code):
@@ -671,7 +741,7 @@ class Backend(QObject):
# --- controllers --- # --- controllers ---
@Property("QVariantList", constant=True) @Property("QVariantList", constant=True)
def controllerActions(self): def controllerActions(self):
return [{"value": a, "text": ACTION_LABELS[a]} for a in CONTROLLER_ACTIONS] return [{"value": a, "text": action_label(a)} for a in CONTROLLER_ACTIONS + profile_actions()]
@Property("QVariantList", constant=True) @Property("QVariantList", constant=True)
def controllerButtons(self): def controllerButtons(self):
@@ -681,7 +751,7 @@ class Backend(QObject):
def controllerMappings(self): def controllerMappings(self):
mapped = read_json(RULES_PATH).get("controller_buttons", {}) mapped = read_json(RULES_PATH).get("controller_buttons", {})
return [{"button": b, "label": label, "action": mapped[b], return [{"button": b, "label": label, "action": mapped[b],
"actionLabel": ACTION_LABELS.get(mapped[b], mapped[b])} "actionLabel": action_label(mapped[b])}
for b, label in CONTROLLER_BUTTONS.items() if b in mapped] for b, label in CONTROLLER_BUTTONS.items() if b in mapped]
@Property("QVariantMap", notify=controllersChanged) @Property("QVariantMap", notify=controllersChanged)
@@ -705,12 +775,12 @@ class Backend(QObject):
@Slot(str, str) @Slot(str, str)
def setControllerMapping(self, button, action): def setControllerMapping(self, button, action):
if button not in CONTROLLER_BUTTONS or action not in CONTROLLER_ACTIONS: if button not in CONTROLLER_BUTTONS or not mappable(action):
return return
rules = read_json(RULES_PATH) rules = read_json(RULES_PATH)
rules.setdefault("controller_buttons", {})[button] = action rules.setdefault("controller_buttons", {})[button] = action
self._save_rules(rules) self._save_rules(rules)
self.message.emit(f"{CONTROLLER_BUTTONS[button]} → {ACTION_LABELS[action]}", False) self.message.emit(f"{CONTROLLER_BUTTONS[button]} → {action_label(action)}", False)
@Slot(str) @Slot(str)
def removeControllerMapping(self, button): def removeControllerMapping(self, button):
@@ -775,6 +845,10 @@ class Backend(QObject):
def gazeServiceRunning(self): def gazeServiceRunning(self):
return bool(self._gaze) return bool(self._gaze)
@Property(bool, notify=gazeChanged)
def gazeServiceInstalled(self):
return os.path.exists(os.path.expanduser("~/.config/systemd/user/frametop-gaze.service"))
@Property(int, notify=gazeChanged) @Property(int, notify=gazeChanged)
def gazeMode(self): def gazeMode(self):
"""The helper's gaze mode now: 1 on, 0 off, -1 no answer (helper not running).""" """The helper's gaze mode now: 1 on, 0 off, -1 no answer (helper not running)."""
@@ -809,37 +883,46 @@ class Backend(QObject):
self.pointerChanged.emit() self.pointerChanged.emit()
self.message.emit(f"Mouse in gaze mode: {GAZE_MOUSE[mode].lower()}", False) self.message.emit(f"Mouse in gaze mode: {GAZE_MOUSE[mode].lower()}", False)
@Property(str, notify=pointerChanged) @Property(bool, notify=pointerChanged)
def pointerRole(self): def gazeMouseHeld(self):
v = read_conf().get("POINTER_ROLE", "right") return read_conf().get("POINTER_GAZE_MOUSE_MOVE", "held") != "free"
return v if v in POINTER_ROLES else "right"
@Property("QVariantList", constant=True) @Slot(bool)
def pointerRoles(self): def setGazeMouseHeld(self, on):
return [{"value": k, "text": v} for k, v in POINTER_ROLES.items()] write_conf_value("POINTER_GAZE_MOUSE_MOVE", "held" if on else "free")
@Slot(str)
def setPointerRole(self, role):
if role in POINTER_ROLES:
write_conf_value("POINTER_ROLE", role)
self.reload_timer.start() self.reload_timer.start()
self.pointerChanged.emit() self.pointerChanged.emit()
self.message.emit(f"Pointer role: {POINTER_ROLES[role].lower()} (from its next wake)", False) self.message.emit("Mouse in gaze mode: " + ("moves the pointer only while a button is held" if on
else "moves the pointer any time"), False)
@Property(bool, notify=pointerChanged)
def gazeDotAlways(self):
return read_conf().get("POINTER_GAZE_DOT", "always") != "moving"
@Slot(bool)
def setGazeDotAlways(self, on):
write_conf_value("POINTER_GAZE_DOT", "always" if on else "moving")
self.reload_timer.start()
self.pointerChanged.emit()
self.message.emit("Gaze dot: " + ("always shown" if on else "shown only while the mouse moves it"), False)
# --- key combinations ("key_bindings") --- # --- key combinations ("key_bindings") ---
def comboName(self, combo): def comboName(self, combo):
parts = [int(c) for c in combo.split("+") if c.isdigit()] parts = [int(c) for c in combo.split("+") if c.isdigit()]
if len(parts) == 1 and parts[0] in MODIFIER_NAMES:
return MODIFIER_NAMES[parts[0]] + " tap"
return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts) return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts)
@Property("QVariantList", notify=mappingsChanged) @Property("QVariantList", notify=mappingsChanged)
def keyShortcuts(self): def keyShortcuts(self):
bound = read_json(RULES_PATH).get("key_bindings", {}) or {} bound = key_bindings(read_json(RULES_PATH))
return [{"combo": c, "label": self.comboName(c), "action": a, "actionLabel": ACTION_LABELS.get(a, a)} return [{"combo": c, "label": self.comboName(c), "action": a, "actionLabel": action_label(a)}
for c, a in sorted(bound.items())] for c, a in sorted(bound.items())]
@Property("QVariantList", constant=True) @Property("QVariantList", constant=True)
def shortcutActions(self): def shortcutActions(self):
return [{"value": a, "text": ACTION_LABELS[a]} for a in CONTROLLER_ACTIONS] return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()] + \
[{"value": COMMAND, "text": "Run a command…"}]
@Property(bool, notify=shortcutCaptureChanged) @Property(bool, notify=shortcutCaptureChanged)
def capturingShortcut(self): def capturingShortcut(self):
@@ -847,9 +930,10 @@ class Backend(QObject):
@Slot(str) @Slot(str)
def startShortcutCapture(self, action): def startShortcutCapture(self, action):
if action in CONTROLLER_ACTIONS: if shortcut_mappable(action):
self._capture_combo = action self._capture_combo = action
self._combo_mods = set() self._combo_mods = set()
self._combo_tap = None
self._send("watch 60") self._send("watch 60")
self.shortcutCaptureChanged.emit() self.shortcutCaptureChanged.emit()
@@ -858,24 +942,48 @@ class Backend(QObject):
self._capture_combo = "" self._capture_combo = ""
self.shortcutCaptureChanged.emit() self.shortcutCaptureChanged.emit()
def _end_shortcut_capture(self, combo):
self._save_shortcut(combo, self._capture_combo)
self._capture_combo = ""
self._combo_mods = set()
self._combo_tap = None
self.shortcutCaptureChanged.emit()
def _save_shortcut(self, combo, action): def _save_shortcut(self, combo, action):
rules = read_json(RULES_PATH) rules = read_json(RULES_PATH)
rules.setdefault("key_bindings", {})[combo] = action rules["key_bindings"] = dict(key_bindings(rules), **{combo: action})
self._save_rules(rules) self._save_rules(rules)
self.message.emit(f"{self.comboName(combo)} → {ACTION_LABELS[action]}", False) self.message.emit(f"{self.comboName(combo)} → {action_label(action)}", False)
@Slot(str) @Slot(str)
def removeShortcut(self, combo): def removeShortcut(self, combo):
rules = read_json(RULES_PATH) rules = read_json(RULES_PATH)
(rules.get("key_bindings") or {}).pop(combo, None) rules["key_bindings"] = key_bindings(rules)
rules["key_bindings"].pop(combo, None)
self._save_rules(rules) self._save_rules(rules)
self.message.emit(f"{self.comboName(combo)} removed", False) self.message.emit(f"{self.comboName(combo)} removed", False)
@Property(str, notify=gazeChanged) @Property(str, notify=gazeChanged)
def gazeTracker(self): def gazeTracker(self):
"""Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER).""" """Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER; auto is
v = read_conf().get("GAZE_TRACKER", "steam") ours when it's installed)."""
return v if v in GAZE_TRACKERS else "steam" v = read_conf().get("GAZE_TRACKER", "auto")
if v in GAZE_TRACKERS:
return v
return "own" if self.gazeOwnInstalled else "steam"
@Property(bool, notify=gazeChanged)
def gazeTrackerAuto(self):
"""GAZE_TRACKER is auto (or missing): ours once it's installed, without a choice here."""
return read_conf().get("GAZE_TRACKER", "auto") not in GAZE_TRACKERS
@Property(bool, notify=gazeChanged)
def gazeOwnInstalled(self):
"""Is our own tracker installed? The gaze service says (it also checks ft-eyes' Python in
its checkout); without it, whether the frame grabber is."""
if "own_installed" in self._gaze:
return bool(self._gaze["own_installed"])
return any(os.path.exists(p) for p in EYEGRAB)
@Property(str, notify=gazeChanged) @Property(str, notify=gazeChanged)
def gazeEye(self): def gazeEye(self):
@@ -929,14 +1037,32 @@ class Backend(QObject):
self.message.emit("The gaze service isn't running (frametop-gaze.service)", True) self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
@Slot() @Slot()
def openGazeProbe(self): def gazeQuickCheck(self):
"""Calibrate in ft-gazeprobe (a GTK app on the host, fullscreen on a Frametop screen).""" """The gaze service's one-dot check, in the panel fixed to the headset."""
self._open_probe([], "Opening the gaze probe: calibrate there, then close it") self._gaze_check("quickcal", "Quick check: look at the dot in front of you")
@Slot() @Slot()
def openHeadsetFit(self): def gazeCalibrate(self):
"""The probe's Headset fit mode: how well the tracker sees each eye, as you adjust.""" """The full calibration in the panel fixed to the headset (gaze/gazecheck.py)."""
self._open_probe(["--mode", "fit"], "Opening the headset fit check in the gaze probe") self._gaze_check("calibrate", "Calibration: look at each dot in the headset; right click or Meta+K stops")
def _gaze_check(self, command, done):
if self._send(command, GAZED):
self._check_asked = time.monotonic()
self.message.emit(done, False)
else:
self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
@Slot()
def gazeFitCheck(self):
"""The headset fit check, in the panel fixed to the headset (gaze/gazecheck.py)."""
self._gaze_check("fitcheck", "Headset fit: in the headset, adjust it while you watch; right click or Meta+K closes it")
@Slot()
def openGazeProbe(self):
"""ft-gazeprobe, the gaze tracking's development tool (a GTK app on the host, fullscreen on
a Frametop screen). Day to day, the checks and the calibration run in the headset panel."""
self._open_probe([], "Opening the gaze probe (a development tool)")
def _open_probe(self, args, done): def _open_probe(self, args, done):
runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else [] runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else []
+154 -61
View File
@@ -192,7 +192,7 @@ Kirigami.ApplicationWindow {
wrapMode: Text.Wrap wrapMode: Text.Wrap
opacity: 0.7 opacity: 0.7
text: "Move or press a device to see which row it is. 3D pointer: grabbed, drives the SteamVR pointer. " text: "Move or press a device to see which row it is. 3D pointer: grabbed, drives the SteamVR pointer. "
+ "Pass through: left alone (a Meta tap toggles the dashboard if META_DASHBOARD=1). Ignore: left alone." + "Pass through: left alone, but its key combinations (Keyboard page) work. Ignore: left alone."
} }
} }
} }
@@ -538,6 +538,70 @@ Kirigami.ApplicationWindow {
Kirigami.FormData.label: "Keyboards connected:" Kirigami.FormData.label: "Keyboards connected:"
text: kpage.keyboards.length ? kpage.keyboards.map(d => d.name).join(", ") : "none" text: kpage.keyboards.length ? kpage.keyboards.map(d => d.name).join(", ") : "none"
} }
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Key combinations" }
Repeater {
model: backend.keyShortcuts
delegate: RowLayout {
required property var modelData
Kirigami.FormData.label: modelData.label + ":"
Controls.Label {
text: modelData.actionLabel
elide: Text.ElideRight
Layout.maximumWidth: Kirigami.Units.gridUnit * 24
}
Controls.ToolButton {
icon.name: "edit-delete"
display: Controls.AbstractButton.IconOnly
text: "Remove"
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.removeShortcut(modelData.combo)
}
}
}
RowLayout {
Kirigami.FormData.label: "New:"
Controls.ComboBox {
id: shortcutAction
model: backend.shortcutActions
textRole: "text"
valueRole: "value"
Component.onCompleted: currentIndex = indexOfValue("float_toggle")
Layout.preferredWidth: Kirigami.Units.gridUnit * 16
}
Controls.Button {
text: backend.capturingShortcut ? "Press the keys… (Cancel)" : "Set keys…"
enabled: backend.capturingShortcut || shortcutAction.currentValue !== "command:"
|| shortcutCommand.text.trim() !== ""
onClicked: backend.capturingShortcut ? backend.cancelShortcutCapture()
: backend.startShortcutCapture(
shortcutAction.currentValue === "command:"
? "command:" + shortcutCommand.text.trim()
: shortcutAction.currentValue)
}
}
Controls.TextField {
id: shortcutCommand
Kirigami.FormData.label: "Command:"
visible: shortcutAction.currentValue === "command:"
placeholderText: "e.g. ft-layout use Work"
Layout.preferredWidth: Kirigami.Units.gridUnit * 24
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
wrapMode: Text.WordWrap
text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. Or tap "
+ "one modifier on its own. The combination's last key isn't typed; the modifiers still reach "
+ "the app. While you're in Steam or a game, they see the keys too. Until you remove or change "
+ "them, a Meta tap opens the Steam menu (or closes the dashboard) instead of the desktop's "
+ "launcher, and Meta+Shift+F floats the desktop window under the pointer in VR, or puts it "
+ "back. A command runs with sh as the input relay's service, outside the desktop's session, "
+ "with Frametop's ft-layout, ft-float and ft-steam on its path; its output goes to the "
+ "relay's log (journalctl --user -u frametop-input-relay)."
opacity: 0.7
font: Kirigami.Theme.smallFont
}
} }
footer: Controls.Label { footer: Controls.Label {
@@ -772,16 +836,25 @@ Kirigami.ApplicationWindow {
property var status: backend.gazeStatus property var status: backend.gazeStatus
actions: [ actions: [
Kirigami.Action { Kirigami.Action {
text: "Calibrate…" text: "Quick check"
icon.name: "crosshairs" icon.name: "crosshairs"
tooltip: "Open the gaze probe to calibrate (fullscreen on a Frametop screen)" enabled: backend.gazeServiceRunning
onTriggered: backend.openGazeProbe() tooltip: "One dot in front of you in the headset: look at it, and the gaze tracker relearns where it sits"
onTriggered: backend.gazeQuickCheck()
}, },
Kirigami.Action { Kirigami.Action {
text: "Check headset fit…" text: "Calibrate"
icon.name: "crosshairs"
enabled: backend.gazeServiceRunning
tooltip: "The full calibration in the headset: dots in three rounds, dark to bright"
onTriggered: backend.gazeCalibrate()
},
Kirigami.Action {
text: "Check headset fit"
icon.name: "view-visible" icon.name: "view-visible"
tooltip: "How well the eye tracker sees each eye, and where it loses one, while you adjust the headset" enabled: backend.gazeServiceRunning
onTriggered: backend.openHeadsetFit() tooltip: "In the headset: how well the eye tracker sees each eye, live, while you adjust it"
onTriggered: backend.gazeFitCheck()
}, },
Kirigami.Action { Kirigami.Action {
text: "Reload calibration" text: "Reload calibration"
@@ -795,6 +868,14 @@ Kirigami.ApplicationWindow {
enabled: backend.gazeServiceRunning enabled: backend.gazeServiceRunning
tooltip: "Drop what your mouse nudges taught; the calibration stays" tooltip: "Drop what your mouse nudges taught; the calibration stays"
onTriggered: backend.forgetGazeLessons() onTriggered: backend.forgetGazeLessons()
},
Kirigami.Action {
// A development tool: the checks and the calibration run in the headset panel.
text: "Gaze probe (development)…"
icon.name: "tools"
displayHint: Kirigami.DisplayHint.AlwaysHide
tooltip: "The gaze tracking's lab tool, for developing it: experiments, accuracy tests, practice"
onTriggered: backend.openGazeProbe()
} }
] ]
@@ -821,6 +902,23 @@ Kirigami.ApplicationWindow {
opacity: 0.7 opacity: 0.7
font: Kirigami.Theme.smallFont font: Kirigami.Theme.smallFont
} }
Controls.Label {
// Why the gaze pointer, on, can't follow your eyes yet, so it isn't left looking
// like a plain mouse. The calibration part is the gaze service's (gaze/gazecheck.py).
readonly property var checks: gpage.status.checks || {}
readonly property string why: !backend.gazeServiceInstalled
? "The gaze service isn't installed: run gaze/run.sh install in the Frametop folder, in a terminal"
: !backend.gazeServiceRunning ? "The gaze service isn't running (it starts with SteamVR)"
: backend.gazeTracker === "own" && !gpage.status.eyegrab
? "Our eye tracker needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
: checks.problem || ""
visible: backend.gazeMode > 0 && why !== ""
text: why
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
wrapMode: Text.WordWrap
color: checks.check ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
font: Kirigami.Theme.smallFont
}
ColumnLayout { ColumnLayout {
Kirigami.FormData.label: "Mouse left button:" Kirigami.FormData.label: "Mouse left button:"
Repeater { Repeater {
@@ -833,69 +931,63 @@ Kirigami.ApplicationWindow {
} }
} }
} }
Controls.Label { Controls.Switch {
Layout.maximumWidth: Kirigami.Units.gridUnit * 30 Kirigami.FormData.label: "Mouse movement:"
wrapMode: Text.WordWrap text: "Only while a button is held (recommended)"
text: "Controllers: map a button to Gaze precision (hold, point the controller to steer, release to " checked: backend.gazeMouseHeld
+ "click) or Gaze drag (the same, pressed at once) on the Controllers page. Gaze pointer on/off " onToggled: backend.setGazeMouseHeld(checked)
+ "can go on a controller button, a mouse button, or a key combination below."
opacity: 0.7
font: Kirigami.Theme.smallFont
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Key combinations" }
Repeater {
model: backend.keyShortcuts
delegate: RowLayout {
required property var modelData
Kirigami.FormData.label: modelData.label + ":"
Controls.Label { text: modelData.actionLabel }
Controls.ToolButton {
icon.name: "edit-delete"
display: Controls.AbstractButton.IconOnly
text: "Remove"
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered Controls.ToolTip.visible: hovered
onClicked: backend.removeShortcut(modelData.combo) Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
} Controls.ToolTip.text: checked
} ? "Your eyes move the pointer, so moving the mouse on its own does nothing. Hold a button "
} + "and the pointer stops where you look: move onto the target and let go to click there "
RowLayout { + "(left or right). To drag after moving, press right while holding left. It's on because a "
Kirigami.FormData.label: "New:" + "bumped mouse can't pull the pointer away, and every mouse move is a real correction the "
Controls.ComboBox { + "eye tracker learns from. If the eye tracker stops, the mouse works as usual."
id: shortcutAction : "The mouse moves the pointer any time, as without gaze. Moves that weren't corrections can "
model: backend.shortcutActions + "teach the eye tracker the wrong thing."
textRole: "text"
valueRole: "value"
Component.onCompleted: currentIndex = indexOfValue("gaze_toggle")
Layout.preferredWidth: Kirigami.Units.gridUnit * 16
}
Controls.Button {
text: backend.capturingShortcut ? "Press the keys… (Cancel)" : "Set keys…"
onClicked: backend.capturingShortcut ? backend.cancelShortcutCapture()
: backend.startShortcutCapture(shortcutAction.currentValue)
} }
Controls.Switch {
Kirigami.FormData.label: "Gaze dot:"
text: "Always shown (off: only while the mouse moves it)"
checked: backend.gazeDotAlways
onToggled: backend.setGazeDotAlways(checked)
} }
Controls.Label { Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 30 Layout.maximumWidth: Kirigami.Units.gridUnit * 30
wrapMode: Text.WordWrap wrapMode: Text.WordWrap
text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. The " text: "Gaze works with the mouse and the keyboard. Keyboard clicks (Meta+J left, Meta+K right by "
+ "combination's last key isn't typed; the modifiers still reach the app." + "default): tap to click where you look, or hold, turn your head until the dot sits on the "
+ "target, and let go; hold still to drag. The Frame controllers don't take part, and moving "
+ "one hands the pointer back to them. Gaze pointer on/off, Gaze precision, and Gaze drag can "
+ "also go on a mouse button (Buttons page) or another key combination (Keyboard page)."
opacity: 0.7 opacity: 0.7
font: Kirigami.Theme.smallFont font: Kirigami.Theme.smallFont
} }
RowLayout { RowLayout {
Kirigami.FormData.label: "Eye tracker:" Kirigami.FormData.label: "Eye tracker:"
Controls.RadioButton {
text: "Own tracker (recommended)"
checked: backend.gazeTracker === "own"
onToggled: if (checked) backend.setGazeTracker("own")
}
Controls.RadioButton { Controls.RadioButton {
text: "SteamVR" text: "SteamVR"
checked: backend.gazeTracker === "steam" checked: backend.gazeTracker === "steam"
onToggled: if (checked) backend.setGazeTracker("steam") onToggled: if (checked) backend.setGazeTracker("steam")
} }
Controls.RadioButton {
text: "Own tracker"
checked: backend.gazeTracker === "own"
onToggled: if (checked) backend.setGazeTracker("own")
} }
Controls.Label {
// Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
+ "Frametop folder, in a terminal (asks for sudo)"
+ (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
: ", then pick Own tracker here and calibrate it once.")
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
wrapMode: Text.WordWrap
opacity: 0.7
font: Kirigami.Theme.smallFont
} }
Controls.Label { Controls.Label {
// The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber // The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber
@@ -904,7 +996,7 @@ Kirigami.ApplicationWindow {
&& (!gpage.status.own_running || gpage.status.own_reseat || !gpage.status.calibration_samples) && (!gpage.status.own_running || gpage.status.own_reseat || !gpage.status.calibration_samples)
text: !gpage.status.eyegrab ? "Needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)" text: !gpage.status.eyegrab ? "Needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
: !gpage.status.own_running ? "Starting…" : !gpage.status.own_running ? "Starting…"
: !gpage.status.calibration_samples ? "Not calibrated: use Calibrate… with Own tracker" : !gpage.status.calibration_samples ? "Not calibrated: use Calibrate"
: "The headset was off: your first nudge and click resets where it sits" : "The headset was off: your first nudge and click resets where it sits"
color: gpage.status.own_running ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor color: gpage.status.own_running ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
font: Kirigami.Theme.smallFont font: Kirigami.Theme.smallFont
@@ -976,7 +1068,8 @@ Kirigami.ApplicationWindow {
Kirigami.FormData.label: "Service:" Kirigami.FormData.label: "Service:"
text: backend.gazeServiceRunning text: backend.gazeServiceRunning
? (gpage.status.ft_gaze ? "running" : "running, eye tracker reader restarting") ? (gpage.status.ft_gaze ? "running" : "running, eye tracker reader restarting")
: "not running (frametop-gaze.service, starts with SteamVR)" : backend.gazeServiceInstalled ? "not running (frametop-gaze.service, starts with SteamVR)"
: "not installed: run gaze/run.sh install in the Frametop folder, in a terminal"
color: backend.gazeServiceRunning ? Kirigami.Theme.textColor : Kirigami.Theme.negativeTextColor color: backend.gazeServiceRunning ? Kirigami.Theme.textColor : Kirigami.Theme.negativeTextColor
} }
Controls.Label { Controls.Label {
@@ -992,7 +1085,7 @@ Kirigami.ApplicationWindow {
+ (gpage.status.one_eye_share > 0.5 ? " · only one eye tracked" : "") + (gpage.status.one_eye_share > 0.5 ? " · only one eye tracked" : "")
color: gpage.status.one_eye_share > 0.5 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor color: gpage.status.one_eye_share > 0.5 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor
Controls.ToolTip.text: "Only one eye tracked: the gaze comes from the other eye, a little less " Controls.ToolTip.text: "Only one eye tracked: the gaze comes from the other eye, a little less "
+ "precisely. Check headset fit… shows where the tracker loses it." + "precisely. Check headset fit shows where the tracker loses it."
Controls.ToolTip.visible: gpage.status.one_eye_share > 0.5 && ghover.hovered Controls.ToolTip.visible: gpage.status.one_eye_share > 0.5 && ghover.hovered
HoverHandler { id: ghover } HoverHandler { id: ghover }
} }
@@ -1016,7 +1109,7 @@ Kirigami.ApplicationWindow {
? gpage.status.calibration_samples + " points (" ? gpage.status.calibration_samples + " points ("
+ (gpage.status.tracker === "own" ? "Own tracker, " + gpage.status.calibration_made + (gpage.status.tracker === "own" ? "Own tracker, " + gpage.status.calibration_made
: gpage.status.kind === "eyes" ? gpage.status.model + ", each eye" : gpage.status.model) + ")" : gpage.status.kind === "eyes" ? gpage.status.model + ", each eye" : gpage.status.model) + ")"
: "none yet: use Calibrate…" : "none yet: use Calibrate"
} }
Controls.Label { Controls.Label {
visible: backend.gazeServiceRunning visible: backend.gazeServiceRunning
@@ -1030,13 +1123,13 @@ Kirigami.ApplicationWindow {
padding: Kirigami.Units.largeSpacing padding: Kirigami.Units.largeSpacing
wrapMode: Text.Wrap wrapMode: Text.Wrap
opacity: 0.7 opacity: 0.7
text: "Map a mouse button (Buttons) or a controller button (Controllers) to \"Gaze pointer on/off\" " text: "Map a mouse button (Buttons) or a key combination (Keyboard) to \"Gaze pointer on/off\" "
+ "to switch it on the fly. A click waits for the release: if the gaze is off, drag onto the target " + "to switch it on the fly. A click waits for the release: if the gaze is off, drag onto the target "
+ "with the button held and let go there. Hold still to drag: hold a press this long without moving " + "with the button held and let go there. Hold still to drag: hold a press this long without moving "
+ "to drag something instead. Look away to hand back: how far from the pointer you look before the " + "to drag something instead. Look away to hand back: how far from the pointer you look before the "
+ "gaze takes it back from the mouse. Largest nudge to learn: bigger mouse moves before a click " + "gaze takes it back from the mouse. Largest correction to learn: a click corrected further than "
+ "are treated as using the mouse, not correcting the gaze. Eye tracker: SteamVR's, or our own " + "this isn't learned; the tracker is that far off, so the quick check runs instead. Eye tracker: SteamVR's, or our own "
+ "(gaze/tracker), which keeps its own calibration (Calibrate… with Own tracker). Eye bias: the gaze " + "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze "
+ "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as " + "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
+ "much, and Auto weights each by how far off it was at your recent nudges." + "much, and Auto weights each by how far off it was at your recent nudges."
} }
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env python3
"""Controller desktop click stability over local IPC; no SteamVR client."""
import argparse
import json
import math
import socket
def request(command):
with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
client.bind(''); client.settimeout(2)
client.sendto(command.encode(), '\0ft_screens')
response=client.recv(8192).decode()
if response.startswith('error'): raise RuntimeError(response)
return response
def main():
parser=argparse.ArgumentParser(description=__doc__)
sub=parser.add_subparsers(dest='command', required=True)
sub.add_parser('status')
sub.add_parser('threshold').add_argument('pixels', type=float)
args=parser.parse_args()
try:
if args.command == 'threshold':
if not math.isfinite(args.pixels) or not 0 <= args.pixels <= 64:
raise ValueError('Threshold must be 0–64 logical pixels; 0 disables stabilization')
if request(f'controller-click {args.pixels:g}') != 'ok': raise RuntimeError('Threshold rejected')
print(json.dumps(json.loads(request('controller-click?')), indent=2))
except (OSError, ValueError, RuntimeError) as error: parser.exit(1, str(error)+'\n')
if __name__ == '__main__': main()
-306
View File
@@ -1,306 +0,0 @@
"""Gaze first, the relay's part (docs/gaze-first.md).
While the pointer helper says gaze first is on ("gazefirst 1" every 5 s: gaze mode, no game,
the headset on; "gazefirst 0" or 12 s of silence ends it), the Frame controllers belong to the
gaze:
- SteamVR: the Frame controller's compositor binding is
pointer/bindings/vrcompositor_frame_controller_gazefirst.json (only the Steam button left),
chosen through vrserver's /input/selectconfig.action, and the stock one again after.
- Steam's UI: input/steam-gamepad-filter.js runs in Steam's SharedJSContext (input/steamui.py)
and drops the controllers' gamepad input (the Steam button still passes). The block lasts 15 s
unless it's renewed, which happens every 5 s, so it ends by itself if the relay dies; and
Steam reloads its UI with SteamVR, which the renewal also covers.
- The trigger and bumper, read from vrserver's web socket (input/vrws.py, which takes nothing
from anyone), go to the helper as "ctrl <left|right> <trigger|bumper> 1|0"; the right
thumbstick scrolls ("scroll <x> <y>", as the mouse's wheel).
Steam still sees the controllers, and SteamVR leaves laser mode after each press and release:
the helper takes the laser back (see "Gaze first" in pointer/helper/ft-pointer.cpp).
The toggle macro works with gaze first on or off: both thumbstick clicks held 1 s turn gaze mode
on or off, and POINTER_GAZE in ~/.config/frametop.conf remembers it. So does `toggle()`, which
the relay's gaze_toggle action uses.
The binding, the filter, and finding new controllers run on a thread of their own: they're HTTP
and web socket round trips that mustn't hold up the relay's mouse and keyboard.
"""
import json
import os
import threading
import time
import urllib.request
import steamui
from vrws import HEADERS, ORIGIN, VrSocket, getstate
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FILTER_JS = os.path.join(ROOT, "input", "steam-gamepad-filter.js")
GAZE_BINDING = "file://" + os.path.join(ROOT, "pointer", "bindings", "vrcompositor_frame_controller_gazefirst.json")
STOCK_BINDING = "file:///opt/steamvr/drivers/frame_controller/resources/input/vrcompositor_bindings_frame_controller.json"
COMPOSITOR = "openvr.component.vrcompositor"
CONF = os.path.expanduser("~/.config/frametop.conf")
BUTTONS = {"/input/trigger/click": "trigger", "/input/bumper/click": "bumper"}
STICK_CLICK = "/input/thumbstick/click"
MACRO_HOLD = 1.0 # seconds both thumbstick clicks are held to toggle gaze mode
SCROLL_DEADZONE = 0.3
ON_FOR = 12.0 # "gazefirst 1" lasts this long without another
RENEW = 5.0 # the binding and filter are checked (and the filter's block renewed) this often
BLOCK_FOR = 15000 # ms the filter blocks without a renewal
def select_binding(url):
body = json.dumps({"app_key": COMPOSITOR, "controller_type": "frame_controller", "url": url}).encode()
req = urllib.request.Request(f"{ORIGIN}/input/selectconfig.action", data=body,
headers={**HEADERS, "Origin": ORIGIN, "Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=2) as resp:
if not json.load(resp).get("success"):
raise OSError("vrserver refused the binding")
def current_binding():
req = urllib.request.Request(f"{ORIGIN}/input/getactions.json?app_key={COMPOSITOR}", headers=HEADERS)
with urllib.request.urlopen(req, timeout=2) as resp:
return (json.load(resp).get("current_binding_url") or {}).get("frame_controller", "")
def read_gaze(path=CONF):
try:
with open(path) as f:
for line in f:
line = line.split("#", 1)[0].strip()
if line.split("=", 1)[0].strip() == "POINTER_GAZE" and "=" in line:
return line.split("=", 1)[1].strip() not in ("", "0")
except OSError:
pass
return False
def write_gaze(on, path=CONF):
"""POINTER_GAZE=1|0 in the config, keeping every other line as it is."""
try:
with open(path) as f:
lines = f.read().splitlines()
except OSError:
lines = []
value = f"POINTER_GAZE={1 if on else 0}"
for i, line in enumerate(lines):
if line.split("#", 1)[0].split("=", 1)[0].strip() == "POINTER_GAZE":
lines[i] = value
break
else:
lines.append(value)
tmp = path + ".tmp"
with open(tmp, "w") as f:
f.write("\n".join(lines) + "\n")
os.replace(tmp, path)
class GazeFirst:
def __init__(self, send, log):
self.send = send # a command to the pointer helper
self.log = log
self.on = False
self.on_until = 0.0
self.ws = None
self.next_connect = 0.0
self.sides = {} # subscribed device path -> "left" or "right"
self.values = {} # (side, component) -> last value
self.held = set() # (side, button) the helper has as pressed
self.macro_since = None
self.macro_fired = False
self.scrolling = (0.0, 0.0)
# The worker thread's: a wake-up, the devices it found, what it last applied.
self.lock = threading.Lock()
self.wake = threading.Event()
self.found = []
self.applied = None
self.problems = {}
threading.Thread(target=self.work, daemon=True, name="gazefirst").start()
# The relay's select loop: readable while connected to vrserver.
def fileno(self):
return self.ws.fileno()
def message(self, words, now):
""""gazefirst 1|0" from the helper."""
on = words[1] == "1"
self.on_until = now + ON_FOR if on else 0.0
if on != self.on:
self.set_on(on)
def set_on(self, on):
self.on = on
if not on:
self.release_all(cancel=True)
self.log(f"gaze first {'on' if on else 'off'}")
self.wake.set()
def release_all(self, cancel=False):
"""Let go of what the helper holds: as releases (a click), or cancelled (no click)."""
if cancel and self.held:
self.send("ctrlcancel")
else:
for side, button in sorted(self.held):
self.send(f"ctrl {side} {button} 0")
self.held.clear()
if self.scrolling != (0.0, 0.0):
self.send("scroll 0 0")
self.scrolling = (0.0, 0.0)
def toggle(self):
"""Gaze mode on or off, remembered (POINTER_GAZE)."""
on = not read_gaze()
try:
write_gaze(on)
except OSError as e:
self.log(f"gaze mode: can't write {CONF}: {e}")
self.send(f"gaze {'on' if on else 'off'}")
self.log(f"gaze mode {'on' if on else 'off'}")
def timeout(self):
return 0.05 if self.macro_since is not None and not self.macro_fired else 0.5
def tick(self, now):
if self.on and now > self.on_until:
self.set_on(False) # the helper went quiet
if self.ws is None and now >= self.next_connect:
self.next_connect = now + 3.0
try:
self.ws = VrSocket(timeout=1)
self.ws.open(f"frametop_relay_{os.getpid()}")
self.sides = {}
self.wake.set() # find the controllers now
except OSError:
self.ws = None
with self.lock:
found, self.found = self.found, []
for path, side in found:
if self.ws and self.sides.get(path) != side:
try:
self.ws.subscribe(path)
self.sides[path] = side
except OSError:
self.drop()
if self.macro_since is not None and not self.macro_fired and now - self.macro_since >= MACRO_HOLD:
self.macro_fired = True
self.release_all(cancel=True) # a press in progress doesn't click
self.toggle()
def drop(self):
if self.ws:
try:
self.ws.close()
except OSError:
pass
self.ws = None
self.release_all(cancel=True)
def readable(self, now):
try:
while True:
msg = self.ws.recv(timeout=0)
if isinstance(msg, dict) and msg.get("type") == "update_component_states":
side = self.sides.get(msg.get("device"))
if side:
self.update(side, msg.get("components") or {}, now)
if not self.ws.pending():
return # the rest, if any, wakes select again
except OSError:
self.drop()
def update(self, side, components, now):
for name, value in components.items():
key = (side, name)
if self.values.get(key) == value:
continue
self.values[key] = value
button = BUTTONS.get(name)
if button:
down = bool(value)
if down and self.on and (side, button) not in self.held:
self.held.add((side, button))
self.send(f"ctrl {side} {button} 1")
elif not down and (side, button) in self.held:
self.held.discard((side, button))
self.send(f"ctrl {side} {button} 0")
elif name == STICK_CLICK:
both = self.values.get(("left", STICK_CLICK)) and self.values.get(("right", STICK_CLICK))
if both and self.macro_since is None:
self.macro_since, self.macro_fired = now, False
elif not both:
self.macro_since = None
elif side == "right" and name in ("/input/thumbstick/x", "/input/thumbstick/y"):
self.scroll()
def scroll(self):
def shape(v):
v = float(v or 0)
if abs(v) < SCROLL_DEADZONE:
return 0.0
return round((abs(v) - SCROLL_DEADZONE) / (1 - SCROLL_DEADZONE) * (1 if v > 0 else -1), 1)
want = (shape(self.values.get(("right", "/input/thumbstick/x"))),
shape(self.values.get(("right", "/input/thumbstick/y")))) if self.on else (0.0, 0.0)
if want != self.scrolling:
self.scrolling = want
self.send(f"scroll {want[0]:.1f} {want[1]:.1f}")
def shutdown(self):
"""The relay is stopping: the controllers go back to SteamVR and Steam."""
self.on = False
self.release_all(cancel=True)
try:
select_binding(STOCK_BINDING)
except OSError:
pass
try:
steamui.evaluate('window.__frametopGaze && (window.__frametopGaze.mode = "off")', timeout=1)
except (OSError, RuntimeError):
pass
# The worker thread.
def problem(self, what, error=None):
"""Log a failure once, and when it's over (error None)."""
if error is None:
if self.problems.pop(what, None) is not None:
self.log(f"gaze first: {what} works again")
return
if self.problems.get(what) != str(error):
self.problems[what] = str(error)
self.log(f"gaze first: {what}: {error}")
def work(self):
while True:
self.wake.wait(RENEW)
self.wake.clear()
on = self.on
try:
found = [(d["root_path"], d.get("side") or d["root_path"].rsplit("/", 1)[-1])
for d in getstate(timeout=1) if d.get("controller_type") == "frame_controller"]
with self.lock:
self.found = [(p, s) for p, s in found if s in ("left", "right")]
self.problem("vrserver")
except (OSError, ValueError) as e:
self.problem("vrserver", e)
continue
try:
want = GAZE_BINDING if on else STOCK_BINDING
if current_binding() != want:
select_binding(want)
self.log(f"gaze first: {'gaze' if on else 'stock'} controller binding")
self.problem("binding")
except (OSError, ValueError) as e:
self.problem("binding", e)
try:
if on:
with open(FILTER_JS) as f:
steamui.evaluate(f.read())
steamui.evaluate('(() => { const G = window.__frametopGaze; G.mode = "block"; '
f'G.until = Date.now() + {BLOCK_FOR}; return G.mode; }})()')
elif self.applied is not False:
steamui.evaluate('window.__frametopGaze && (window.__frametopGaze.mode = "off")')
self.applied = on
self.problem("Steam's UI")
except (OSError, RuntimeError, ValueError) as e:
self.problem("Steam's UI", e)
+249 -81
View File
@@ -14,24 +14,43 @@ keyboard node for its extra buttons). Roles, from ~/.config/frametop-input.json
(written by the Frametop Input Settings app): (written by the Frametop Input Settings app):
pointer grabbed; drives the universal 3D mouse (default for devices with a mouse node) pointer grabbed; drives the universal 3D mouse (default for devices with a mouse node)
passthrough keys go to the desktop; grabbed only while typing goes there, otherwise only observed, passthrough keys go to the desktop; grabbed only while typing goes there, otherwise only observed,
e.g. for the Meta dashboard shortcut (default for keyboards; the shortcut is off for the key combinations below (default for keyboards)
unless META_DASHBOARD=1 is in ~/.config/frametop.conf)
ignore not grabbed, only observed for identification in the settings app ignore not grabbed, only observed for identification in the settings app
Buttons and keys of pointer devices go through a per-device map to actions Buttons and keys of pointer devices go through a per-device map to actions
(left, right, middle, back, scroll_up, scroll_down, dashboard, recenter, (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode on or off, pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode on or off
remembered as POINTER_GAZE (the pointer goes where you look; see pointer/helper/ft-pointer.cpp (the pointer goes where you look; see pointer/helper/ft-pointer.cpp), gaze_precision = while
and input/gazefirst.py), gaze_precision = while held, the pointer stops where you look and the mouse steers it, and the release clicks there,
held, the pointer stops where you look and the button's device (the mouse, or that gaze_drag = the same, but pressed at once, so it drags ("precision|gazedrag mouse|keyboard 1|0"
controller's aim) steers it, and the release clicks there, gaze_drag = the same, but pressed to the helper), gaze_left and gaze_right = keyboard clicks at the gaze: a tap clicks where you
at once, so it drags ("precision|gazedrag mouse|left|right|keyboard 1|0" to the helper), sens_up, sens_down, look; held, the pointer stops there and your head steers it (it stays put in your view), and
the release clicks; held still for half a second, it's a real press that your head drags
("gazekey left|right 1|0" to the helper; by default Meta+J and Meta+K, DEFAULT_KEY_BINDINGS),
gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens_up, sens_down,
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens, layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
keyboard_toggle = open or close Frametop's keyboard, key = pass through as a key, none). keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
window back (both to ft-floatd, @frametop_float), profile:NAME = switch to that profile (ft-layout
use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), command:CMD = run
CMD with sh -c (on the host, as this service: its environment, output to its log, and
COMMAND_PATH, so ft-layout, ft-float and ft-steam need no path), key = pass through as a key, none).
Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
rules file; any action but key). So can key combinations on any keyboard ("key_bindings": rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature,
docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
{"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for {"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed. The controllers aren't input devices here, only SteamVR sees either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed.
A modifier on its own ("125": Meta) is a tap: pressed and released with no other key, mouse
button, or scroll in between; the desktop gets an F24 press before its release, so Plasma's
launcher doesn't open on a Meta tap that's bound. A rules file without "key_bindings" gets
DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left, Meta+K: gaze_right, Meta+Shift+F:
float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu and
command actions work without pointer mode too. A combination with Meta also sends the desktop an
F24 press and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and
a gaze click isn't Meta+click (KWin's window move and resize). Another key while Meta is still
held gives the desktop Meta back. While typing goes to Steam, keyboards aren't grabbed, so Steam
or the game sees a combination's keys too. The controllers aren't input devices here, only SteamVR sees
them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0". them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper, It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper,
sent on start, reload, and when the helper says "vrhello"), and only while no game runs, sent on start, reload, and when the helper says "vrhello"), and only while no game runs,
@@ -85,7 +104,6 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
vrcapture <s> take every controller button for s seconds (0: stop), so the settings vrcapture <s> take every controller button for s seconds (0: stop), so the settings
app can capture one; watchers see them as events with id frame_controller app can capture one; watchers see them as events with id frame_controller
vrbtn, vrhello, gazeawake from the pointer helper (above) vrbtn, vrhello, gazeawake from the pointer helper (above)
gazefirst 1|0 from the pointer helper: gaze first on or off (input/gazefirst.py)
textfield 1|0 from the desktop's input method (above) textfield 1|0 from the desktop's input method (above)
Runs on the Frame host as a user service (frametop-input-relay.service). The Runs on the Frame host as a user service (frametop-input-relay.service). The
@@ -112,14 +130,13 @@ import subprocess
import sys import sys
import time import time
import gazefirst
# Linux input constants (include/uapi/linux/input-event-codes.h, input.h, uinput.h). # Linux input constants (include/uapi/linux/input-event-codes.h, input.h, uinput.h).
EV_SYN, EV_KEY, EV_REL, EV_MSC = 0x00, 0x01, 0x02, 0x04 EV_SYN, EV_KEY, EV_REL, EV_MSC = 0x00, 0x01, 0x02, 0x04
SYN_REPORT = 0 SYN_REPORT = 0
BTN_MISC, KEY_MAX = 0x100, 0x2FF BTN_MISC, KEY_MAX = 0x100, 0x2FF
KEY_A = 30 KEY_A = 30
REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C} # REL_HWHEEL, REL_WHEEL and their _HI_RES
BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114 BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126 KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115 KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
@@ -177,8 +194,17 @@ def eviocguniq(length):
VIRTUAL_PREFIX = "frametop virtual" VIRTUAL_PREFIX = "frametop virtual"
RULES_PATH = os.path.expanduser("~/.config/frametop-input.json") RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter", ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "sens_up", "sens_down", "pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
"layout_reset", "screens_toggle", "keyboard_toggle", "key", "none") "gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
"dock_all", "steam_menu", "key", "none")
# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
# Meta+Shift+F floats a window.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
KEY_F24 = 194 # sent to the desktop with a Meta combination (see the top)
# Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's. # Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125} MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never") # when Frametop's keyboard opens VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never") # when Frametop's keyboard opens
@@ -189,8 +215,28 @@ VR_BUTTONS = ("left/view", "left/dpad_up", "left/dpad_down", "left/dpad_left", "
"right/bumper", "right/trigger", "right/grip", "right/thumbstick") "right/bumper", "right/trigger", "right/grip", "right/thumbstick")
VR_DEVICE = "frame_controller" # the id controller buttons have in watch events VR_DEVICE = "frame_controller" # the id controller buttons have in watch events
SCREENS = "\0ft_screens" SCREENS = "\0ft_screens"
GAZED = "\0ft_gazed"
FLOAT = "\0frametop_float" # ft-floatd, floating windows in the Frametop desktop
# Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
PROFILE = "profile:" # "profile:NAME": switch to that profile (doesn't need pointer mode either)
COMMAND = "command:" # "command:CMD": run CMD (nor does this)
def known_action(a):
return a in ACTIONS or (isinstance(a, str) and any(a.startswith(p) and a[len(p):].strip()
for p in (PROFILE, COMMAND)))
def needs_pointer(a):
return a not in FLOAT_ACTIONS and a != "steam_menu" and not a.startswith((PROFILE, COMMAND))
KEYS = "\0frametop_keys" # keys of keyboards grabbed for the desktop, for other readers KEYS = "\0frametop_keys" # keys of keyboards grabbed for the desktop, for other readers
FT_LAYOUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "layout", "ft-layout") REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FT_LAYOUT = os.path.join(REPO, "layout", "ft-layout")
FT_STEAM = os.path.join(REPO, "steam", "ft-steam")
# Commands ("command:CMD") find Frametop's own tools (ft-layout, ft-float, ft-steam) on their PATH.
COMMAND_PATH = ":".join([os.path.join(REPO, d) for d in ("layout", "float", "steam")]
+ [os.environ.get("PATH", "/usr/local/bin:/usr/bin")])
DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle", DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
BTN_SIDE: "back", BTN_EXTRA: "back"} BTN_SIDE: "back", BTN_EXTRA: "back"}
@@ -296,7 +342,8 @@ def read_rules(path=RULES_PATH):
rules.setdefault("devices", {}) rules.setdefault("devices", {})
rules.setdefault("buttons", {}) rules.setdefault("buttons", {})
rules.setdefault("controller_buttons", {}) rules.setdefault("controller_buttons", {})
rules.setdefault("key_bindings", {}) if not isinstance(rules.get("key_bindings"), dict):
rules["key_bindings"] = dict(DEFAULT_KEY_BINDINGS)
return rules return rules
@@ -305,21 +352,34 @@ def remap_volume(fd, restore=False):
Returns how many keymap entries are volume keys or stand-ins, or None when the Returns how many keymap entries are volume keys or stand-ins, or None when the
device has no keymap to change (uinput devices, some platform buttons). device has no keymap to change (uinput devices, some platform buttons).
A swap that fails doesn't stop the others: every entry is still tried, then the
first failure is raised. The entries that did swap stay swapped, for the caller
to handle their stand-ins and restore them.
""" """
swap = VOLUME_ORIGINAL if restore else VOLUME_STANDIN swap = VOLUME_ORIGINAL if restore else VOLUME_STANDIN
found = 0 found = 0
failed = None # the first swap that failed
for index in range(8192): for index in range(8192):
entry = bytearray(KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, 0, index, 0, b"")) entry = bytearray(KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, 0, index, 0, b""))
try: try:
fcntl.ioctl(fd, EVIOCGKEYCODE_V2, entry) fcntl.ioctl(fd, EVIOCGKEYCODE_V2, entry)
except OSError: except OSError:
return found if index else None # past the last entry if not index:
return None
break # past the last entry
_, length, _, code, scancode = KEYMAP_ENTRY.unpack(entry) _, length, _, code, scancode = KEYMAP_ENTRY.unpack(entry)
if code in VOLUME_CODES: if code in VOLUME_CODES:
found += 1 found += 1
if code in swap: if code in swap:
try:
fcntl.ioctl(fd, EVIOCSKEYCODE_V2, fcntl.ioctl(fd, EVIOCSKEYCODE_V2,
KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, length, index, swap[code], scancode)) KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, length, index, swap[code], scancode))
except OSError as e:
if failed is None:
failed = e
if failed is not None:
raise failed
return found return found
@@ -434,10 +494,19 @@ class Pointer:
def action(self, name, value, now, source="mouse"): def action(self, name, value, now, source="mouse"):
"""A mapped button: value 1 press, 0 release, 2 autorepeat (ignored). source: what """A mapped button: value 1 press, 0 release, 2 autorepeat (ignored). source: what
pressed it (mouse, left, right for a controller, keyboard), for the gaze actions.""" pressed it (mouse, left, right for a controller, keyboard), for the gaze actions,
which take the mouse or the keyboard only."""
if value == 2: if value == 2:
return return
if name in ("gaze_left", "gaze_right"):
if value == 1:
self.wake(now)
self.flush()
self.send(f"gazekey {name[5:]} {value}")
return
if name in ("gaze_precision", "gaze_drag"): if name in ("gaze_precision", "gaze_drag"):
if source not in ("mouse", "keyboard"):
return # gaze mode is a mouse feature (GAZE_ACTIONS)
if value == 1: if value == 1:
self.wake(now) self.wake(now)
self.flush() self.flush()
@@ -469,6 +538,14 @@ class Pointer:
elif name == "follow_toggle": elif name == "follow_toggle":
self.send("follow toggle") # until the next restart; the setting is POINTER_FOLLOW self.send("follow toggle") # until the next restart; the setting is POINTER_FOLLOW
log("head follow toggled") log("head follow toggled")
elif name == "gaze_toggle":
self.send("gaze toggle") # until the next restart; the setting is POINTER_GAZE
log("gaze mode toggled")
elif name == "gaze_quickcal":
try:
self.sock.sendto(b"quickcal", GAZED)
except OSError:
pass # the gaze service isn't running
elif name == "screens_toggle": elif name == "screens_toggle":
try: try:
self.sock.sendto(b"toggle", SCREENS) self.sock.sendto(b"toggle", SCREENS)
@@ -611,13 +688,12 @@ def main():
# desktop_until: typing goes to the Frametop desktop until then (ft-screens says so # desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
# every second); typing_applied: the grabs match that as of the last apply_roles(). # every second); typing_applied: the grabs match that as of the last apply_roles().
# vr_capture_until: every controller button is taken until then (the settings app capturing one). # vr_capture_until: every controller button is taken until then (the settings app capturing one).
state = {"pointer": None, "rules": {}, "meta_dashboard": False, "share_keys": False, state = {"pointer": None, "rules": {}, "share_keys": False,
"desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0} "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0}
def load_config(): def load_config():
conf = read_config() conf = read_config()
state["rules"] = read_rules() state["rules"] = read_rules()
state["meta_dashboard"] = conf.get("META_DASHBOARD", "0") == "1"
state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1" state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
if conf.get("POINTER", "0") == "1": if conf.get("POINTER", "0") == "1":
p = state["pointer"] or Pointer(0.03, 30) p = state["pointer"] or Pointer(0.03, 30)
@@ -631,7 +707,7 @@ def main():
log("pointer mode off: pointer devices feed the virtual mouse and keyboard") log("pointer mode off: pointer devices feed the virtual mouse and keyboard")
load_config() load_config()
meta_down = False # Meta pressed with no other key yet: a tap toggles the dashboard tap = None # the modifier (folded, MODIFIERS) pressed alone, with nothing since: its release is a tap
screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK) screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
last_typing = 0.0 # the helper was last told of a key then (see "typing" at the top) last_typing = 0.0 # the helper was last told of a key then (see "typing" at the top)
@@ -642,7 +718,8 @@ def main():
else: else:
state["vr_capture_until"] = 0.0 state["vr_capture_until"] = 0.0
buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items() buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items()
if b in VR_BUTTONS and a in ACTIONS and a not in ("key", "none")) or "-" if b in VR_BUTTONS and known_action(a) and a not in ("key", "none")
and a not in GAZE_ACTIONS) or "-"
if state["rules"].get("controller_in_games"): if state["rules"].get("controller_in_games"):
buttons = "+games " + buttons buttons = "+games " + buttons
try: try:
@@ -659,9 +736,8 @@ def main():
reply(addr, {"t": "event", "id": VR_DEVICE, "path": "", "name": "Steam Frame controllers", reply(addr, {"t": "event", "id": VR_DEVICE, "path": "", "name": "Steam Frame controllers",
"type": "vr", "code": button, "value": value}) "type": "vr", "code": button, "value": value})
action = state["rules"]["controller_buttons"].get(button) action = state["rules"]["controller_buttons"].get(button)
if gaze.on and button.split("/")[-1] in ("trigger", "bumper"): if (state["pointer"] or (action and not needs_pointer(action))) and known_action(action) \
return # gaze first has them (input/gazefirst.py) and action not in ("key", "none") and action not in GAZE_ACTIONS:
if state["pointer"] and action in ACTIONS and action not in ("key", "none"):
do_action(action, value, now, button.split("/")[0]) do_action(action, value, now, button.split("/")[0])
def vr_keyboard_mode(): def vr_keyboard_mode():
@@ -686,42 +762,110 @@ def main():
vr_keyboard("show") vr_keyboard("show")
def do_action(action, value, now, source="mouse"): def do_action(action, value, now, source="mouse"):
"""A mapped mouse or controller button, or key combination (pointer mode only).""" """A mapped mouse or controller button, or key combination (pointer mode only, but
if action == "gaze_toggle": for the actions needs_pointer() says don't)."""
if value == 1: if action == "keyboard_toggle":
gaze.toggle() # remembered (POINTER_GAZE), like the toggle macro
elif action == "keyboard_toggle":
if value == 1 and vr_keyboard_mode() != "never": if value == 1 and vr_keyboard_mode() != "never":
vr_keyboard("toggle") vr_keyboard("toggle")
else: elif action.startswith(PROFILE):
if value == 1:
# Runs a few seconds and borrows the pointer, like layout_reset.
subprocess.Popen([FT_LAYOUT, "use", action[len(PROFILE):]], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
log(action)
elif action == "steam_menu":
if value == 1:
# Talks to Steam's UI for a moment; its errors go to this service's log.
subprocess.Popen([FT_STEAM, "menu"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
start_new_session=True)
log(action)
elif action.startswith(COMMAND):
if value == 1:
subprocess.Popen(["sh", "-c", action[len(COMMAND):]], stdin=subprocess.DEVNULL,
env=dict(os.environ, PATH=COMMAND_PATH), start_new_session=True)
log(action)
elif action in FLOAT_ACTIONS:
if value == 1:
try:
screens_sock.sendto(FLOAT_ACTIONS[action], FLOAT)
except OSError:
pass # the Frametop desktop isn't running
log(action)
elif state["pointer"]:
state["pointer"].action(action, value, now, source) state["pointer"].action(action, value, now, source)
held_modifiers = set() # on any keyboard, folded (MODIFIERS) held_modifiers = set() # on any keyboard, folded (MODIFIERS)
held_meta = set() # the Meta keys held, as they are (KEY_LEFTMETA, KEY_RIGHTMETA)
meta_hidden = set() # held Meta keys the desktop was told came up (a combination; key_binding)
combos_down = {} # key code -> the action its combination started (released with it) combos_down = {} # key code -> the action its combination started (released with it)
def key_binding(code, value, now): def key_binding(code, value, now):
"""A key from a keyboard: does it complete a key combination ("key_bindings")? True if """A key from a keyboard: does it complete a key combination ("key_bindings")? True if
it was taken for one (then it isn't typed).""" it was taken for one (then it isn't typed)."""
nonlocal tap
if code in MODIFIERS: if code in MODIFIERS:
(held_modifiers.add if value else held_modifiers.discard)(MODIFIERS[code]) mod = MODIFIERS[code]
if value == 1:
tap = mod if not held_modifiers and not combos_down else None
(held_modifiers.add if value else held_modifiers.discard)(mod)
if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
(held_meta.add if value else held_meta.discard)(code)
if value == 0 and code in meta_hidden:
meta_hidden.discard(code)
return True # the desktop already had it come up (below)
if value == 0 and tap == mod:
tap = None
modifier_tap(mod, now)
return False return False
if value == 1 and code not in combos_down and meta_hidden:
# Another key while Meta is still held after a combination: the desktop gets Meta
# back first, so Meta+that key still works there.
for c in sorted(meta_hidden):
to_screens(c, 1)
meta_hidden.clear()
if value == 0 and code in combos_down: if value == 0 and code in combos_down:
action = combos_down.pop(code) action = combos_down.pop(code)
if state["pointer"]: if state["pointer"] or not needs_pointer(action):
do_action(action, 0, now, "keyboard") do_action(action, 0, now, "keyboard")
return True return True
if value != 1 or not state["rules"]["key_bindings"]: if value != 1 or not state["rules"]["key_bindings"]:
return value == 2 and code in combos_down return value == 2 and code in combos_down
combo = "+".join(str(c) for c in sorted(held_modifiers) + [code]) combo = "+".join(str(c) for c in sorted(held_modifiers) + [code])
action = state["rules"]["key_bindings"].get(combo) action = state["rules"]["key_bindings"].get(combo)
if action not in ACTIONS or action in ("key", "none"): if not known_action(action) or action in ("key", "none"):
return False return False
combos_down[code] = action combos_down[code] = action
if state["pointer"]: if held_meta - meta_hidden:
# The desktop saw Meta go down. Another key in between keeps its release from
# opening Plasma's launcher, and Meta comes up there now: KWin takes Meta with a
# mouse button for moving or resizing windows, which would swallow a gaze click.
# Its real release is dropped (above).
to_screens(KEY_F24, 1)
to_screens(KEY_F24, 0)
for c in sorted(held_meta - meta_hidden):
to_screens(c, 0)
meta_hidden.update(held_meta)
if state["pointer"] or not needs_pointer(action):
do_action(action, 1, now, "keyboard") do_action(action, 1, now, "keyboard")
log(f"key combination {combo}: {action}") log(f"key combination {combo}: {action}")
return True return True
def modifier_tap(mod, now):
"""A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
action = state["rules"]["key_bindings"].get(str(mod))
if not known_action(action) or action in ("key", "none"):
return
# The desktop gets a key in between before the release goes there, so the tap isn't one
# there too: a Meta tap would open Plasma's launcher.
to_screens(KEY_F24, 1)
to_screens(KEY_F24, 0)
if state["pointer"] or not needs_pointer(action):
do_action(action, 1, now, "keyboard")
do_action(action, 0, now, "keyboard")
log(f"key combination {mod}: {action}")
screens_down = set() # keys the desktop was told went down and not yet up (see reconcile_desktop_keys)
def to_screens(code, value): def to_screens(code, value):
"""A key for the desktop screens (ft-screens decides whether it types).""" """A key for the desktop screens (ft-screens decides whether it types)."""
@@ -729,7 +873,8 @@ def main():
try: try:
screens_sock.sendto(f"key {code} {value}".encode(), SCREENS) screens_sock.sendto(f"key {code} {value}".encode(), SCREENS)
except OSError: except OSError:
pass # ft-screens not running return # ft-screens not running
(screens_down.add if value else screens_down.discard)(code)
nodes = {} # fd -> Node nodes = {} # fd -> Node
# Nodes already probed (rejected or open): path -> inode. A device that disconnects and # Nodes already probed (rejected or open): path -> inode. A device that disconnects and
# reconnects between two scans often gets the same event numbers back, so the path alone # reconnects between two scans often gets the same event numbers back, so the path alone
@@ -811,6 +956,35 @@ def main():
mouse.sync() mouse.sync()
keyboard.sync() keyboard.sync()
def physically_down():
"""The keys down on every device read here, as the kernel has them (EVIOCGKEY)."""
down = set()
for node in nodes.values():
buf = bytearray((KEY_MAX + 8) // 8)
try:
fcntl.ioctl(node.fd, EVIOCGKEY, buf)
except OSError:
continue
down.update(i * 8 + bit for i, b in enumerate(buf) if b for bit in range(8) if b >> bit & 1)
return down
def reconcile_desktop_keys():
"""A key the desktop has down that no device holds comes up there, and the key
combinations forget a Meta or modifier no device holds. A key can be left down when
its device vanishes with it held (release_held only lets go of it here) or a release
goes astray: on 2026-10-01, after a calibration, Meta stayed down in KWin, so typing
opened the overview and clicks on the desktop did other things."""
if not screens_down and not held_meta and not held_modifiers:
return
down = physically_down()
for code in sorted(screens_down - down):
to_screens(code, 0)
log(f"key {code} released on the desktop: no keyboard holds it")
for code in [c for c in held_meta if c not in down]:
held_meta.discard(code)
meta_hidden.discard(code)
held_modifiers.intersection_update({MODIFIERS[c] for c in down if c in MODIFIERS})
def keys_down(node): def keys_down(node):
buf = bytearray((KEY_MAX + 8) // 8) buf = bytearray((KEY_MAX + 8) // 8)
try: try:
@@ -874,6 +1048,10 @@ def main():
return return
words = data.decode(errors="replace").split() words = data.decode(errors="replace").split()
cmd = words[0] if words else "" cmd = words[0] if words else ""
# One malformed datagram must not end the relay: it would drop every grab,
# including the volume keys that keep gamescope from aborting. The control
# socket is an abstract socket, so any local process can send to it.
try:
if cmd == "keyboard": if cmd == "keyboard":
# From ft-screens (unbound, no reply): where typing goes, repeated every second. # From ft-screens (unbound, no reply): where typing goes, repeated every second.
desktop = len(words) > 1 and words[1] == "desktop" desktop = len(words) > 1 and words[1] == "desktop"
@@ -888,9 +1066,6 @@ def main():
if cmd == "textfield" and len(words) == 2: if cmd == "textfield" and len(words) == 2:
text_field(words[1] == "1") text_field(words[1] == "1")
continue continue
if cmd == "gazefirst" and len(words) == 2:
gaze.message(words, now)
continue
if cmd == "gazeawake" and len(words) == 2: if cmd == "gazeawake" and len(words) == 2:
if state["pointer"]: if state["pointer"]:
state["pointer"].gaze_awake_until = now + 12.0 if words[1] == "1" else 0.0 state["pointer"].gaze_awake_until = now + 12.0 if words[1] == "1" else 0.0
@@ -919,6 +1094,8 @@ def main():
reply(addr, {"t": "vrcapture", "seconds": seconds}) reply(addr, {"t": "vrcapture", "seconds": seconds})
else: else:
reply(addr, {"t": "error", "error": f"unknown command {cmd!r}"}) reply(addr, {"t": "error", "error": f"unknown command {cmd!r}"})
except Exception as e:
log(f"bad control datagram {data!r}: {e!r}")
def broadcast(node, etype, code, value, now): def broadcast(node, etype, code, value, now):
if not watchers or not node.candidate: if not watchers or not node.candidate:
@@ -934,24 +1111,18 @@ def main():
else: else:
reply(addr, msg) reply(addr, msg)
def to_helper(command):
try:
screens_sock.sendto(command.encode(), HELPER)
except OSError:
pass # helper not running
# Gaze first (input/gazefirst.py): the controllers' trigger, bumper, thumbstick, and the
# toggle macro, from vrserver's web socket; SteamVR's and Steam's muting.
gaze = gazefirst.GazeFirst(to_helper, log)
atexit.register(gaze.shutdown)
vr_bind(time.monotonic()) # a helper that's already running keeps its buttons in step vr_bind(time.monotonic()) # a helper that's already running keeps its buttons in step
waiting = False # a keyboard's grab waits for its keys to come up waiting = False # a keyboard's grab waits for its keys to come up
# A relay that went away with a key down left it down on the desktop, where this one
# never sent it: modifiers come up there now (a release of a key that isn't down is nothing).
for code in sorted(MODIFIERS):
to_screens(code, 0)
while True: while True:
now = time.monotonic() now = time.monotonic()
pointer = state["pointer"] pointer = state["pointer"]
if now >= next_scan: if now >= next_scan:
next_scan = now + 1.0 next_scan = now + 1.0
reconcile_desktop_keys()
current = {} current = {}
for name in os.listdir("/dev/input"): for name in os.listdir("/dev/input"):
if name.startswith("event"): if name.startswith("event"):
@@ -968,6 +1139,11 @@ def main():
# New here: a new device, or one that came back in the same place. # New here: a new device, or one that came back in the same place.
for old in [n for n in nodes.values() if n.path == path]: for old in [n for n in nodes.values() if n.path == path]:
drop(old, "replaced by a new device node") drop(old, "replaced by a new device node")
# A new node is root's alone until udev gives it to the input group, a moment
# after it appears. Opened in that gap, it would fail and never be tried
# again: leave it for the next scan instead.
if not os.access(path, os.R_OK):
continue
seen[path] = ino seen[path] = ino
node = probe(path) node = probe(path)
if node and node.volume_keys and not take_volume(node): if node and node.volume_keys and not take_volume(node):
@@ -979,12 +1155,11 @@ def main():
if added: if added:
apply_roles() apply_roles()
ready, _, _ = select.select(list(nodes) + [control] + ([gaze] if gaze.ws else []), [], [], ready, _, _ = select.select(list(nodes) + [control], [], [],
min(gaze.timeout(), volume.timeout(now, pointer.timeout() if pointer else 0.5))) volume.timeout(now, pointer.timeout() if pointer else 0.5))
now = time.monotonic() now = time.monotonic()
if pointer: if pointer:
pointer.tick(now) pointer.tick(now)
gaze.tick(now)
volume.tick(now) volume.tick(now)
if state["vr_capture_until"] and now >= state["vr_capture_until"]: if state["vr_capture_until"] and now >= state["vr_capture_until"]:
vr_bind(now) # capture over: back to the mapped buttons vr_bind(now) # capture over: back to the mapped buttons
@@ -994,10 +1169,6 @@ def main():
if fd is control: if fd is control:
handle_control(now) handle_control(now)
continue continue
if fd is gaze:
if gaze.ws:
gaze.readable(now)
continue
node = nodes[fd] node = nodes[fd]
try: try:
data = os.read(fd, EVENT.size * 64) data = os.read(fd, EVENT.size * 64)
@@ -1015,40 +1186,37 @@ def main():
if etype in (EV_KEY, EV_REL): if etype in (EV_KEY, EV_REL):
broadcast(node, etype, VOLUME_ORIGINAL.get(code, code) if node.remapped else code, broadcast(node, etype, VOLUME_ORIGINAL.get(code, code) if node.remapped else code,
value, now) value, now)
if (etype == EV_KEY and value == 1 and code not in MODIFIERS) or (etype == EV_REL and code in SCROLLS):
tap = None # a key, button or scroll in between: a modifier's release isn't a tap
if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL) if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL)
or (node.grabbed and code in VOLUME_STANDIN)): or (node.grabbed and code in VOLUME_STANDIN)):
volume.key(fd, code, value, now) volume.key(fd, code, value, now)
if value == 1:
meta_down = False # Meta used as a modifier, not a tap
continue continue
if node.role == "volume": if node.role == "volume":
continue continue
if node.role != "pointer": if node.role != "pointer":
# Observed only, unless typing goes to the desktop. With META_DASHBOARD=1, # Observed only, unless typing goes to the desktop. Key combinations work on
# a Meta tap on any keyboard toggles the dashboard. # any pass-through keyboard.
if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now): if (node.role == "passthrough" and etype == EV_KEY and node.grabbed
continue and code < BTN_MISC and value in (0, 1)):
if node.role == "passthrough" and etype == EV_KEY: # Shared as pressed, key combinations included: the Meta release a
if value == 1 and code < BTN_MISC and now - last_typing >= 0.25: # combination keeps from the desktop must still reach frame-voice, or
last_typing = now # it waits for that release before typing anything.
screens_sock.sendto(b"typing", HELPER)
to_screens(code, value)
if node.grabbed and code < BTN_MISC and value in (0, 1):
share_key(node, code, value) share_key(node, code, value)
if value: if value:
node.held.add(code) node.held.add(code)
else: else:
node.held.discard(code) node.held.discard(code)
if (pointer and state["meta_dashboard"] and node.role == "passthrough" if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
and etype == EV_KEY): continue
if code in (KEY_LEFTMETA, KEY_RIGHTMETA): if node.role == "passthrough" and etype == EV_KEY:
if value == 1: if value == 1 and code < BTN_MISC and now - last_typing >= 0.25:
meta_down = True last_typing = now
elif value == 0 and meta_down: try:
meta_down = False screens_sock.sendto(b"typing", HELPER)
pointer.dashboard() except OSError:
elif value == 1: pass # helper not running (SteamVR not running)
meta_down = False # Meta used as a modifier, not a tap to_screens(code, value)
continue continue
if etype == EV_KEY: if etype == EV_KEY:
action = buttons.get(str(code), DEFAULT_BUTTONS.get(code, "key")) action = buttons.get(str(code), DEFAULT_BUTTONS.get(code, "key"))
-106
View File
@@ -1,106 +0,0 @@
// Frametop gaze first: a filter in Steam's UI (SharedJSContext, through Steam's CEF debugger on
// 127.0.0.1:8080) that keeps the Frame controllers' gamepad input away from Steam's UI while
// gaze mode has the dashboard (docs/gaze-first.md). Steam reads the controllers as its own
// virtual gamepad (Steam Input), so no SteamVR binding can do this.
//
// Steam's gamepad input source (webpack module 17900, class E) turns each controller message
// into this.OnButtonDown / OnButtonUp / OnAnalogPad calls, which live on its base class. The
// filter defines wrappers on E's prototype, so both live instances go through them. They look
// up window.__frametopGaze at every call, so evaluating this file again updates the logic in
// place; a wrapper from an older version is replaced. It returns the state as JSON.
//
// The dashboard's mode (laser or gamepad) is SteamVR's, not the UI's: the UI only mirrors it
// (VRFocus, module 84114: J.Instance.ShowGamepadFocusMode, from SteamVR's
// system_panel_interaction_mode).
//
// window.__frametopGaze:
// mode "off" (pass everything), "log" (pass, and log), "block" (drop all but `allow`),
// "auto" (block while the dashboard is in laser mode, pass in gamepad mode)
// until with "block" or "auto": blocking stops at this time (Date.now(), ms) unless it's
// moved on. The input relay (input/gazefirst.py) sets it 15 s ahead every 10 s or so,
// so if the relay dies, the controllers come back to Steam's UI by themselves.
// allow buttons that always pass: the Steam button's guide and quick menu, and Steam's
// own dummy input
// log the last 256 events: [ms, "down"|"up"|"analog", button, controller, passed]
// modes the dashboard's mode changes: [ms, "gamepad"|"laser"] (sampled every 50 ms)
// A release passes if its press did, so nothing stays held when blocking starts.
// Buttons (Steam's enum): 1 OK 2 CANCEL 3 SECONDARY 4 OPTIONS 5/6 bumpers 7/8 triggers 9-12 dpad
// 13 SELECT 14 START 15/16 stick clicks 17/18 stick touch 23-26 rear buttons (24 left grip,
// 26 right grip) 27 STEAM_GUIDE 28 STEAM_QUICK_MENU 29 DUMMY_INPUT.
(() => {
const VERSION = 2;
if (!window.__ftreq) {
const name = Object.keys(window).find(k => k.startsWith("webpackChunk"));
window[name].push([[Symbol("frametop")], {}, r => { window.__ftreq = r; }]);
}
const E = window.__ftreq(17900).E;
const proto = E.prototype;
if (!("HandleControllerInputMessages" in proto)) throw new Error("Steam's gamepad source moved (module 17900)");
const base = Object.getPrototypeOf(proto);
const G = window.__frametopGaze || (window.__frametopGaze = {mode: "log", allow: [27, 28, 29], log: [], held: {}, dropped: 0});
G.modes = G.modes || [];
G.gamepadMode = () => {
try { return !!window.__ftreq(84114).J.Instance?.ShowGamepadFocusMode; } catch (e) { return false; }
};
G.note = (ev, button, controller, passed) => {
G.log.push([Math.round(performance.now()), ev, button, controller, passed]);
if (G.log.length > 256) G.log.shift();
};
G.blocks = button => {
if (G.allow.includes(button) || (G.until && Date.now() > G.until)) return false;
return G.mode === "block" || (G.mode === "auto" && !G.gamepadMode());
};
if (proto.OnButtonDown && proto.OnButtonDown.__frametop !== VERSION) {
delete proto.OnButtonDown;
delete proto.OnButtonUp;
delete proto.OnAnalogPad;
}
if (!Object.prototype.hasOwnProperty.call(proto, "OnButtonDown")) {
proto.OnButtonDown = function (button, controller, ...rest) {
let pass = true;
try {
const g = window.__frametopGaze;
g.inst = this;
pass = !g.blocks(button);
if (g.mode !== "off") g.note("down", button, controller, pass);
if (pass) g.held[controller + ":" + button] = true;
else g.dropped++;
} catch (e) { pass = true; }
if (pass) return base.OnButtonDown.call(this, button, controller, ...rest);
};
proto.OnButtonUp = function (button, controller, ...rest) {
let pass = true;
try {
const g = window.__frametopGaze, key = controller + ":" + button;
pass = !!g.held[key] || !g.blocks(button);
delete g.held[key];
if (g.mode !== "off") g.note("up", button, controller, pass);
} catch (e) { pass = true; }
if (pass) return base.OnButtonUp.call(this, button, controller, ...rest);
};
proto.OnAnalogPad = function (button, x, y, controller, ...rest) {
let pass = true;
try {
const g = window.__frametopGaze;
pass = !g.blocks(button);
if (g.mode !== "off" && (x || y)) g.note("analog", button, controller, pass);
} catch (e) { pass = true; }
if (pass) return base.OnAnalogPad.call(this, button, x, y, controller, ...rest);
};
for (const f of [proto.OnButtonDown, proto.OnButtonUp, proto.OnAnalogPad]) f.__frametop = VERSION;
G.installed = Date.now();
}
if (!G.sampler) {
let last = null;
G.sampler = setInterval(() => {
const m = G.gamepadMode() ? "gamepad" : "laser";
if (m !== last) {
last = m;
G.modes.push([Math.round(performance.now()), m]);
if (G.modes.length > 128) G.modes.shift();
}
}, 50);
}
return JSON.stringify({version: VERSION, mode: G.mode, gamepad: G.gamepadMode(), installed: G.installed,
dropped: G.dropped, modes: G.modes.slice(-10), log: G.log.slice(-10)});
})()
-40
View File
@@ -1,40 +0,0 @@
"""steamui: run JavaScript in Steam's UI (its SharedJSContext) through Steam's CEF debugger on
127.0.0.1:8080, with nothing but the standard library (input/vrws.py's web socket client).
evaluate(expression) the expression's value (JSON-able), or an OSError when Steam or its
UI isn't there, or a RuntimeError when the script threw
"""
import json
import time
import urllib.request
from vrws import VrSocket
DEBUGGER = "http://127.0.0.1:8080"
def evaluate(expression, target="SharedJSContext", timeout=2.0):
with urllib.request.urlopen(f"{DEBUGGER}/json", timeout=timeout) as resp:
pages = json.load(resp)
url = next((p.get("webSocketDebuggerUrl") for p in pages if p.get("title") == target), None)
if not url:
raise OSError(f"Steam's {target} isn't open")
ws = VrSocket(timeout=timeout, url=url)
try:
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
"params": {"expression": expression, "returnByValue": True}}))
deadline = time.monotonic() + timeout
while True:
left = deadline - time.monotonic()
msg = ws.recv(timeout=max(left, 0)) if left > 0 else None
if msg is None:
raise OSError("Steam's UI didn't answer")
if isinstance(msg, dict) and msg.get("id") == 1:
break
finally:
ws.close()
result = msg.get("result") or {}
if "exceptionDetails" in result:
details = result["exceptionDetails"]
raise RuntimeError((details.get("exception") or {}).get("description") or details.get("text", "error"))
return (result.get("result") or {}).get("value")
+219
View File
@@ -0,0 +1,219 @@
#!/usr/bin/env python3
"""Offline test of the input relay's key combinations and modifier taps.
Runs the relay's main() against a fake pass-through keyboard and a fake mouse (pipes), with
every socket it sends to renamed, no uinput devices, no grabs, and ft-steam swapped for a
logger. Nothing reaches the live desktop, SteamVR, or the running relay, so it's safe next
to them. Rules come from the test, not ~/.config/frametop-input.json.
input/test/keys-test.py [RELAY] (default: input/input-relay.py next to this folder)
"""
import fcntl
import importlib.util
import os
import shutil
import socket
import struct
import sys
import tempfile
import threading
import time
HERE = os.path.dirname(os.path.abspath(__file__))
relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "input-relay.py")
tag = f"ft_keys_test_{os.getpid()}"
with open(relay_path) as f:
src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
relay.__file__ = os.path.abspath(relay_path)
exec(compile(src, relay_path, "exec"), relay.__dict__)
for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "KEYS"):
setattr(relay, name, f"\0{tag}_{name.lower()}")
OUT = tempfile.mkdtemp(prefix="ft-keys-test-")
steam_log, cmd_log = f"{OUT}/steam.log", f"{OUT}/cmd.log"
relay.FT_STEAM = f"{OUT}/ft-steam"
with open(relay.FT_STEAM, "w") as f:
f.write(f'#!/bin/sh\necho "$@" >> {steam_log}\n')
os.chmod(relay.FT_STEAM, 0o755)
class NoDevice:
def __init__(self, *args, **kwargs):
pass
def emit(self, *args):
pass
def sync(self):
pass
relay.Virtual = NoDevice
relay.log = lambda *args: None # the relay's own log
bindings = {"now": None} # the rules' key_bindings; None: the relay's defaults
def read_rules(path=None):
rules = {"devices": {}, "buttons": {}, "controller_buttons": {}}
rules["key_bindings"] = dict(relay.DEFAULT_KEY_BINDINGS if bindings["now"] is None else bindings["now"])
return rules
relay.read_rules = read_rules
relay.read_config = lambda path=None: {"POINTER": "0"}
# The fake devices: /dev/input/event900 (keyboard) and event901 (mouse), each a pipe.
kb_r, kb_w = os.pipe()
ms_r, ms_w = os.pipe()
for fd in (kb_r, ms_r):
os.set_blocking(fd, False)
FAKE = {"/dev/input/event900": kb_r, "/dev/input/event901": ms_r}
HELD = {kb_r: set(), ms_r: set()} # what EVIOCGKEY says each holds
class Inode:
st_ino = 1
fake_os = type(os)("os")
fake_os.__dict__.update(os.__dict__)
fake_os.listdir = lambda path: ["event900", "event901"] if path == "/dev/input" else os.listdir(path)
fake_os.stat = lambda path, *a, **k: Inode() if path in FAKE else os.stat(path, *a, **k)
relay.os = fake_os
def ioctl(fd, request, arg=0, *rest):
if fd in HELD and request == relay.EVIOCGKEY:
for c in HELD[fd]:
arg[c // 8] |= 1 << (c % 8)
return 0
return fcntl.ioctl(fd, request, arg, *rest)
fake_fcntl = type(fcntl)("fcntl")
fake_fcntl.__dict__.update(fcntl.__dict__)
fake_fcntl.ioctl = ioctl
relay.fcntl = fake_fcntl
def probe(path):
if FAKE[path] == kb_r:
return relay.Node(path, kb_r, "test keyboard", relay.BUS_USB, 1, 2, "", False, True)
return relay.Node(path, ms_r, "test mouse", relay.BUS_USB, 3, 4, "", True, False)
relay.probe = probe
screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
screens.bind(relay.SCREENS)
screens.settimeout(0.05)
EVENT = struct.Struct("llHHi")
def send(fd, etype, code, value):
os.write(fd, EVENT.pack(0, 0, etype, code, value) + EVENT.pack(0, 0, 0, 0, 0))
time.sleep(0.03)
def key(code, value, fd=kb_w):
held = HELD[ms_r if fd == ms_w else kb_r]
(held.add if value else held.discard)(code)
send(fd, relay.EV_KEY, code, value)
def typed():
"""What the desktop was sent since the last call."""
got = []
while True:
try:
got.append(screens.recv(256).decode())
except socket.timeout:
return got
def use(key_bindings):
bindings["now"] = key_bindings
c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
c.bind("")
c.settimeout(2)
c.sendto(b"reload", f"\0{tag}_relay")
c.recv(4096)
time.sleep(0.05)
typed()
def lines(path, wait=0.3):
time.sleep(wait)
try:
with open(path) as f:
return f.read().split()
except OSError:
return []
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
META, RMETA, SHIFT, CTRL, RCTRL, A, F, J = 125, 126, 42, 29, 97, 30, 33, 36
F24 = ["key 194 1", "key 194 0"]
def tests():
time.sleep(1.5) # the relay's first device scan
typed()
key(META, 1); key(META, 0)
check("Meta tap (default): Steam menu", lines(steam_log), ["menu"])
check("Meta tap: the desktop gets F24 before Meta's release", typed(), ["key 125 1"] + F24 + ["key 125 0"])
key(RMETA, 1); key(RMETA, 2); key(RMETA, 2); key(RMETA, 0)
check("right Meta held (autorepeat), then released: a tap", lines(steam_log), ["menu", "menu"])
typed()
key(META, 1); key(A, 1); key(A, 0); key(META, 0)
check("Meta+A: no tap", lines(steam_log), ["menu", "menu"])
check("Meta+A (unbound) is typed as is", typed(), ["key 125 1", "key 30 1", "key 30 0", "key 125 0"])
key(META, 1); key(relay.BTN_LEFT, 1, ms_w); key(relay.BTN_LEFT, 0, ms_w); key(META, 0)
check("Meta+click: no tap", lines(steam_log), ["menu", "menu"])
key(META, 1); send(ms_w, relay.EV_REL, relay.REL_WHEEL, 1); key(META, 0)
check("Meta+scroll: no tap", lines(steam_log), ["menu", "menu"])
key(META, 1); key(SHIFT, 1); key(SHIFT, 0); key(META, 0)
check("Meta+Shift: no tap", lines(steam_log), ["menu", "menu"])
typed()
key(META, 1); key(J, 1); key(J, 0); key(META, 0)
check("Meta+J (default gaze click): no tap", lines(steam_log), ["menu", "menu"])
check("Meta+J: F24, Meta up early, its real release dropped", typed(), ["key 125 1"] + F24 + ["key 125 0"])
use({"29+42+33": f"command:echo combo >> {cmd_log}", "125": "none"})
key(CTRL, 1); key(SHIFT, 1); key(F, 1); key(F, 0); key(SHIFT, 0); key(CTRL, 0)
check("Ctrl+Shift+F runs its command", lines(cmd_log), ["combo"])
check("the combination's last key isn't typed", [k for k in typed() if k.startswith("key 33 ")], [])
key(META, 1); key(META, 0)
check("Meta tap bound to none: nothing", lines(steam_log), ["menu", "menu"])
check("Meta tap bound to none: no F24", typed(), ["key 125 1", "key 125 0"])
use({"29": f"command:command -v ft-steam ft-layout ft-float | wc -l >> {cmd_log}"})
key(RCTRL, 1); key(RCTRL, 0)
check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "3"])
check("Ctrl tap: F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
check("an empty command isn't an action", relay.known_action("command: "), False)
check("steam_menu and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("command:ls")), (False, False))
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
shutil.rmtree(OUT, ignore_errors=True)
os._exit(1 if failures else 0)
threading.Thread(target=tests, daemon=True).start()
sys.argv = [relay_path, "--no-grab"]
relay.main()
-220
View File
@@ -1,220 +0,0 @@
#!/usr/bin/python3
"""vrws: vrserver's local web socket, with nothing but the standard library.
vrserver (SteamVR) serves its controller binding page on 127.0.0.1:27062, and that page
follows the controllers' raw input through a web socket there. Reading it takes nothing
from anyone: whatever the buttons are bound to still happens, and it works whatever app
has focus, in games too. frame-voice reads its push-to-talk button the same way. The host's
Python has no `websockets` module, so this is a small RFC 6455 client of its own.
getstate() the devices SteamVR has now, from /input/getstate.json: each with its
root path (/user/hand/left, /devices/cv/<serial> while our pointer holds
that hand, /user/head), controller type, side, and input components
VrSocket() a connection: open(mailbox), subscribe(device), recv() -> dict or None
Run as a program, it prints component changes as they come, for the gaze-first tests
(docs/gaze-first.md, test 4):
input/vrws.py [--all] [--seconds N] [component ...]
By default it follows the buttons (…/click), the thumbsticks' axes, the trigger's value,
and the headset's proximity; --all shows every component. At the end it prints how often
each one updated.
"""
import base64
import json
import os
import select
import socket
import struct
import sys
import time
import urllib.request
HOST, PORT = "127.0.0.1", 27062
ORIGIN = f"http://{HOST}:{PORT}"
HEADERS = {"Referer": f"{ORIGIN}/dashboard/controllerbinding.html"}
def getstate(timeout=3):
"""The devices SteamVR has now (only the ones with a root path)."""
req = urllib.request.Request(f"{ORIGIN}/input/getstate.json", headers=HEADERS)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return [d for d in json.load(resp).get("devices", []) if d.get("root_path")]
class VrSocket:
"""vrserver's web socket, or with `url` (ws://host:port/path) any other local one, such as
Steam's CEF debugger (input/steamui.py)."""
def __init__(self, timeout=3, url=None):
if url:
hostport, path = url.removeprefix("ws://").split("/", 1)
host, port = hostport.rsplit(":", 1)
extra = ""
else:
host, port, path, hostport = HOST, PORT, "", f"{HOST}:{PORT}"
extra = f"Origin: {ORIGIN}\r\nReferer: {HEADERS['Referer']}\r\n"
self.sock = socket.create_connection((host, int(port)), timeout=timeout)
key = base64.b64encode(os.urandom(16)).decode()
self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {hostport}\r\nUpgrade: websocket\r\n"
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n"
f"{extra}\r\n").encode())
head = b""
while b"\r\n\r\n" not in head:
chunk = self.sock.recv(4096)
if not chunk:
raise OSError("vrserver closed the connection during the handshake")
head += chunk
head, self.buf = head.split(b"\r\n\r\n", 1)
status = head.split(b"\r\n", 1)[0]
if b" 101 " not in status + b" ":
raise OSError(f"vrserver refused the web socket: {status.decode(errors='replace')}")
self.sock.settimeout(None)
self.parts = b""
def fileno(self):
return self.sock.fileno()
def close(self):
try:
self._frame(0x8, b"")
except OSError:
pass
self.sock.close()
def _frame(self, opcode, payload):
n = len(payload)
head = bytes([0x80 | opcode])
if n < 126:
head += bytes([0x80 | n])
elif n < 65536:
head += bytes([0x80 | 126]) + struct.pack(">H", n)
else:
head += bytes([0x80 | 127]) + struct.pack(">Q", n)
mask = os.urandom(4)
self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
def send(self, text):
self._frame(0x1, text.encode())
def open(self, mailbox):
self.send(f"mailbox_open {mailbox}")
self.mailbox = mailbox
def subscribe(self, device, on=True):
kind = "request_input_state_updates" if on else "cancel_input_state_updates"
self.send("mailbox_send input_server " +
json.dumps({"type": kind, "device_path": device, "returnAddress": self.mailbox}))
def _read(self, n):
while len(self.buf) < n:
chunk = self.sock.recv(65536)
if not chunk:
raise OSError("vrserver closed the web socket")
self.buf += chunk
out, self.buf = self.buf[:n], self.buf[n:]
return out
def pending(self):
"""A message (or part of one) is already read and waiting."""
return bool(self.buf)
def recv(self, timeout=None):
"""The next text message as parsed JSON (None for one that isn't JSON), or None at
the timeout. Raises OSError when the connection ends."""
if not self.buf and timeout is not None:
if not select.select([self.sock], [], [], timeout)[0]:
return None
while True:
b0, b1 = self._read(2)
opcode, n = b0 & 0x0F, b1 & 0x7F
if n == 126:
n = struct.unpack(">H", self._read(2))[0]
elif n == 127:
n = struct.unpack(">Q", self._read(8))[0]
mask = self._read(4) if b1 & 0x80 else None
data = self._read(n)
if mask:
data = bytes(b ^ mask[i % 4] for i, b in enumerate(data))
if opcode == 0x8:
raise OSError("vrserver closed the web socket")
if opcode == 0x9:
self._frame(0xA, data)
continue
if opcode in (0x1, 0x2, 0x0):
self.parts += data
if not b0 & 0x80:
continue
text, self.parts = self.parts, b""
try:
return json.loads(text)
except ValueError:
return None
def main():
args = [a for a in sys.argv[1:]]
show_all = "--all" in args
seconds = 0.0
if "--seconds" in args:
i = args.index("--seconds")
seconds = float(args[i + 1])
del args[i:i + 2]
wanted = [a for a in args if not a.startswith("--")]
def followed(name):
if show_all:
return True
if wanted:
return name in wanted
return (name.endswith("/click") or name in ("/input/thumbstick/x", "/input/thumbstick/y",
"/input/trigger/value", "/proximity"))
names = {}
ws = VrSocket()
ws.open(f"frametop_vrws_{os.getpid()}")
def follow_new():
"""Subscribe to devices that appeared (vrserver only streams changes, and a device
that connects later, or changes its path with a role, is a new subscription)."""
for d in getstate():
path = d["root_path"]
name = f"{path} ({d.get('controller_type', '?')}{', ' + d['side'] if d.get('side') else ''})"
if names.get(path) != name:
names[path] = name
print(f"{time.monotonic() - start:9.3f} device {name}", flush=True)
ws.subscribe(path)
start = time.monotonic()
follow_new()
next_poll = start + 2
last, counts = {}, {}
try:
while not seconds or time.monotonic() - start < seconds:
if time.monotonic() >= next_poll:
next_poll = time.monotonic() + 2
follow_new()
msg = ws.recv(timeout=0.5)
if not isinstance(msg, dict) or msg.get("type") != "update_component_states":
continue
dev = msg.get("device")
for name, value in (msg.get("components") or {}).items():
if not followed(name):
continue
key = (dev, name)
counts[key] = counts.get(key, 0) + 1
if last.get(key) != value:
last[key] = value
print(f"{time.monotonic() - start:9.3f} {names.get(dev, dev)} {name} = {value}", flush=True)
except KeyboardInterrupt:
pass
finally:
ws.close()
took = max(time.monotonic() - start, 1e-6)
for (dev, name), n in sorted(counts.items()):
print(f"updates: {names.get(dev, dev)} {name}: {n} ({n / took:.1f}/s)")
if __name__ == "__main__":
main()
+62 -20
View File
@@ -1,12 +1,13 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Install everything on the Steam Frame: the build container, Frametop (multi-screen # Install everything on the Steam Frame: the build container, Frametop (multi-screen
# desktop, input relay, universal 3D mouse, settings app), and optionally the Bluetooth # desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode, our own
# fixes and hand tracking. Run it on the headset in a terminal, from this repo. It's safe # eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
# to re-run, for example after `git pull`. # for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
# (It also works from a PC over SSH; see "Developing from a PC" in the README.) # (It also works from a PC over SSH; see "Developing from a PC" in the README.)
# #
# Usage: ./install.sh [--yes] [--no-bluetooth] # Usage: ./install.sh [--yes] [--no-bluetooth]
# --yes don't ask; skips the Bluetooth fixes and the SteamVR restart # --yes don't ask; installs gaze mode, and our eye tracker if sudo can run without
# a password prompt; skips the Bluetooth fixes and the SteamVR restart
# --no-bluetooth don't offer the Bluetooth fixes # --no-bluetooth don't offer the Bluetooth fixes
set -euo pipefail set -euo pipefail
@@ -18,7 +19,7 @@ for arg in "$@"; do
case $arg in case $arg in
--yes) assume_yes=1 ;; --yes) assume_yes=1 ;;
--no-bluetooth) bluetooth=0 ;; --no-bluetooth) bluetooth=0 ;;
-h|--help) sed -n '2,11p' "$0"; exit 0 ;; -h|--help) sed -n '2,12p' "$0"; exit 0 ;;
*) echo "unknown option: $arg" >&2; exit 2 ;; *) echo "unknown option: $arg" >&2; exit 2 ;;
esac esac
done done
@@ -37,6 +38,13 @@ ask() { # ask "question" default(y|n)
answer=${answer:-$2} answer=${answer:-$2}
[[ $answer =~ ^[Yy] ]] [[ $answer =~ ^[Yy] ]]
} }
# Can sudo run here without a password prompt (SUDO_ASKPASS, the repo's .env, a recent sudo)?
# See frame_sudo in scripts/_env.sh.
sudo_quiet() {
{ [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; } && return 0
grep -q '^steamos_root_pwd=.' "$REPO_ROOT/.env" 2>/dev/null && return 0
on_frame 'sudo -n true' 2>/dev/null
}
if [ "$FRAME_LOCAL" = 1 ]; then if [ "$FRAME_LOCAL" = 1 ]; then
echo "Installing on this Steam Frame from $FRAME_REPO" echo "Installing on this Steam Frame from $FRAME_REPO"
@@ -44,8 +52,12 @@ else
echo "Installing on $FRAME_HOST over SSH (repo copy at $FRAME_REPO)" echo "Installing on $FRAME_HOST over SSH (repo copy at $FRAME_REPO)"
"$root/scripts/sync.sh" >/dev/null "$root/scripts/sync.sh" >/dev/null
fi fi
if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
echo "Over SSH: if the connection drops, run this again (it keeps what it downloaded). Services"
echo "that need SteamVR start with it if it isn't running now."
fi
step "1/9 distrobox (container tool, installed in your home folder)" step "1/10 distrobox (container tool, installed in your home folder)"
if on_frame 'test -x ~/.local/bin/distrobox'; then if on_frame 'test -x ~/.local/bin/distrobox'; then
echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')" echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
else else
@@ -55,25 +67,25 @@ else
cd ~/dev/src/distrobox && ./install --prefix ~/.local' cd ~/dev/src/distrobox && ./install --prefix ~/.local'
fi fi
step "2/9 build container (Fedora 44 'dev', about 1-2 GB the first time)" step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
"$root/setup/dev-container.sh" "$root/setup/dev-container.sh"
step "3/9 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)" step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
"$root/desktops.sh" relay install "$root/desktops.sh" relay install
step "4/9 3D mouse: SteamVR driver" step "4/10 3D mouse: SteamVR driver"
"$root/pointer/driver/build.sh" "$root/pointer/driver/build.sh"
"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open "$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
step "5/9 3D mouse: pointer helper service" step "5/10 3D mouse: pointer helper service"
"$root/pointer/helper/build.sh" "$root/pointer/helper/build.sh"
"$root/pointer/helper/run.sh" install "$root/pointer/helper/run.sh" install
step "6/9 power service (turns the displays off while the headset isn't used, even on a stand)" step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
"$root/power/build.sh" "$root/power/build.sh"
"$root/power/run.sh" install "$root/power/run.sh" install
step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings" step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
"$root/screens/build.sh" "$root/screens/build.sh"
"$root/desktops.sh" install >/dev/null "$root/desktops.sh" install >/dev/null
"$root/input-settings/install.sh" "$root/input-settings/install.sh"
@@ -82,20 +94,50 @@ step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Framet
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf" on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)" echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
step "8/9 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)" step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
gaze=0
if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
"$root/gaze/run.sh" install
gaze=1
else
echo "skipped. Install later with: gaze/run.sh install"
fi
# Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
# uses SteamVR's eye tracker.
step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
if [ "$gaze" = 0 ]; then
echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: gaze/tracker/install.sh"
elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo), and it downloads about 165 MB (numpy, OpenCV)." y; then
if "$root/gaze/tracker/install.sh" install; then
# Configs made before GAZE_TRACKER=auto say steam, which keeps SteamVR's.
tracker=$(on_frame "sed -n 's/^GAZE_TRACKER=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1")
if [ "$tracker" = steam ] &&
ask "Your settings pick SteamVR's eye tracker (GAZE_TRACKER=steam, the old default). Use ours instead?" y; then
on_frame "sed -i 's~^GAZE_TRACKER=steam\b.*~GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when installed, else SteamVR | own | steam~' ~/.config/frametop.conf"
tracker=auto
fi
if [ "$tracker" = steam ]; then
echo "installed; your settings keep SteamVR's eye tracker (pick Own tracker on the Gaze page to use ours)"
else
echo "gaze mode uses our eye tracker: calibrate it once, with Calibrate on the Gaze page"
fi
else
echo "our eye tracker didn't install, so gaze mode uses SteamVR's. Try again with: gaze/tracker/install.sh"
fi
else
echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
fi
step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
"$root/setup/bluetooth/install.sh" install "$root/setup/bluetooth/install.sh" install
else else
echo "skipped. Install later with: setup/bluetooth/install.sh install" echo "skipped. Install later with: setup/bluetooth/install.sh install"
fi fi
step "9/9 hand tracking (optional, experimental: your hands show over the screens)"
if ask "Install hand tracking? It needs your password (sudo) to let its camera service read the headset cameras." n; then
"$root/hands/run.sh" install
else
echo "skipped. Install later with: hands/run.sh install"
fi
step "Done" step "Done"
cat <<'EOF' cat <<'EOF'
SteamVR has to restart once, to load the 3D mouse driver and to start the input relay SteamVR has to restart once, to load the 3D mouse driver and to start the input relay
+248 -3
View File
@@ -18,6 +18,10 @@ you face (yaw only), like a recenter. It lives in ~/.config/frametop-layout.json
"layouts": {"Work": [{"pos": ..., "face": ..., "roll": ..., "metres": ..., "curve": ..., "layouts": {"Work": [{"pos": ..., "face": ..., "roll": ..., "metres": ..., "curve": ...,
"pin": ...}, ...]}, named layouts: each screen's place (SPATIAL) "pin": ...}, ...]}, named layouts: each screen's place (SPATIAL)
"active": "Work", the named layout the custom arrangement came from "active": "Work", the named layout the custom arrangement came from
"profiles": {"Work": {"hidden": [3], "windows": [...]}}, what a named layout opens too
(docs/profiles.md): screens hidden on their own, and the
apps' windows (ft-floatd's "windows")
"default_profile": "Work", the profile the desktop starts with (FT_PROFILE overrides)
"visibility": {"mode": "always", ft-screens: always | dashboard (only with the SteamVR "visibility": {"mode": "always", ft-screens: always | dashboard (only with the SteamVR
"wrist_angle": 60, dashboard open) | gesture (while you look at a controller) "wrist_angle": 60, dashboard open) | gesture (while you look at a controller)
"gesture_hand": "left", "gesture_angle": 20}, | toggle (hidden until shown); "gesture_hand": "left", "gesture_angle": 20}, | toggle (hidden until shown);
@@ -28,6 +32,8 @@ you face (yaw only), like a recenter. It lives in ~/.config/frametop-layout.json
"pin": {"hand": "left", "rel": [12]}, ft-screens: riding on that controller "pin": {"hand": "left", "rel": [12]}, ft-screens: riding on that controller
(left | right), or on the headset (head) (left | right), or on the headset (head)
"scale": 1.0, KWin output scale (1.0 = 100%) "scale": 1.0, KWin output scale (1.0 = 100%)
"hidden": true, ft-screens: hidden on its own, whatever the
visibility mode (ft-layout hide N)
"pos": [x, y, z], "face": [yaw, pitch], "roll": 0, custom layout "pos": [x, y, z], "face": [yaw, pitch], "roll": 0, custom layout
"rotation": "normal" | "left" | "right"}, ...], gamescope only "rotation": "normal" | "left" | "right"}, ...], gamescope only
"panel_size": [w, h]} gamescope: last measured panel size "panel_size": [w, h]} gamescope: last measured panel size
@@ -42,16 +48,26 @@ Usage (on the Frame host; Frametop Display Settings calls it too):
ft-layout apply [--wait SECONDS] arrange every screen; --wait is for desktop start: ft-layout apply [--wait SECONDS] arrange every screen; --wait is for desktop start:
wait for the screens, skip if "auto" is off wait for the screens, skip if "auto" is off
ft-layout capture save the current arrangement as the custom layout ft-layout capture save the current arrangement as the custom layout
ft-layout save NAME save it as a named layout too, and use that ft-layout save NAME save it as a named layout too, and use that; with the
ft-layout use NAME switch to a named layout and arrange the screens in it desktop's apps and hidden screens, as a profile
ft-layout use NAME switch to a named layout, arrange the screens in it, and
open its apps (moving open windows, nothing closed)
ft-layout start [--wait SECONDS] desktop start: the profile in FT_PROFILE or default_profile,
or else as apply --wait
ft-layout open NAME a profile's launcher entry: use it, or start the desktop in it
ft-layout default NAME|none the profile the desktop starts with
ft-layout layouts list the named layouts (* = the one in use) ft-layout layouts list the named layouts (* = the one in use)
ft-layout rename OLD NEW | delete NAME ft-layout rename OLD NEW | delete NAME
ft-layout launchers write each profile's launcher entry again (and drop stale ones)
ft-layout plan print the arrangement as JSON (no VR needed) ft-layout plan print the arrangement as JSON (no VR needed)
ft-layout scale per-screen scale, positions (as the screens are around ft-layout scale per-screen scale, positions (as the screens are around
you), and primary to KWin you), and primary to KWin
ft-layout screen-args ft-screens' --screen arguments for the session script ft-layout screen-args ft-screens' --screen arguments for the session script
ft-layout remote-view the primary screen's place in the workspace, for the VNC bridge ft-layout remote-view the primary screen's place in the workspace, for the VNC bridge
ft-layout toggle hide or show all screens (ft-screens) ft-layout toggle hide or show all screens (ft-screens)
ft-layout hide N|all hide a screen on its own (it stays hidden whatever the
ft-layout show N|all visibility mode or the hotkey say), or show it again
ft-layout hidden the screens hidden on their own
ft-layout pin all|N left|right|head pin screens to a wrist or your head as they are; ft-layout pin all|N left|right|head pin screens to a wrist or your head as they are;
unpin all|N unpin all|N
""" """
@@ -71,6 +87,9 @@ VRCMD = "/opt/steamvr/bin/linuxarm64/vrcmd"
HELPER = "\0ft_pointer_helper" HELPER = "\0ft_pointer_helper"
SCREENS = "\0ft_screens" SCREENS = "\0ft_screens"
LOCK_PATH = "/tmp/ft-layout.lock" LOCK_PATH = "/tmp/ft-layout.lock"
FLOAT = "\0frametop_float" # ft-floatd: the apps' windows (profiles)
REPO = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
LAUNCHERS = os.path.expanduser("~/.local/share/applications") # a profile's launcher entry each
DEFAULT_PANEL = (1.18, 0.664) # gamescope: a floating 16:9 dashboard panel, measured on the Frame DEFAULT_PANEL = (1.18, 0.664) # gamescope: a floating 16:9 dashboard panel, measured on the Frame
PIXELS_PER_METRE = 800 # ft-screens: a new screen's default size in VR (1920 px: 2.4 m) PIXELS_PER_METRE = 800 # ft-screens: a new screen's default size in VR (1920 px: 2.4 m)
# controllers: when controllers' lasers work the screens (always | outside_games | dashboard). # controllers: when controllers' lasers work the screens (always | outside_games | dashboard).
@@ -300,6 +319,10 @@ def plan(layout, count, panel_size=None):
p = layout["preset"] p = layout["preset"]
rows = max(1, min(int(p.get("rows", 1)), count)) rows = max(1, min(int(p.get("rows", 1)), count))
cols = math.ceil(count / rows) cols = math.ceil(count / rows)
# A row setting above what the screens fill leaves empty grid rows (4 screens,
# 3 rows -> cols 2 -> a third row with nothing in it), and max() over an empty
# row crashes plan(). Trim rows to what the screens actually fill.
rows = max(1, math.ceil(count / cols))
d = max(0.3, float(p.get("distance", 2.0))) d = max(0.3, float(p.get("distance", 2.0)))
gap = max(0.0, float(p.get("gap", 0.05))) gap = max(0.0, float(p.get("gap", 0.05)))
height = float(p.get("height", 0.0)) height = float(p.get("height", 0.0))
@@ -393,6 +416,42 @@ def send_visibility(sock, layout):
sock.ask(f"ingames {v['in_games']}") sock.ask(f"ingames {v['in_games']}")
def send_hidden(sock, layout):
"""Screens hidden on their own (ft-screens' conceal/reveal)."""
for i in range(screen_count(layout)):
word = "conceal" if screen_entry(layout, i).get("hidden") else "reveal"
try:
sock.ask(f"{word} {i + 1}")
except RuntimeError as e:
log(f"screens hidden on their own: {e}") # an ft-screens from before conceal
return
def set_hidden(which, hidden):
"""Hide (or show) screen N (1-based) or "all" on its own: saved, and applied if the
desktop runs."""
layout = load_layout()
n = screen_count(layout)
picked = range(n) if which == "all" else [int(which) - 1] if which.isdigit() else []
if not picked or not all(0 <= i < n for i in picked):
raise RuntimeError(f"no screen {which} (1 to {n})")
screens = layout.setdefault("screens", [])
while len(screens) < n:
screens.append({})
for i in picked:
if hidden:
screens[i]["hidden"] = True
else:
screens[i].pop("hidden", None)
save_layout(layout)
try:
sock = screens_socket()
for i in picked:
sock.ask(f"{'conceal' if hidden else 'reveal'} {i + 1}")
except RuntimeError as e:
log(f"saved; not applied now: {e}")
def parse_get(reply): def parse_get(reply):
"""ft-screens' "get": pose, size, curve, and the pin (hand and controller->screen).""" """ft-screens' "get": pose, size, curve, and the pin (hand and controller->screen)."""
f = reply.split()[1:] f = reply.split()[1:]
@@ -441,6 +500,7 @@ def apply_screens(wait=0):
sock.ask(f"pin {i + 1} {pin['hand']} " + " ".join(f"{v:.5f}" for v in pin["rel"])) sock.ask(f"pin {i + 1} {pin['hand']} " + " ".join(f"{v:.5f}" for v in pin["rel"]))
except RuntimeError as e: except RuntimeError as e:
log(f"screen {i + 1}: {e}") # that controller isn't on log(f"screen {i + 1}: {e}") # that controller isn't on
send_hidden(sock, layout)
try: try:
sock.ask("vrkeyboard close") # the keyboard, if open, goes too: a reset starts over sock.ask("vrkeyboard close") # the keyboard, if open, goes too: a reset starts over
except RuntimeError: except RuntimeError:
@@ -654,6 +714,11 @@ def rename_named(layout, old, new):
if new != old and new in named: if new != old and new in named:
raise RuntimeError(f"there's already a layout called {new!r}") raise RuntimeError(f"there's already a layout called {new!r}")
named[new] = named.pop(old) named[new] = named.pop(old)
profiles = layout.get("profiles", {})
if old in profiles:
profiles[new] = profiles.pop(old)
if layout.get("default_profile") == old:
layout["default_profile"] = new
if layout.get("active") == old: if layout.get("active") == old:
layout["active"] = new layout["active"] = new
return new return new
@@ -663,10 +728,133 @@ def delete_named(layout, name):
"""The screens stay where the layout put them, as an unnamed custom arrangement.""" """The screens stay where the layout put them, as an unnamed custom arrangement."""
if layout.get("layouts", {}).pop(name, None) is None: if layout.get("layouts", {}).pop(name, None) is None:
raise RuntimeError(f"no layout called {name!r}") raise RuntimeError(f"no layout called {name!r}")
layout.get("profiles", {}).pop(name, None)
if layout.get("default_profile") == name:
layout.pop("default_profile")
if layout.get("active") == name: if layout.get("active") == name:
layout.pop("active") layout.pop("active")
# ---------------------------------------------------------------- profiles (docs/profiles.md)
def ask_float(text, timeout=6.0):
"""ft-floatd (floating windows, in the desktop's session); None if it isn't running."""
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind("")
sock.settimeout(timeout)
try:
sock.sendto(text.encode(), FLOAT)
return sock.recv(1 << 20).decode()
except OSError:
return None
finally:
sock.close()
def capture_profile(layout, name):
"""The desktop's apps and hidden screens into profile `name` (kept as they were if
ft-floatd doesn't answer)."""
hidden = [i + 1 for i in range(screen_count(layout)) if screen_entry(layout, i).get("hidden")]
profile = layout.setdefault("profiles", {}).setdefault(name, {})
profile["hidden"] = hidden
reply = ask_float("windows")
if reply and reply.startswith("ok "):
profile["windows"] = json.loads(reply[3:])
log(f"profile {name!r}: {len(profile['windows'])} windows, hidden screens {hidden or 'none'}")
else:
log(f"profile {name!r}: the apps weren't saved ({reply or 'ft-floatd is not running'})")
def use_hidden(layout, name):
"""A profile's hidden screens as the screens' own setting (applied with the arrangement)."""
profile = layout.get("profiles", {}).get(name)
if profile is None:
return
hidden = set(profile.get("hidden", []))
screens = layout.setdefault("screens", [])
for i in range(screen_count(layout)):
while len(screens) <= i:
screens.append({})
if i + 1 in hidden:
screens[i]["hidden"] = True
else:
screens[i].pop("hidden", None)
def open_apps(name, wait=0):
"""Have ft-floatd open a profile's apps (waiting up to `wait` seconds for it to start)."""
if not load_layout().get("profiles", {}).get(name, {}).get("windows"):
return
deadline = time.time() + wait
while True:
reply = ask_float(f"profile {name}")
if reply is not None or time.time() >= deadline:
break
time.sleep(1)
log(f"apps: {reply or 'ft-floatd is not running'}")
def launcher_name(name):
slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") or "profile"
return f"frametop-profile-{slug}.desktop"
def write_launchers(layout):
"""A launcher entry for each profile (SteamVR's Launch a program list, the Application
Launcher, KRunner), and none for ones that are gone."""
os.makedirs(LAUNCHERS, exist_ok=True)
want = {}
for name in layout_names(layout):
# Quoted for Exec (\" \` \$ \\), then each backslash doubled for the key file.
quoted = ('"' + re.sub(r'(["`$\\])', r"\\\1", name) + '"').replace("\\", "\\\\")
want[launcher_name(name)] = "\n".join([
"[Desktop Entry]", "Type=Application", f"Name=Frametop: {name}",
"Comment=Open the Frametop desktop in this profile: its screens and apps",
f"Exec={os.path.join(REPO, 'layout', 'ft-layout')} open {quoted}",
"Icon=preferences-desktop-display", "Categories=Utility;", "X-Frametop-Profile=true", ""])
for f in os.listdir(LAUNCHERS):
if f.startswith("frametop-profile-") and f.endswith(".desktop") and f not in want:
os.remove(os.path.join(LAUNCHERS, f))
for f, text in want.items():
path = os.path.join(LAUNCHERS, f)
try:
with open(path) as old:
if old.read() == text:
continue
except OSError:
pass
with open(path, "w") as out:
out.write(text)
def start_profile(layout):
"""The profile the desktop starts with: FT_PROFILE, else default_profile."""
name = os.environ.get("FT_PROFILE") or layout.get("default_profile")
return name if name and name in layout.get("layouts", {}) else None
def desktop_running():
try:
screens_socket().ask("screens", timeout=2)
return True
except RuntimeError:
return False
def start_desktop(profile):
"""Start the Frametop desktop in a profile (as desktops.sh start does)."""
if subprocess.run(["pgrep", "-x", "vrcompositor"], capture_output=True).returncode != 0:
raise RuntimeError("SteamVR isn't running")
subprocess.run(["systemctl", "--user", "reset-failed", "frametop-desktop"], capture_output=True)
session = os.path.join(REPO, "session", "frametop-session.sh")
r = subprocess.run(["systemd-run", "--user", "--collect", "--quiet", "--unit", "frametop-desktop",
f"--setenv=FT_PROFILE={profile}", "bash", "-c",
f'exec "{session}" > /tmp/frametop-session.log 2>&1'], capture_output=True, text=True)
if r.returncode != 0:
raise RuntimeError(f"couldn't start the desktop: {r.stderr.strip()}")
log(f"starting the desktop in {profile!r}")
# ---------------------------------------------------------------- KWin (scale, positions, primary) # ---------------------------------------------------------------- KWin (scale, positions, primary)
def nested_env(): def nested_env():
@@ -855,6 +1043,11 @@ def main(argv):
print(remote_view()) print(remote_view())
elif cmd == "toggle": elif cmd == "toggle":
log(screens_socket().ask("toggle")) log(screens_socket().ask("toggle"))
elif cmd in ("hide", "show") and len(argv) == 3:
set_hidden(argv[2], cmd == "hide")
elif cmd == "hidden":
layout = load_layout()
print(" ".join(str(i + 1) for i in range(screen_count(layout)) if screen_entry(layout, i).get("hidden")))
elif cmd == "layouts": elif cmd == "layouts":
layout = load_layout() layout = load_layout()
for name in layout_names(layout): for name in layout_names(layout):
@@ -866,6 +1059,48 @@ def main(argv):
else: else:
delete_named(layout, argv[2]) delete_named(layout, argv[2])
save_layout(layout) save_layout(layout)
write_launchers(layout)
elif cmd == "launchers":
write_launchers(load_layout())
elif cmd == "default" and len(argv) == 3:
layout = load_layout()
if argv[2] == "none":
layout.pop("default_profile", None)
elif argv[2] in layout.get("layouts", {}):
layout["default_profile"] = argv[2]
else:
raise RuntimeError(f"no profile called {argv[2]!r}")
save_layout(layout)
elif cmd == "open" and len(argv) == 3:
if argv[2] not in load_layout().get("layouts", {}):
raise RuntimeError(f"no profile called {argv[2]!r}")
if desktop_running():
return main([argv[0], "use", argv[2]])
start_desktop(argv[2])
elif cmd == "start":
# Desktop start (the session script): the profile it starts with, or the arrangement.
layout = load_layout()
name = start_profile(layout)
if not name:
return main([argv[0], "apply"] + argv[2:])
use_named(layout, name)
use_hidden(layout, name)
save_layout(layout)
log(f"starting in profile {name!r}")
wait = float(argv[argv.index("--wait") + 1]) if "--wait" in argv else 60
with open(LOCK_PATH, "w") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
apply(wait)
for _ in range(30): # Plasma may still be starting
try:
log("kwin: " + (" ".join(apply_scales()) or "unchanged"))
break
except RuntimeError as e:
last = e
time.sleep(1)
else:
log(f"kwin: {last}")
open_apps(name, wait=90) # ft-floatd starts with Plasma
elif cmd in ("pin", "unpin") and len(argv) >= 3: elif cmd in ("pin", "unpin") and len(argv) >= 3:
log(screens_socket().ask(" ".join(argv[1:]))) log(screens_socket().ask(" ".join(argv[1:])))
kwin_follow() # pinned screens go last kwin_follow() # pinned screens go last
@@ -887,6 +1122,7 @@ def main(argv):
while screens_up(sock) < screen_count() and time.time() < deadline: while screens_up(sock) < screen_count() and time.time() < deadline:
time.sleep(1) time.sleep(1)
send_visibility(sock, load_layout()) send_visibility(sock, load_layout())
send_hidden(sock, load_layout())
except RuntimeError as e: except RuntimeError as e:
log(f"visibility: {e}") log(f"visibility: {e}")
else: else:
@@ -912,20 +1148,29 @@ def main(argv):
check_name(argv[2]) check_name(argv[2])
capture() capture()
layout = load_layout() layout = load_layout()
log(f"saved layout {save_named(layout, argv[2])!r}") name = save_named(layout, argv[2])
capture_profile(layout, name)
save_layout(layout) save_layout(layout)
write_launchers(layout)
log(f"saved layout {name!r}")
kwin_follow() kwin_follow()
elif cmd == "use": elif cmd == "use":
layout = load_layout() layout = load_layout()
use_named(layout, argv[2]) use_named(layout, argv[2])
use_hidden(layout, argv[2])
save_layout(layout) save_layout(layout)
log(f"using layout {argv[2]!r}") log(f"using layout {argv[2]!r}")
try: try:
apply() apply()
except RuntimeError as e: except RuntimeError as e:
log(f"not arranged now: {e}") log(f"not arranged now: {e}")
try: # the screens stay put, but the profile's hidden ones still hide
send_hidden(screens_socket(), layout)
except RuntimeError:
pass
else: else:
kwin_follow() kwin_follow()
open_apps(argv[2]) # floating windows go relative to the screens, wherever they are
else: else:
changes = apply_scales() changes = apply_scales()
log("kwin: " + (" ".join(changes) if changes else "unchanged")) log("kwin: " + (" ".join(changes) if changes else "unchanged"))
@@ -1,187 +0,0 @@
{
"action_manifest_version": 0,
"alias_info": {},
"app_key": "openvr.component.vrcompositor",
"bindings": {
"/actions/system": {
"chords": [
{
"inputs": [
[
"/user/hand/left/input/system",
"held"
],
[
"/user/hand/left/input/trigger",
"click"
]
],
"output": "/actions/system/in/TakeScreenshot"
},
{
"inputs": [
[
"/user/hand/right/input/system",
"held"
],
[
"/user/hand/right/input/trigger",
"click"
]
],
"output": "/actions/system/in/TakeScreenshot"
},
{
"inputs": [
[
"/user/hand/left/input/system",
"held"
],
[
"/user/hand/right/input/trigger",
"click"
]
],
"output": "/actions/system/in/TakeScreenshot"
},
{
"inputs": [
[
"/user/hand/right/input/system",
"held"
],
[
"/user/hand/left/input/trigger",
"click"
]
],
"output": "/actions/system/in/TakeScreenshot"
}
],
"sources": [
{
"inputs": {},
"mode": "button",
"parameters": {
"haptic_amplitude": "0"
},
"path": "/user/hand/left/input/trigger"
},
{
"inputs": {},
"mode": "button",
"parameters": {
"haptic_amplitude": "0"
},
"path": "/user/hand/right/input/trigger"
},
{
"inputs": {
"click": {
"output": "/actions/system/in/ToggleDashboard"
},
"double": {
"output": "/actions/system/in/ToggleRoomView"
},
"held": {
"output": "/actions/system/in/SystemButtonChord"
}
},
"mode": "button",
"path": "/user/hand/left/input/system"
},
{
"inputs": {
"click": {
"output": "/actions/system/in/ToggleDashboard"
},
"double": {
"output": "/actions/system/in/ToggleRoomView"
},
"held": {
"output": "/actions/system/in/SystemButtonChord"
}
},
"mode": "button",
"path": "/user/hand/right/input/system"
}
]
},
"/actions/quickrecenter": {
"sources": [
{
"inputs": {
"long": {
"output": "/actions/quickrecenter/in/Recenter"
}
},
"parameters": {
"long_press_delay": 1.0,
"long_press_expiry": 4.0
},
"mode": "button",
"path": "/user/hand/left/input/system"
},
{
"inputs": {
"long": {
"output": "/actions/quickrecenter/in/Recenter"
}
},
"parameters": {
"long_press_delay": 1.0,
"long_press_expiry": 4.0
},
"mode": "button",
"path": "/user/hand/right/input/system"
}
]
},
"/actions/roomsetup": {
"sources": [
{
"inputs": {
"position": {
"output": "/actions/roomsetup/in/FloorFineAdjust"
}
},
"mode": "joystick",
"path": "/user/hand/left/input/thumbstick"
},
{
"inputs": {
"position": {
"output": "/actions/roomsetup/in/FloorFineAdjust"
}
},
"mode": "joystick",
"path": "/user/hand/right/input/thumbstick"
},
{
"inputs": {
"click": {
"output": "/actions/roomsetup/in/Accept"
}
},
"mode": "button",
"path": "/user/hand/right/input/a"
},
{
"inputs": {
"click": {
"output": "/actions/roomsetup/in/Back"
}
},
"mode": "button",
"path": "/user/hand/right/input/b"
}
]
}
},
"category": "steamvr_input",
"controller_type": "frame_controller",
"description": "Gaze first (Frametop, docs/gaze-first.md): the stock compositor binding with only the Steam button (dashboard, room view, recenter, screenshot chord) and room setup left. The relay reads the rest from vrserver's web socket.",
"name": "Frametop gaze first: Frame controller, Steam button only",
"options": {},
"simulated_actions": []
}
+10 -18
View File
@@ -17,12 +17,11 @@
// btn <name> <0|1> name: trigger, b, x, system, joystick, a (a = claim the laser, no click) // btn <name> <0|1> name: trigger, b, x, system, joystick, a (a = claim the laser, no click)
// scroll <x> <y> joystick deflection -1..1 // scroll <x> <y> joystick deflection -1..1
// show | hide connect (take the hand role) or disconnect (give it back) // show | hide connect (take the hand role) or disconnect (give it back)
// role right|left|stylus|treadmill which role to hint while connected, from now on (the // role right|left|stylus which role to hint while connected, from now on (the helper
// helper sends POINTER_ROLE). A Frame controller in your hand // sends POINTER_ROLE). A Frame controller in your hand takes
// takes its hand's role back (it counts as used while held), so a // its hand's role back (it counts as used while held), so a
// controller used beside the pointer needs the pointer on the // controller used beside the pointer needs the pointer on the
// other hand, or on no hand at all. Treadmill is no hand: its // other hand, or on no hand at all
// bindings are under /user/treadmill (docs/gaze-first.md, test 1)
// //
// The device starts disconnected, so it never holds a hand role at boot (holding // The device starts disconnected, so it never holds a hand role at boot (holding
// the right hand while SteamVR started left the Steam UI stuck loading). It // the right hand while SteamVR started left the Steam UI stuck loading). It
@@ -31,8 +30,7 @@
// SteamVR keeps a hand role reserved for a disconnected device that still hints // SteamVR keeps a hand role reserved for a disconnected device that still hints
// that hand, so the real controller would never get it back otherwise. // that hand, so the real controller would never get it back otherwise.
// //
// Settings (steamvr.vrsettings section "driver_ft_pointer"): role (int, 2 = right hand, 4 = treadmill, // Settings (steamvr.vrsettings section "driver_ft_pointer"): role (int, 2 = right hand, 5 = stylus).
// 5 = stylus).
#include <openvr_driver.h> #include <openvr_driver.h>
#include <atomic> #include <atomic>
@@ -117,9 +115,7 @@ public:
props->SetStringProperty(c, Prop_ControllerType_String, "ft_pointer"); props->SetStringProperty(c, Prop_ControllerType_String, "ft_pointer");
props->SetStringProperty(c, Prop_InputProfilePath_String, "{ft_pointer}/input/ft_pointer_profile.json"); props->SetStringProperty(c, Prop_InputProfilePath_String, "{ft_pointer}/input/ft_pointer_profile.json");
props->SetStringProperty(c, Prop_RenderModelName_String, "{ft_pointer}/rendermodels/ft_pointer_invisible"); props->SetStringProperty(c, Prop_RenderModelName_String, "{ft_pointer}/rendermodels/ft_pointer_invisible");
// A hand only while shown (see the top); a role that's no hand from the start, since props->SetInt32Property(c, Prop_ControllerRoleHint_Int32, TrackedControllerRole_OptOut); // until "show"
// SteamVR gives a treadmill its path only when the device is added (docs/gaze-first.md).
props->SetInt32Property(c, Prop_ControllerRoleHint_Int32, HintFor(false));
props->SetInt32Property(c, Prop_DeviceClass_Int32, TrackedDeviceClass_Controller); props->SetInt32Property(c, Prop_DeviceClass_Int32, TrackedDeviceClass_Controller);
props->SetBoolProperty(c, Prop_NeverTracked_Bool, false); props->SetBoolProperty(c, Prop_NeverTracked_Bool, false);
@@ -215,11 +211,12 @@ public:
} }
if (snapshot.role > 0 && snapshot.role != role_) { if (snapshot.role > 0 && snapshot.role != role_) {
role_ = snapshot.role; role_ = snapshot.role;
VRProperties()->SetInt32Property(container_, Prop_ControllerRoleHint_Int32, HintFor(hinted_)); if (hinted_) VRProperties()->SetInt32Property(container_, Prop_ControllerRoleHint_Int32, role_);
} }
if (snapshot.visible != hinted_) { if (snapshot.visible != hinted_) {
// Claim the hand before connecting; give it up when disconnecting. // Claim the hand before connecting; give it up when disconnecting.
VRProperties()->SetInt32Property(container_, Prop_ControllerRoleHint_Int32, HintFor(snapshot.visible)); VRProperties()->SetInt32Property(container_, Prop_ControllerRoleHint_Int32,
snapshot.visible ? role_ : int32_t(TrackedControllerRole_OptOut));
hinted_ = snapshot.visible; hinted_ = snapshot.visible;
} }
const bool ok = hmd.bPoseIsValid && snapshot.visible; const bool ok = hmd.bPoseIsValid && snapshot.visible;
@@ -240,11 +237,7 @@ private:
uint32_t objectId_ = k_unTrackedDeviceIndexInvalid; uint32_t objectId_ = k_unTrackedDeviceIndexInvalid;
PropertyContainerHandle_t container_ = k_ulInvalidPropertyContainer; PropertyContainerHandle_t container_ = k_ulInvalidPropertyContainer;
int32_t role_ = TrackedControllerRole_RightHand; int32_t role_ = TrackedControllerRole_RightHand;
bool hinted_ = false; // whether we're shown (a hand role is hinted only then) bool hinted_ = false; // whether the role hint currently claims role_
int32_t HintFor(bool shown) const {
const bool hand = role_ == TrackedControllerRole_LeftHand || role_ == TrackedControllerRole_RightHand;
return shown || !hand ? role_ : int32_t(TrackedControllerRole_OptOut);
}
DriverPose_t pose_{}; DriverPose_t pose_{};
VRInputComponentHandle_t buttons_[kButtons] = {}; VRInputComponentHandle_t buttons_[kButtons] = {};
VRInputComponentHandle_t scrollX_ = 0, scrollY_ = 0; VRInputComponentHandle_t scrollX_ = 0, scrollY_ = 0;
@@ -350,7 +343,6 @@ private:
state_.role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand state_.role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand
: !std::strcmp(name, "right") ? TrackedControllerRole_RightHand : !std::strcmp(name, "right") ? TrackedControllerRole_RightHand
: !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus : !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
: !std::strcmp(name, "treadmill") ? TrackedControllerRole_Treadmill
: state_.role; : state_.role;
} else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) { } else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
for (int i = 0; i < kButtons; ++i) for (int i = 0; i < kButtons; ++i)
@@ -82,42 +82,6 @@
"output": "/actions/lasermouse/in/back" "output": "/actions/lasermouse/in/back"
} }
} }
},
{
"path": "/user/treadmill/input/trigger",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse/in/leftclick"
}
}
},
{
"path": "/user/treadmill/input/b",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse/in/rightclick"
}
}
},
{
"path": "/user/treadmill/input/x",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse/in/middleclick"
}
}
},
{
"path": "/user/treadmill/input/joystick",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse/in/back"
}
}
} }
], ],
"poses": [ "poses": [
@@ -128,10 +92,6 @@
{ {
"output": "/actions/lasermouse/in/Pointer", "output": "/actions/lasermouse/in/Pointer",
"path": "/user/hand/right/pose/raw" "path": "/user/hand/right/pose/raw"
},
{
"output": "/actions/lasermouse/in/Pointer",
"path": "/user/treadmill/pose/raw"
} }
] ]
}, },
@@ -154,15 +114,6 @@
"output": "/actions/scroll_discrete/in/scroll" "output": "/actions/scroll_discrete/in/scroll"
} }
} }
},
{
"path": "/user/treadmill/input/joystick",
"mode": "scroll",
"inputs": {
"scroll": {
"output": "/actions/scroll_discrete/in/scroll"
}
}
} }
] ]
}, },
@@ -185,15 +136,6 @@
"output": "/actions/scroll_smooth/in/scroll" "output": "/actions/scroll_smooth/in/scroll"
} }
} }
},
{
"path": "/user/treadmill/input/joystick",
"mode": "scroll",
"inputs": {
"scroll": {
"output": "/actions/scroll_smooth/in/scroll"
}
}
} }
] ]
}, },
@@ -216,15 +158,6 @@
"output": "/actions/system/in/ToggleDashboard" "output": "/actions/system/in/ToggleDashboard"
} }
} }
},
{
"path": "/user/treadmill/input/system",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/system/in/ToggleDashboard"
}
}
} }
] ]
}, },
@@ -265,24 +198,6 @@
"output": "/actions/lasermouse_secondary/in/switchlaserhand" "output": "/actions/lasermouse_secondary/in/switchlaserhand"
} }
} }
},
{
"path": "/user/treadmill/input/trigger",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse_secondary/in/switchlaserhand"
}
}
},
{
"path": "/user/treadmill/input/a",
"mode": "button",
"inputs": {
"click": {
"output": "/actions/lasermouse_secondary/in/switchlaserhand"
}
}
} }
] ]
} }
+1 -1
View File
@@ -5,5 +5,5 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null "$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer/helper 'set -e; mkdir -p build exec "$root/scripts/frame.sh" -C pointer/helper 'set -e; mkdir -p build
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \ g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \
-o build/ft-pointer ft-pointer.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread -o build/ft-pointer ft-pointer.cpp pointer_*.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
echo "built build/ft-pointer"' echo "built build/ft-pointer"'
File diff suppressed because it is too large. Load diff
+478
View File
@@ -0,0 +1,478 @@
// ft-pointer's shared code: helpers, the Frame, and the Pointer struct, whose member functions are
// in ft-pointer.cpp and pointer_*.cpp.
// "The top" in comments here is the header comment of ft-pointer.cpp.
#pragma once
#include <openvr.h>
#include "pointer_config.h"
#include "vrbuttons.h"
#include "vrmath.h"
extern "C" {
#include "../../hands/include/fh_gestures.h"
}
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <deque>
#include <fstream>
#include <map>
#include <mutex>
#include <string>
#include <thread>
#include <tuple>
#include <vector>
#include <climits>
#include <fcntl.h>
#include <fnmatch.h>
#include <sys/mman.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/un.h>
#include <unistd.h>
using namespace md;
// Hand gestures from ft-hands (see "Hands" at the top): /run/user/UID/frametop-hands/gestures,
// mapped read-only. Version 1 files have pinches only; their grips read as zero.
class HandGestures {
public:
~HandGestures() { Close(); }
// A consistent copy of the file (its sequence lock), if it's there. Opens it, and
// checks it's still the same file, at most once a second.
bool Read(fh_gestures_t &out) {
const auto now = std::chrono::steady_clock::now();
if (now - checked_ > std::chrono::seconds(1)) {
checked_ = now;
const std::string path = "/run/user/" + std::to_string(getuid()) + "/frametop-hands/gestures";
struct stat st;
if (stat(path.c_str(), &st) != 0 || size_t(st.st_size) != len_ || st.st_ino != ino_) {
Close();
const int fd = open(path.c_str(), O_RDONLY | O_CLOEXEC | O_NOFOLLOW);
if (fd >= 0 && fstat(fd, &st) == 0 && size_t(st.st_size) >= offsetof(fh_gestures_t, grip)) {
void *m = mmap(nullptr, size_t(st.st_size), PROT_READ, MAP_SHARED, fd, 0);
if (m != MAP_FAILED) map_ = m, len_ = size_t(st.st_size), ino_ = st.st_ino, ++opens;
}
if (fd >= 0) close(fd);
}
}
if (!map_) return false;
const auto *g = static_cast<const fh_gestures_t *>(map_);
if (std::memcmp(g->magic, FH_GESTURES_MAGIC, 8) != 0) return false;
const size_t n = std::min(len_, sizeof out);
for (int tries = 0; tries < 3; ++tries) {
const uint64_t seq = __atomic_load_n(&g->seq, __ATOMIC_ACQUIRE);
if (seq & 1) continue;
std::memset(&out, 0, sizeof out);
std::memcpy(&out, map_, n);
__atomic_thread_fence(__ATOMIC_ACQUIRE);
if (__atomic_load_n(&g->seq, __ATOMIC_RELAXED) != seq) continue;
if (out.version < 2) std::memset(out.grip, 0, sizeof out.grip);
return true;
}
return false;
}
int opens = 0; // a new file: the counters start over
private:
void Close() {
if (map_) munmap(map_, len_);
map_ = nullptr, len_ = 0, ino_ = 0;
}
void *map_ = nullptr;
size_t len_ = 0;
ino_t ino_ = 0;
std::chrono::steady_clock::time_point checked_{};
};
// The HMD's recent poses (standing universe), to turn the gestures' head-frame points into
// the room as the head was when the cameras took them.
class PoseHistory {
public:
void Add(std::chrono::steady_clock::time_point t, const vr::HmdMatrix34_t &m) {
poses_.push_back({t, m});
while (poses_.size() > 128) poses_.pop_front(); // about a second
}
// The pose at CLOCK_MONOTONIC time t_ns (steady_clock's), or the nearest kept.
bool At(uint64_t t_ns, vr::HmdMatrix34_t &out) const {
if (poses_.empty()) return false;
const std::chrono::steady_clock::time_point t{std::chrono::nanoseconds(t_ns)};
const auto *best = &poses_.front();
for (const auto &p : poses_)
if (std::chrono::abs(p.first - t) < std::chrono::abs(best->first - t)) best = &p;
out = best->second;
return true;
}
private:
std::deque<std::pair<std::chrono::steady_clock::time_point, vr::HmdMatrix34_t>> poses_;
};
// One of ft-screens' panels showing a desktop: a screen (frametop.screen.N), a floating
// window (frametop.float.N), or a floating window's popup (frametop.float.N.sub.K), not a
// control of theirs.
inline bool FramePanel(const std::string &key) {
for (const char *prefix : {"frametop.screen.", "frametop.float."}) {
if (key.rfind(prefix, 0) != 0) continue;
const std::string rest = key.substr(std::strlen(prefix));
const size_t dot = rest.find('.');
return dot == std::string::npos || rest.compare(dot, 5, ".sub.") == 0;
}
return false;
}
inline void SendTo(int fd, const char *name, const std::string &msg) {
sockaddr_un addr{};
addr.sun_family = AF_UNIX;
std::memcpy(addr.sun_path + 1, name, std::strlen(name));
const socklen_t len = offsetof(sockaddr_un, sun_path) + 1 + std::strlen(name);
sendto(fd, msg.data(), msg.size(), 0, reinterpret_cast<sockaddr *>(&addr), len);
}
// JSON string literal (names come from other apps).
inline std::string JsonQuote(const std::string &s) {
std::string out = "\"";
for (const unsigned char c : s) {
if (c == '"' || c == '\\') out += '\\', out += char(c);
else if (c < 0x20) {
char esc[8];
std::snprintf(esc, sizeof esc, "\\u%04x", c);
out += esc;
} else out += char(c);
}
return out + "\"";
}
// Overlay keys, refreshed in the background from `vrcmd --overlays` (OpenVR has no
// public call to enumerate other apps' overlays). Hidden ones are listed too: the
// window controls under a floating panel only appear while something hovers the
// panel, and the cursor has to find them the moment they do, not a second later.
// Paused while the pointer is off: each vrcmd run connects to SteamVR as a new app, and a new
// app every second kept SteamVR (and the headset's displays) from going to standby.
// "overlays" requests (Frametop Input Settings' Ignored panels page) refresh the list even
// while paused, and are answered from this thread once it's fresh.
class OverlayList {
public:
void Start() {
out_ = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
thread_ = std::thread([this] {
while (running_) {
if (!paused_ || requested_) Refresh();
// Wait a second, or less when the pointer wakes (refresh right away then).
for (int i = 0; i < 10 && running_; ++i) {
const bool wasPaused = paused_;
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if ((wasPaused && !paused_) || requested_) break;
}
}
});
}
void SetPaused(bool paused) { paused_ = paused; }
void Stop() {
running_ = false;
if (thread_.joinable()) thread_.join();
}
std::vector<std::string> Keys() {
std::lock_guard<std::mutex> guard(lock_);
std::vector<std::string> keys;
for (const auto &e : entries_) keys.push_back(e.key);
return keys;
}
// Answer `to` with {"t":"overlays","list":[{"key","name","visible"}...]} after the next refresh.
void Request(const sockaddr_un &to, socklen_t len) {
if (len <= offsetof(sockaddr_un, sun_path)) return;
std::lock_guard<std::mutex> guard(lock_);
if (waiting_.size() < 8) waiting_.push_back({to, len});
requested_ = true;
}
private:
struct Entry {
std::string key, name;
bool visible;
};
void Refresh() {
requested_ = false;
FILE *p = popen("LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null", "r");
if (!p) return;
std::vector<Entry> entries;
char line[1024];
while (std::fgets(line, sizeof line, p)) {
// 'key' -- 'name', WxH visible VROverlayType_...
if (line[0] != '\'') continue;
const char *end = std::strchr(line + 1, '\'');
if (!end) continue;
const std::string key(line + 1, size_t(end - (line + 1)));
const std::string rest(end);
if (rest.find("Thumbnail") != std::string::npos || rest.find("Subview") != std::string::npos) continue;
if (key.rfind("system.pointer", 0) == 0 || key.rfind("system.cursor", 0) == 0 ||
key.rfind("frametop.pointer", 0) == 0 || key.rfind("frametop.guide", 0) == 0 ||
key == "frametop.gazepanel" || // the gaze calibration panel, fixed to the headset
key == "frametop.catcher" || // ft-screens' release catcher: only on a laser mid-drag
key == "system.HeadsetView" || key == "system.toast")
continue;
// The name can hold quotes; it ends at the last "', " (the size and state follow).
const auto nameAt = rest.find("-- '"), nameEnd = rest.rfind("', ");
const std::string name =
nameAt != std::string::npos && nameEnd > nameAt + 3 ? rest.substr(nameAt + 4, nameEnd - nameAt - 4) : key;
entries.push_back({key, name, rest.find(" not_visible ") == std::string::npos});
}
pclose(p);
std::vector<std::pair<sockaddr_un, socklen_t>> waiting;
{
std::lock_guard<std::mutex> guard(lock_);
entries_ = std::move(entries);
waiting.swap(waiting_);
}
if (waiting.empty()) return;
std::string msg = "{\"t\":\"overlays\",\"list\":[";
for (size_t i = 0; i < entries_.size(); ++i)
msg += std::string(i ? "," : "") + "{\"key\":" + JsonQuote(entries_[i].key) + ",\"name\":" +
JsonQuote(entries_[i].name) + ",\"visible\":" + (entries_[i].visible ? "true" : "false") + "}";
msg += "]}";
for (const auto &[to, len] : waiting)
sendto(out_, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<const sockaddr *>(&to), len);
}
std::thread thread_;
int out_ = -1;
std::atomic<bool> paused_{false};
std::atomic<bool> running_{true};
std::atomic<bool> requested_{false};
std::mutex lock_;
std::vector<Entry> entries_; // written only by the thread; the lock guards readers
std::vector<std::pair<sockaddr_un, socklen_t>> waiting_;
};
using Clock = std::chrono::steady_clock;
// One frame's poses and time: ReadFrame reads them after the relay's commands, and the sections
// after it share them. handOk: PinchesAndGrips read the hand gestures (for HandsStopped).
struct Frame {
vr::TrackedDevicePose_t all[vr::k_unMaxTrackedDeviceCount];
vr::TrackedDevicePose_t hmdRaw;
Clock::time_point tnow;
Vec3 eye;
bool handOk = false;
const vr::TrackedDevicePose_t &Hmd() const { return all[0]; }
};
// Everything the pointer keeps: its settings, what Init sets up, and the state the main loop
// carries from one frame to the next.
struct Pointer {
PointerConfig cfg;
// Head follow (see the top). followConf is POINTER_FOLLOW as last read: a reload only
// overrides a "follow" command when the setting itself changed.
bool follow = false, followConf = false, followReset = true;
// Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
bool gazeOn = false, gazeConf = false;
// Set in Init, after VR_Init.
vr::IVRSystem *sys = nullptr;
vr::IVROverlay *overlay = nullptr;
// The dot, the marker, and laser mode's overlay, created in Init.
vr::VROverlayHandle_t cursor = vr::k_ulOverlayHandleInvalid;
vr::VROverlayHandle_t marker = vr::k_ulOverlayHandleInvalid;
vr::VROverlayHandle_t laserMode = vr::k_ulOverlayHandleInvalid;
bool laserModeShown = false;
int in = -1, out = -1; // @ft_pointer_helper, and the socket we send from (opened in Init)
ControllerButtons controllerButtons; // Frame controller buttons (vrbuttons.h)
OverlayList overlays;
std::map<std::string, vr::VROverlayHandle_t> handles;
std::map<std::string, bool> sceneGraph; // no texture: plane test instead of ComputeOverlayIntersection
std::map<std::string, bool> visible; // refreshed every 50 ms
Clock::time_point lastVisible{}; // set in Init
// The plane of the last panel the cursor was on, and the last point on it (panel edges).
Vec3 edgePoint, edgeNormal, edgeLast;
std::string edgeKey;
bool active = false, recenter = false, anchored = false;
Clock::time_point lastMouse{}, claimAt{}, claimRelease{}, wokeAt{}, noWakeUntil{};
bool claimPending = false, claimHeld = false;
// Last used wins: since when each controller has been moving (zero: it isn't).
Clock::time_point movingSince[vr::k_unMaxTrackedDeviceCount] = {};
// Tilt mode (see top of file).
bool leftHeld = false, tilting = false, tiltStart = false, swallowedRight = false;
// The mouse's buttons in gaze mode (see the top): leftDown, rightDown as the relay last said;
// aimRight, the held-back press is the right button's; chordDrag, a drag the right button
// began during the left's held-back press (the first release drops it, the other's is nothing).
bool leftDown = false, rightDown = false, aimRight = false, chordDrag = false;
bool ignoreLeftUp = false;
// The keyboard clicks' keys as the relay last said, and a tilt gaze_right holds during a
// keyboard drag (see the top): the head turns the panel, from where it was (keyTiltYaw, keyTiltPitch).
bool keyLeftDown = false, keyRightDown = false, keyTilting = false;
double keyTiltYaw = 0, keyTiltPitch = 0;
double tiltYaw = 0, tiltPitch = 0;
double dragDistance = 0, lastDistance = 1.5; // drag lock: distance from the anchor at the press
bool onVrSettings = false; // the cursor is on the SteamVR Settings page (kept while dragging)
bool catcherHidesHit = false; // the laser-catching dot hides SteamVR's hit dot (Settings page)
Clock::time_point dropHoldUntil{}; // after a left release: keep the drag pose this long
bool debug = false;
std::string lastHit;
// The ft-screens panel the left button was pressed on, and where it was then (see the top).
std::string pressKey;
vr::HmdMatrix34_t pressPose{};
Clock::time_point lastDebug{}; // set in Init
vr::VROverlayHandle_t systemPointer = vr::k_ulOverlayHandleInvalid; // found in Init
Vec3 pivot, tiltOrigin, lastPoint, lastOrigin, lastAim{0, 0, -1};
Basis tiltBasis{};
bool headsetOff = false; // nobody is wearing the headset (see the main loop)
Vec3 anchor;
double yaw = 0, pitch = 0;
Vec3 followRef{0, 0, -1}; // head follow's reference direction (see the top)
Clock::time_point followAt{}; // its last update, for the easing (set in Init)
bool following = false; // past the leash: easing toward the head
double followLag = 0; // radians the reference trails the head
std::chrono::steady_clock::time_point leashOutSince{}; // head past the leash since (delay)
// Gaze mode (see the top).
struct Gaze {
double hy = 0, hp = 0, rhy = 0, rhp = 0; // corrected, and raw
Clock::time_point at{};
} gz;
bool gazeOwns = true; // the pointer follows the gaze; false: the mouse has it
bool nudging = false; // the mouse took it from the gaze: the next click may be a lesson
double nudgeRawHy = 0, nudgeRawHp = 0, nudgeMoved = 0;
vr::HmdMatrix34_t nudgeHead{}, lastHead{};
bool haveHead = false, havePoint = false;
Clock::time_point nudgeAt{}, retakeSince{};
// A held-back press (see the top): aimHeld while the button is down; then the click
// (clickPress at the end of the next frame, clickRelease 40 ms later). gazeBack: give
// the gaze the pointer again when the press or click is over.
vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
bool aimHeld = false, clickPress = false, clickRelease = false, gazeBack = false;
bool aimHand = false; // the held-back press is a hold's (below): it never turns into a real press
Clock::time_point aimSince{}, clickReleaseAt{};
// Holds (see "Gaze precision" and "Hands" at the top): a pinch or grip, or a gaze
// precision or gaze drag button, held now. What steers the pointer meanwhile (a hand, or
// the mouse through its own moves), from where it pointed when the hold began (for a
// hand: seen from the eye then, in the room), and the pointer then.
HandGestures handFile;
PoseHistory poses;
enum class Src { None, Hand, Mouse, Head };
struct Hold {
Src src = Src::None;
int side = -1; // a hand's side (0 left, 1 right)
bool grip = false; // pressed at once and dragging (a grip, gaze drag), not a click on release
bool engaged = false; // past the dead zone
bool pressed = false; // a real press went out (a grip or gaze drag, or a pinch without gaze mode)
bool promote = false; // held still for POINTER_GAZE_HOLD, it becomes a real press (keyboard clicks)
bool right = false; // the right button (gaze_right)
bool keyDrag = false; // gaze_right pressed while gaze_left aimed: a left press, either key ends it
double pressYaw = 0, pressPitch = 0; // the pointer at the press (a quick tap clicks there)
Vec3 origin;
double refYaw = 0, refPitch = 0, startYaw = 0, startPitch = 0, lastYaw = 0, lastPitch = 0;
} hold;
// "precision|gazedrag <source> 1|0" from the relay, done in the frame (see Holds).
struct DevicePress {
std::string source;
bool drag, down;
};
std::vector<DevicePress> devicePresses;
// "gazekey left|right 1|0" from the relay (keyboard clicks), done in the frame.
struct KeyPress {
bool right, down;
};
std::vector<KeyPress> keyPresses;
uint32_t seenBegins[2][2] = {}, seenEnds[2][2] = {}; // [pinch, grip][side], as last read
bool handBaseline = false;
int handOpens = 0;
uint64_t handSeq = 0, handPublished = 0;
Clock::time_point handUsed{}; // a gesture began then (keeps the pointer, like gaze mode)
Clock::time_point lastTyping{}; // the relay's last "typing": a key on a keyboard
// Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
bool inGame = false, gazeAwake = false;
Clock::time_point inGameAt{}, gazeAwakeAt{};
Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
// The left button, as sent to the driver; pressRight: the next press is the right button
// instead (gaze_right), heldButton: the one pressed.
bool pressRight = false;
std::string heldButton = "trigger";
bool confirmLesson = false; // a keyboard click's quick tap: a lesson with no correction (see the top)
// The gaze calibration panel is up until then ("calpanel 1"; see the top); calOpened: it just
// came up, so a press in progress ends without a click.
Clock::time_point calPanelUntil{};
bool calOpened = false;
Clock::time_point lastSlow{}; // set in Init
void ApplyConfig();
void Init();
void Wake(Clock::time_point t);
void PressLeft();
void ReleaseLeft();
bool CanAim();
void AimStart(bool right);
void ChordDrop();
void LeftButton(bool down);
bool RightButton(bool down);
bool MouseMoveHeld();
void SendPose(Vec3 originStanding, const Basis &b);
std::pair<bool, Vec3> HeadPos();
void Borrow();
void GiveBack();
std::string FindPanel(const char *key, Panel &p, Vec3 &eye);
void GrabProbe(const char *key);
std::string Place(const char *key, Vec3 target, const Basis &bt, double grabBelow);
// Nearest's answer: the nearest overlay along a ray.
struct Hit {
double along = 1e9;
std::string key;
bool scene = false;
Vec3 point, normal;
};
// Where the cursor is this frame (FindCursor), for DrawDot and SendRay.
struct Spot {
bool dragging = false;
Vec3 dir, point;
double best = 1e9, distance = 0;
bool onEdge = false, occluded = false, onScene = false, onPanel = false;
};
// The main loop's sections, in the order main calls them, and what they share.
void HeadsetOff();
void GazeAwake();
void RelayCommands();
void Command(const char *buf, const sockaddr_un &sender, socklen_t senderLen);
void ReadFrame(Frame &frame);
void ClaimPulse(const Frame &frame);
void PauseOverlayList();
void LaserMode();
void HandRole(const Frame &frame);
void LastUsedWins(const Frame &frame);
void Recenter(const Frame &frame);
void GazeMode(const Frame &frame);
void Hands(const Frame &frame);
bool HandAngles(const float p[3], uint64_t t_ns, const Vec3 &from, double &hy, double &hp);
void EndHold(bool lost);
void StartHold(bool press, Clock::time_point tnow);
void Steer(double hy, double hp, double deadzone, double gain, Clock::time_point tnow);
void BeginHold(int side, bool grip, const fh_pinch_t &g, Clock::time_point tnow);
void GazePrecisionButtons(const Frame &frame);
bool HeadAngles(const vr::TrackedDevicePose_t &hmd, double &hy, double &hp);
void KeyboardClicks(const Frame &frame);
void CalPanelOpened();
void HeadSteer(const Frame &frame);
void PinchesAndGrips(Frame &frame);
void HandsStopped(const Frame &frame);
void MarkHeld(const Frame &frame);
void HeadFollow(const Frame &frame);
void SlowWork();
void Cursor(const Frame &frame);
void Tilt(const Frame &frame);
Hit Nearest(Vec3 from, Vec3 d);
Spot FindCursor(const Frame &frame);
void DrawDot(const Frame &frame, const Spot &spot);
void SendRay(const Frame &frame, const Spot &spot);
void HeldBackPress(const Frame &frame);
void KeyboardClickHold(const Frame &frame);
void Click(const Frame &frame);
void PollControllerButtons();
bool SteamVRQuit();
};
+114
View File
@@ -0,0 +1,114 @@
// ft-pointer's settings, from ~/.config/frametop.conf. "The top" in comments here is the
// header comment of ft-pointer.cpp, which describes every setting.
#pragma once
#include <algorithm>
#include <cstdlib>
#include <fstream>
#include <map>
#include <string>
#include <vector>
inline std::map<std::string, std::string> ReadConfig() {
std::map<std::string, std::string> conf;
const char *home = std::getenv("HOME");
std::ifstream in(std::string(home ? home : "") + "/.config/frametop.conf");
std::string line;
while (std::getline(in, line)) {
line = line.substr(0, line.find('#'));
const auto eq = line.find('=');
if (eq == std::string::npos) continue;
auto trim = [](std::string s) {
s.erase(0, s.find_first_not_of(" \t"));
s.erase(s.find_last_not_of(" \t") + 1);
return s;
};
conf[trim(line.substr(0, eq))] = trim(line.substr(eq + 1));
}
return conf;
}
inline double ConfDouble(const std::map<std::string, std::string> &c, const char *key, double fallback) {
auto it = c.find(key);
return it == c.end() ? fallback : std::atof(it->second.c_str());
}
// POINTER_IGNORE: overlay keys the pointer passes through, as if they weren't there
// (display-only panels such as a performance overlay). Comma-separated shell patterns
// (fnmatch, no escapes), so "vendor.app*" covers an app's overlays.
inline std::vector<std::string> ParseIgnore(const std::string &list) {
std::vector<std::string> out;
size_t at = 0;
while (at <= list.size()) {
const size_t comma = std::min(list.find(',', at), list.size());
std::string item = list.substr(at, comma - at);
item.erase(0, item.find_first_not_of(" \t"));
item.erase(item.find_last_not_of(" \t") + 1);
if (!item.empty()) out.push_back(item);
at = comma + 1;
}
return out;
}
struct PointerConfig {
double freeDistance = 1.5, cursorDeg = 0.4, originFraction = 0.95, originMargin = 0.15, sceneRadius = 0.5,
edgeReach = 0.3, grabOffset = 0.075,
slideSpeed = 0.5, leashDeg = 10, leashReturn = 0.2, leashDelay = 0.2, followReach = 70;
// POINTER_FOLLOW and POINTER_GAZE as read. They turn head follow and gaze mode on or off
// only when they change, so a "follow" or "gaze" command wins until then (ApplyConfig in
// ft-pointer.cpp).
bool follow = false, gaze = false;
double gazeRetake = 5, gazeNudgeMax = 55, gazeHold = 0.5, gazeShow = 1;
bool gazeDotAlways = true; // POINTER_GAZE_DOT (see the top)
double headDeadzone = 0.5, keyTap = 0.25; // POINTER_HEAD_DEADZONE, POINTER_KEY_TAP (keyboard clicks, see the top)
// Hands (see the top): POINTER_HANDS, POINTER_PINCH_GAIN, POINTER_PINCH_DEADZONE, POINTER_GRIP_GAIN.
bool handsOn = false;
double pinchGain = 0.5, pinchDeadzone = 1.5, gripGain = 1.0, handBelow = 0.35, typingHold = 1.0;
// Gaze precision (see the top): POINTER_GAZE_MOUSE (precision: the mouse's left button
// holds back its press in gaze mode; direct: it clicks at once), POINTER_ROLE.
bool gazeMousePrecision = true;
bool gazeMouseHeld = true; // POINTER_GAZE_MOUSE_MOVE (see the top)
std::string role = "right";
double pickupScale = 1; // POINTER_CONTROLLER_PICKUP: scales the controller-moved limits
std::vector<std::string> ignore; // POINTER_IGNORE (see ParseIgnore)
void Load(const std::map<std::string, std::string> &conf) {
freeDistance = std::clamp(ConfDouble(conf, "POINTER_DISTANCE", 1.5), 0.3, 10.0);
cursorDeg = std::clamp(ConfDouble(conf, "POINTER_CURSOR_DEG", 0.4), 0.05, 5.0);
originFraction = std::clamp(ConfDouble(conf, "POINTER_ORIGIN_FRACTION", 0.95), 0.0, 0.98);
originMargin = std::clamp(ConfDouble(conf, "POINTER_ORIGIN_MARGIN", 0.15), 0.0, 1.0);
sceneRadius = std::clamp(ConfDouble(conf, "POINTER_SCENE_RADIUS", 0.5), 0.05, 2.0);
edgeReach = std::clamp(ConfDouble(conf, "POINTER_EDGE_REACH", 0.3), 0.0, 2.0);
grabOffset = std::clamp(ConfDouble(conf, "LAYOUT_GRAB_OFFSET", 0.075), 0.0, 1.0);
slideSpeed = std::clamp(ConfDouble(conf, "LAYOUT_SLIDE_SPEED", 0.5), 0.02, 2.0);
leashDeg = std::clamp(ConfDouble(conf, "POINTER_LEASH_DEG", 10), 0.0, 90.0);
leashReturn = std::clamp(ConfDouble(conf, "POINTER_LEASH_RETURN", 0.2), 0.0, 5.0);
leashDelay = std::clamp(ConfDouble(conf, "POINTER_LEASH_DELAY", 0.2), 0.0, 5.0);
followReach = std::clamp(ConfDouble(conf, "POINTER_FOLLOW_REACH", 70), 10.0, 89.0);
follow = ConfDouble(conf, "POINTER_FOLLOW", 0) != 0;
gazeRetake = std::clamp(ConfDouble(conf, "POINTER_GAZE_RETAKE", 5), 1.0, 45.0);
gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 55), 1.0, 110.0);
gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
headDeadzone = std::clamp(ConfDouble(conf, "POINTER_HEAD_DEADZONE", 0.5), 0.0, 5.0);
keyTap = std::clamp(ConfDouble(conf, "POINTER_KEY_TAP", 0.25), 0.0, 1.0);
const auto gd = conf.find("POINTER_GAZE_DOT");
gazeDotAlways = gd == conf.end() || gd->second != "moving";
gaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
handsOn = ConfDouble(conf, "POINTER_HANDS", 0) != 0;
pinchGain = std::clamp(ConfDouble(conf, "POINTER_PINCH_GAIN", 0.5), 0.05, 3.0);
pinchDeadzone = std::clamp(ConfDouble(conf, "POINTER_PINCH_DEADZONE", 1.5), 0.0, 10.0);
gripGain = std::clamp(ConfDouble(conf, "POINTER_GRIP_GAIN", 1.0), 0.05, 3.0);
handBelow = std::clamp(ConfDouble(conf, "POINTER_GRIP_BELOW", 0.35), 0.05, 1.0);
typingHold = std::clamp(ConfDouble(conf, "POINTER_PINCH_TYPING", 1.0), 0.0, 5.0);
const auto gm = conf.find("POINTER_GAZE_MOUSE");
gazeMousePrecision = gm == conf.end() || gm->second != "direct";
const auto mm = conf.find("POINTER_GAZE_MOUSE_MOVE");
gazeMouseHeld = mm == conf.end() || mm->second != "free";
const auto ro = conf.find("POINTER_ROLE");
role = ro != conf.end() && (ro->second == "left" || ro->second == "stylus") ? ro->second : "right";
pickupScale = std::clamp(ConfDouble(conf, "POINTER_CONTROLLER_PICKUP", 1), 0.5, 5.0);
const auto ig = conf.find("POINTER_IGNORE");
ignore = ParseIgnore(ig == conf.end() ? "" : ig->second);
}
};
+474
View File
@@ -0,0 +1,474 @@
// ft-pointer: recenter, head follow, the overlay handles, and the cursor (see "Cursor model" at
// the top).
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
namespace {
bool Ignored(const std::vector<std::string> &patterns, const std::string &key) {
for (const auto &p : patterns)
if (fnmatch(p.c_str(), key.c_str(), FNM_NOESCAPE) == 0) return true;
return false;
}
vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye) {
// Overlay faces +Z; point +Z at the eye, keep +Y roughly up.
const Vec3 z = Normalize(eye - at);
const Vec3 x = Normalize(Cross({0, 1, 0}, z));
const Vec3 y = Cross(z, x);
vr::HmdMatrix34_t m{};
const Vec3 cols[3] = {x, y, z};
for (int c = 0; c < 3; ++c) {
m.m[0][c] = float(cols[c].x);
m.m[1][c] = float(cols[c].y);
m.m[2][c] = float(cols[c].z);
}
m.m[0][3] = float(at.x);
m.m[1][3] = float(at.y);
m.m[2][3] = float(at.z);
return m;
}
// Unit vector v, turned toward unit vector `center` until it's at most maxRad from it.
Vec3 PullWithin(Vec3 v, Vec3 center, double maxRad) {
if (std::acos(std::clamp(Dot(v, center), -1.0, 1.0)) <= maxRad) return v;
Vec3 axis = Cross(center, v);
if (Length(axis) < 1e-9) axis = Cross(center, {0, 1, 0}); // opposite: any perpendicular
return RotateAbout(center, Normalize(axis), maxRad);
}
// Unit direction with its pitch limited to +-maxDeg (AimBasis needs it off vertical).
Vec3 LimitPitch(Vec3 d, double maxDeg) {
const double pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -maxDeg, maxDeg);
return Direction(std::atan2(-d.x, -d.z) * 180 / M_PI, pitch);
}
// SteamVR Settings page (see the top): the laser starts this far from the eye, the dot is
// drawn this far out, and the laser-catching dot sits this far out (metres).
constexpr double SETTINGS_ORIGIN = 0.25, SETTINGS_DOT = 0.6, SETTINGS_CATCHER = 8.0;
// Whether the line of sight from `eye` along `d` crosses the SteamVR Settings page, drawn by
// the dashboard's scene-graph panel (valve.steam.gamepadui.frame.menu.N, transform t); see
// "SteamVR Settings" at the top. The page has no size in OpenVR, so this is its area as
// measured on the Frame, generously: in metres from the panel's origin, which is near the
// page's left edge, it ran from about -0.35 (the sidebar) to 1.15 across and +-0.4 up and
// down, with the transform scaled 0.369. Kept in the transform's units so it scales with it.
bool OnSettingsPage(const vr::HmdMatrix34_t &t, Vec3 eye, Vec3 d) {
const Vec3 c = Position(t), x{t.m[0][0], t.m[1][0], t.m[2][0]}, y{t.m[0][1], t.m[1][1], t.m[2][1]},
z{t.m[0][2], t.m[1][2], t.m[2][2]};
const double sx = Dot(x, x), sy = Dot(y, y), denom = Dot(d, z);
if (sx < 1e-9 || sy < 1e-9 || std::fabs(denom) < 1e-6) return false;
const double along = Dot(c - eye, z) / denom;
if (along <= 0) return false;
const Vec3 off = eye + d * along - c;
const double u = Dot(off, x) / sx, v = Dot(off, y) / sy; // in the transform's units
return u >= -1.35 && u <= 3.4 && std::fabs(v) <= 1.25;
}
} // namespace
// A pending recenter (the command, a wake, a first move): the anchor goes to the eye, aimed where the
// head faces.
void Pointer::Recenter(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto &hm = hmd.mDeviceToAbsoluteTracking.m;
const Vec3 &eye = frame.eye;
if (recenter && hmd.bPoseIsValid) {
anchor = eye;
const Vec3 f{-hm[0][2], -hm[1][2], -hm[2][2]};
yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
anchored = true;
recenter = false;
followReset = true;
}
}
// Head follow (see the top): past the leash (for the delay), ease the reference to the
// head's facing, and turn the cursor with it. Not in gaze mode: the gaze places it.
void Pointer::HeadFollow(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto &hm = hmd.mDeviceToAbsoluteTracking.m;
const auto tnow = frame.tnow;
const Vec3 &eye = frame.eye;
if (follow && !gazeOn && active && anchored && hmd.bPoseIsValid) {
const Vec3 head = LimitPitch(Vec3{-hm[0][2], -hm[1][2], -hm[2][2]}, 85);
if (followReset) {
followRef = head, followLag = 0, leashOutSince = {};
followReset = following = false;
}
const double dt = std::min(0.1, std::chrono::duration<double>(tnow - followAt).count());
const double leash = cfg.leashDeg * M_PI / 180;
const double lag = std::acos(std::clamp(Dot(followRef, head), -1.0, 1.0));
double keep = lag; // how far the reference stays behind the head after this frame
if (leash <= 0) {
keep = 0; // head-locked
} else if (following) {
keep = lag * (cfg.leashReturn > 0 ? std::exp(-dt / cfg.leashReturn) : 0.0);
// A fast turn drags it at the leash's end; past it already (after the delay), it
// can't fall further behind, and it closes in from there without a jump.
keep = std::min(keep, std::max(leash, followLag));
if (keep < 0.05 * M_PI / 180) keep = 0, following = false; // landed on the facing
} else if (lag > leash) {
if (leashOutSince == decltype(leashOutSince){}) leashOutSince = tnow;
if (std::chrono::duration<double>(tnow - leashOutSince).count() >= cfg.leashDelay)
following = true, leashOutSince = {};
} else {
leashOutSince = {}; // back inside before the delay: a glance
}
followLag = keep;
const Vec3 ref = LimitPitch(PullWithin(followRef, head, keep), 85);
if (!leftHeld && tnow >= dropHoldUntil) {
// The cursor keeps its offset from the reference (frames without roll).
Vec3 d = FromBasis(AimBasis(ref), ToBasis(AimBasis(followRef), Direction(yaw, pitch)));
d = PullWithin(Normalize(d), ref, cfg.followReach * M_PI / 180);
yaw = std::atan2(-d.x, -d.z) * 180 / M_PI;
pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
// The ray origin closes in on the eye as the reference does on the facing.
anchor = eye + (anchor - eye) * (leash <= 0 ? 0.0 : lag > 1e-6 ? keep / lag : following ? 0.0 : 1.0);
}
followRef = ref;
}
followAt = tnow;
}
// Slow work, once a second: overlay handles, our device index, laser width.
void Pointer::SlowWork() {
const auto now = std::chrono::steady_clock::now();
if (now - lastSlow > std::chrono::seconds(1)) {
lastSlow = now;
handles.clear();
for (const auto &key : overlays.Keys()) {
if (Ignored(cfg.ignore, key)) continue;
vr::VROverlayHandle_t h;
if (overlay->FindOverlay(key.c_str(), &h) != vr::VROverlayError_None) continue;
handles[key] = h;
// Scene-graph overlays are placed absolutely. Other overlays can report no
// texture too (gamescope's app panels, placed as dashboard tabs, share theirs
// from another process), and ComputeOverlayIntersection handles those.
uint32_t tw = 0, th = 0;
overlay->GetOverlayTextureSize(h, &tw, &th);
vr::VROverlayTransformType tt = vr::VROverlayTransform_Invalid;
overlay->GetOverlayTransformType(h, &tt);
// ft-screens' panels (frametop.screen.N, floating windows with their popups,
// frametop.float.N..., the keyboard) are 0x0 and absolute too (a shared
// texture), but they're real panels of any size.
sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute &&
key.rfind("frametop.", 0) != 0;
}
ours = vr::k_unTrackedDeviceIndexInvalid;
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
char type[64] = "";
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
if (std::strcmp(type, "ft_pointer") == 0) ours = i;
}
}
if (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size()) {
lastVisible = now;
visible.clear();
for (const auto &[key, h] : handles) visible[key] = overlay->IsOverlayVisible(h);
}
}
// The cursor: tilting a grabbed panel, or pointing (find the spot, draw the dot, send the ray).
void Pointer::Cursor(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
if (active && anchored && hmd.bPoseIsValid && tilting) {
Tilt(frame);
} else if (active && anchored && hmd.bPoseIsValid) {
const Spot spot = FindCursor(frame);
DrawDot(frame, spot);
SendRay(frame, spot);
}
}
void Pointer::Tilt(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const Vec3 &eye = frame.eye;
const vr::TrackedDevicePose_t &hmdRaw = frame.hmdRaw;
// Rotate the device around the grab point; the grabbed panel turns with it.
if (tiltStart) {
pivot = lastPoint;
tiltOrigin = lastOrigin;
tiltBasis = AimBasis(lastAim);
tiltStart = false; // angles carry on from any earlier tilt in this drag
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
}
const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
const Vec3 up{0, 1, 0}, side = tiltBasis.x;
auto turn = [&](Vec3 v) { return RotateAbout(RotateAbout(v, side, pr), up, yr); };
const Vec3 originStanding = pivot + turn(tiltOrigin - pivot);
const Basis b{turn(tiltBasis.x), turn(tiltBasis.y), turn(tiltBasis.z)};
const auto &S = hmd.mDeviceToAbsoluteTracking, &R = hmdRaw.mDeviceToAbsoluteTracking;
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); }; // directions: raw <- standing
const Vec3 originRaw = Position(R) + toRaw(originStanding - eye);
double q[4];
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
char msg[200];
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", originRaw.x, originRaw.y,
originRaw.z, q[0], q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
}
// Nearest visible overlay along a ray (frozen while dragging).
Pointer::Hit Pointer::Nearest(Vec3 from, Vec3 d) {
Hit h;
for (const auto &[key, handle] : handles) {
if (!visible[key]) continue;
if (sceneGraph[key]) {
// Plane test: overlay origin and its +Z normal, within sceneRadius of the origin.
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t t{};
if (overlay->GetOverlayTransformAbsolute(handle, &uo, &t) != vr::VROverlayError_None) continue;
const Vec3 center = Position(t), normal{t.m[0][2], t.m[1][2], t.m[2][2]};
const double denom = Dot(d, normal);
if (std::fabs(denom) < 1e-4) continue;
const double along = Dot(center - from, normal) / denom;
const Vec3 at = from + d * along;
if (along > 0.05 && along < h.along && std::sqrt(Dot(at - center, at - center)) <= cfg.sceneRadius)
h.along = along, h.key = key, h.scene = true, h.point = at, h.normal = normal;
continue;
}
vr::VROverlayIntersectionParams_t params{};
params.vSource = {float(from.x), float(from.y), float(from.z)};
params.vDirection = {float(d.x), float(d.y), float(d.z)};
params.eOrigin = vr::TrackingUniverseStanding;
vr::VROverlayIntersectionResults_t r{};
if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
r.fDistance < h.along) {
h.along = r.fDistance, h.key = key, h.scene = false;
h.point = {r.vPoint.v[0], r.vPoint.v[1], r.vPoint.v[2]};
h.normal = {r.vNormal.v[0], r.vNormal.v[1], r.vNormal.v[2]};
}
}
return h;
}
// Where the cursor is this frame: on a panel, a scene-graph plane, or a panel's edge, or out in free
// space.
Pointer::Spot Pointer::FindCursor(const Frame &frame) {
const auto tnow = frame.tnow;
const Vec3 &eye = frame.eye;
const bool dragging = leftHeld || tnow < dropHoldUntil;
if (!dragging) tiltYaw = tiltPitch = 0; // drop finished: back to plain pointing
const Vec3 dir = Direction(yaw, pitch);
Hit first;
if (!dragging) first = Nearest(anchor, dir);
double best = first.along;
std::string bestKey = first.key;
bool bestScene = first.scene;
Vec3 bestPoint = first.point, bestNormal = first.normal;
bool onEdge = false;
if (!dragging && best < 1e8 && !bestScene) {
edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
} else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) {
// Just off a panel: stay on its plane (see "Panel edges" at the top).
const double denom = Dot(dir, edgeNormal);
if (std::fabs(denom) > 1e-4) {
const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
const Vec3 at = anchor + dir * along;
if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= cfg.edgeReach) {
best = along;
bestKey = edgeKey;
onEdge = true;
}
}
}
// Held on one of ft-screens' panels that stays where it is: onto whichever of them
// the ray meets (see the top).
if (leftHeld && !pressKey.empty()) {
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t now{};
auto it = handles.find(pressKey);
bool still = it != handles.end() &&
overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
for (int i = 0; still && i < 3; ++i)
for (int j = 0; j < 4; ++j)
if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
if (still) {
Hit h;
for (const auto &[key, handle] : handles) {
if (!visible[key] || !FramePanel(key)) continue;
vr::VROverlayIntersectionParams_t params{};
params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
params.eOrigin = vr::TrackingUniverseStanding;
vr::VROverlayIntersectionResults_t r{};
if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
r.fDistance < h.along)
h.along = r.fDistance, h.key = key;
}
if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
} else {
pressKey.clear(); // carried: the lock holds for the rest of this press
}
}
// While dragging: keep the press-time distance and show the non-interactive marker.
// On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
double distance = dragging ? dragDistance : (best < 1e8 ? best : cfg.freeDistance);
Vec3 point = anchor + dir * distance;
bool occluded = false;
if (!dragging) {
// The cursor lands on what you see under it: the ray above starts at the anchor,
// not the eye, so after leaning it can pick a panel that something nearer
// covers from where you are now (panels close together in view, at different
// depths). Anything in front of the point on the eye's line of sight wins.
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
const Hit front = Nearest(eye, Normalize(point - eye));
if (front.along < toPoint - 0.02) {
occluded = true;
bestKey = front.key, bestScene = front.scene;
point = front.point;
if (!front.scene) edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
edgeLast = front.point;
distance = std::sqrt(Dot(point - anchor, point - anchor));
best = distance;
onEdge = false;
}
lastDistance = distance, lastHit = bestKey;
}
const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge);
const bool onPanel = dragging || (best < 1e8 && !onScene);
Spot s;
s.dragging = dragging, s.dir = dir, s.point = point, s.best = best, s.distance = distance;
s.onEdge = onEdge, s.occluded = occluded, s.onScene = onScene, s.onPanel = onPanel;
return s;
}
// Our dot where the cursor is (see "Looks" and "SteamVR Settings" at the top).
void Pointer::DrawDot(const Frame &frame, const Spot &spot) {
const auto tnow = frame.tnow;
const Vec3 &eye = frame.eye;
const bool dragging = spot.dragging, onScene = spot.onScene, onPanel = spot.onPanel;
const Vec3 dir = spot.dir, point = spot.point;
const Vec3 sight = Normalize(point - eye);
if (!dragging) {
// SteamVR Settings (see the top): the dashboard's main panel is hidden and its
// scene-graph panel shows the page.
onVrSettings = false;
const auto mainIt = visible.find("valve.steam.gamepadui.main");
if (overlay->IsDashboardVisible() && !(mainIt != visible.end() && mainIt->second)) {
for (const auto &[key, handle] : handles) {
if (!visible[key] || key.rfind("valve.steam.gamepadui.frame.menu.", 0) != 0) continue;
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t t{};
if (overlay->GetOverlayTransformAbsolute(handle, &uo, &t) == vr::VROverlayError_None &&
OnSettingsPage(t, eye, sight))
onVrSettings = true;
}
}
}
if (onVrSettings != catcherHidesHit) {
overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings);
catcherHidesHit = onVrSettings;
}
if (onVrSettings) {
// The dot close in front of the page, which is nearer than any guess of ours;
// the laser-catching dot far behind everything, invisible, so it never covers
// the page, with SteamVR's hit dot hidden on it.
const Vec3 near = eye + sight * SETTINGS_DOT, far = eye + sight * SETTINGS_CATCHER;
double alpha = 1;
if (gazeOn && !cfg.gazeDotAlways) {
auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
alpha = std::clamp(1 - std::min(secs(lastMove) - cfg.gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
}
if (tnow < calPanelUntil) alpha = 0;
overlay->SetOverlayAlpha(marker, float(alpha));
overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cfg.cursorDeg * M_PI / 360)));
auto mm = Billboard(near, eye);
overlay->SetOverlayTransformAbsolute(marker, vr::TrackingUniverseStanding, &mm);
overlay->SetOverlayAlpha(cursor, 0);
overlay->SetOverlayWidthInMeters(cursor, float(2 * SETTINGS_CATCHER * std::tan(cfg.cursorDeg * M_PI / 360)));
auto mc = Billboard(far, eye);
overlay->SetOverlayTransformAbsolute(cursor, vr::TrackingUniverseStanding, &mc);
overlay->ShowOverlay(marker);
overlay->ShowOverlay(cursor);
} else {
// On a panel: the non-interactive marker, pulled 5 mm toward the eye so it
// draws on top. In free space: the interactive dot the laser lands on.
const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
const double dist = std::sqrt(Dot(at - eye, at - eye));
// Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only
// while something moves it or a press holds it (see the top); transparent
// otherwise, the laser still lands on it.
double scale = 1, alpha = 1;
if (gazeOn) {
auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
alpha = cfg.gazeDotAlways ? 1.0 : std::clamp(1 - std::min(secs(lastMove) - cfg.gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
const double pulse = secs(pulseAt);
if (pulse < 0.6) {
scale = 1 + 1.5 * std::max(0.0, 1 - pulse / 0.3);
alpha = std::max(alpha, std::clamp((0.6 - pulse) / 0.3, 0.0, 1.0));
}
}
if (tnow < calPanelUntil) alpha = 0; // the calibration panel is up (see the top)
overlay->SetOverlayAlpha(show, float(alpha));
overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cfg.cursorDeg * M_PI / 360)));
auto m = Billboard(at, eye);
overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m);
overlay->ShowOverlay(show);
overlay->HideOverlay(hide);
}
}
void Pointer::SendRay(const Frame &frame, const Spot &spot) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto tnow = frame.tnow;
const Vec3 &eye = frame.eye;
const vr::TrackedDevicePose_t &hmdRaw = frame.hmdRaw;
const bool dragging = spot.dragging, occluded = spot.occluded, onEdge = spot.onEdge, onScene = spot.onScene;
const double best = spot.best, distance = spot.distance;
const Vec3 point = spot.point;
// Controller ray: from the eye, aimed at the cursor point, converted from the
// standing universe to raw tracking space via the HMD's pose in both.
const Vec3 aimStanding = Normalize(point - eye);
const auto &S = hmd.mDeviceToAbsoluteTracking, &R = hmdRaw.mDeviceToAbsoluteTracking;
const Vec3 aim = Rotate(R, RotateInverse(S, aimStanding)); // raw <- head <- standing
// Origin partway along the line of sight to the cursor (smaller hit dot).
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
double originDist = std::max(0.0, std::min(toPoint * cfg.originFraction, toPoint - cfg.originMargin));
if (onVrSettings) originDist = std::min(originDist, SETTINGS_ORIGIN); // SteamVR finds the page
const Vec3 originStanding = eye + Normalize(point - eye) * originDist;
const Vec3 eyeRaw = Position(R) + Rotate(R, RotateInverse(S, originStanding - eye));
lastPoint = point, lastOrigin = originStanding, lastAim = aimStanding; // tilt starts from here
havePoint = true;
if (debug && tnow - lastDebug > std::chrono::milliseconds(500)) {
lastDebug = tnow;
if (systemPointer == vr::k_ulOverlayHandleInvalid) overlay->FindOverlay("system.pointer", &systemPointer);
std::printf("dbg %s hit=%s dist=%.2f eye->point=%.2f origin=%.2f vrsettings=%d yaw=%.1f pitch=%.1f gaze=%s steamvr_dot=%d primary=%u\n",
dragging ? "DRAG" : occluded ? "INFRONT" : onEdge ? "EDGE" : onScene ? "SCENE" : (best < 1e8 ? "PANEL" : "FREE"), lastHit.empty() ? "-" : lastHit.c_str(),
distance, toPoint, originDist, onVrSettings, yaw, pitch,
!gazeOn ? "off" : tnow - gz.at > std::chrono::milliseconds(150) ? "stale" : gazeOwns ? "owns" : "mouse",
systemPointer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(systemPointer),
overlay->GetPrimaryDashboardDevice());
std::fflush(stdout);
}
const double ayaw = std::atan2(-aim.x, -aim.z) * 180 / M_PI;
const double apitch = std::asin(std::clamp(aim.y, -1.0, 1.0)) * 180 / M_PI;
if (dragging && (tiltYaw != 0 || tiltPitch != 0)) {
// Keep this drag's tilt applied, about the current cursor point.
const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
const Basis base = AimBasis(aimStanding);
const Vec3 up{0, 1, 0}, side = base.x;
auto turn = [&](Vec3 v) { return RotateAbout(RotateAbout(v, side, pr), up, yr); };
const Vec3 o = point + turn(originStanding - point);
const Basis b{turn(base.x), turn(base.y), turn(base.z)};
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); };
const Vec3 oRaw = Position(R) + toRaw(o - eye);
double q[4];
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
char msg[200];
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", oRaw.x, oRaw.y, oRaw.z, q[0],
q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
} else {
char msg[160];
std::snprintf(msg, sizeof msg, "pose %.5f %.5f %.5f %.4f %.4f", eyeRaw.x, eyeRaw.y, eyeRaw.z, ayaw, apitch);
SendTo(out, "ft_pointer", msg);
}
}
+91
View File
@@ -0,0 +1,91 @@
// ft-pointer: gaze mode and its held-back press (see "Gaze mode" at the top).
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
// Outside games, gaze mode keeps the pointer (see the top); the relay needs to know.
void Pointer::GazeAwake() {
const auto t = Clock::now();
if (t - inGameAt > std::chrono::milliseconds(500)) {
inGameAt = t;
inGame = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
}
// Hand gestures in the last two minutes keep it the same way.
const bool handsRecent = cfg.handsOn && handUsed != Clock::time_point{} && t - handUsed < std::chrono::minutes(2);
const bool awake = (gazeOn || handsRecent) && !inGame && !headsetOff;
if (awake != gazeAwake || t - gazeAwakeAt > std::chrono::seconds(5)) {
if (awake != gazeAwake)
std::printf("%s keeps the pointer: %s\n", gazeOn ? "gaze" : "hand use",
awake ? "yes" : "no (off, in a game, or headset off)");
if (awake != gazeAwake) std::fflush(stdout);
gazeAwake = awake;
gazeAwakeAt = t;
SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
}
}
// Gaze mode (see the top): the gaze has the pointer, or takes it back when you look
// well away from it. Not while a press holds the pointer, and only on fresh gaze.
void Pointer::GazeMode(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto tnow = frame.tnow;
const Vec3 &eye = frame.eye;
if (hmd.bPoseIsValid) lastHead = hmd.mDeviceToAbsoluteTracking, haveHead = true;
if (gazeOn && active && hmd.bPoseIsValid && !tilting && !leftHeld && !aimHeld && !clickPress && !clickRelease &&
tnow >= dropHoldUntil &&
tnow - gz.at < std::chrono::milliseconds(150)) {
const Vec3 g = Rotate(hmd.mDeviceToAbsoluteTracking, Direction(gz.hy, gz.hp));
if (!gazeOwns && havePoint) {
const double off = std::acos(std::clamp(Dot(g, Normalize(lastPoint - eye)), -1.0, 1.0)) * 180 / M_PI;
if (off > cfg.gazeRetake && tnow - lastMouse > std::chrono::milliseconds(300)) {
if (retakeSince == Clock::time_point{}) retakeSince = tnow;
if (tnow - retakeSince >= std::chrono::milliseconds(120)) gazeOwns = true, nudging = false;
} else {
retakeSince = {};
}
}
if (gazeOwns) {
retakeSince = {};
anchor = eye;
anchored = true;
yaw = std::atan2(-g.x, -g.z) * 180 / M_PI;
pitch = std::clamp(std::asin(std::clamp(g.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
}
}
}
void Pointer::CalPanelOpened() {
if (calOpened) { // the calibration panel came up: a press in progress ends, no click
calOpened = false;
if (hold.src != Src::None) EndHold(true);
if (leftHeld) ReleaseLeft();
aimHeld = aimHand = clickPress = false;
}
}
// A held-back press (see the top): held still long enough, it's a real press (a drag);
// released, it's a click where the pointer is now (this frame's pose has gone out).
void Pointer::HeldBackPress(const Frame &frame) {
const auto tnow = frame.tnow;
if (!active) aimHeld = aimRight = clickPress = aimHand = confirmLesson = false; // released meanwhile: nothing to click
if (aimHeld && !aimHand && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration<double>(cfg.gazeHold)) {
aimHeld = false;
gazeBack = true;
pressRight = aimRight; // the right button's: a real right press (see the top)
aimRight = false;
PressLeft();
}
}
// The click a held-back press ends in: the press now, the release 40 ms later.
void Pointer::Click(const Frame &frame) {
const auto tnow = frame.tnow;
if (clickPress) {
clickPress = false;
PressLeft();
clickRelease = true;
clickReleaseAt = tnow + std::chrono::milliseconds(40);
} else if (clickRelease && tnow >= clickReleaseAt) {
clickRelease = false;
ReleaseLeft();
}
}
+202
View File
@@ -0,0 +1,202 @@
// ft-pointer: holds, hands, and the gaze precision buttons (see "Gaze precision" and "Hands" at
// the top).
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
// Hands (see the top): pinches and grips from ft-hands.
void Pointer::Hands(const Frame &frame) {
const auto tnow = frame.tnow;
vr::TrackedDevicePose_t h0;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h0, 1);
if (h0.bPoseIsValid) poses.Add(tnow, h0.mDeviceToAbsoluteTracking);
}
// Where a gesture's point is, seen from the hold's origin: yaw and pitch, degrees.
bool Pointer::HandAngles(const float p[3], uint64_t t_ns, const Vec3 &from, double &hy, double &hp) {
vr::HmdMatrix34_t head;
if (!poses.At(t_ns, head)) return false;
const Vec3 d = Normalize(Position(head) + Rotate(head, Vec3{p[0], p[1], p[2]}) - from);
hy = std::atan2(-d.x, -d.z) * 180 / M_PI;
hp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI;
return true;
}
void Pointer::EndHold(bool lost) {
if (hold.pressed) {
if (leftHeld) ReleaseLeft();
} else if (aimHeld) {
aimHeld = aimHand = false;
if (lost) {
if (gazeOn) gazeOwns = true, nudging = false; // no click: the pointer goes back to the gaze
} else {
clickPress = true; // the click is on the release, where the pointer is now
pressRight = hold.right;
gazeBack = nudgeMoved < 0.2;
}
}
if (debug)
std::printf("hold %s %s%s\n", hold.src == Src::Hand ? (hold.grip ? "grip" : "pinch")
: hold.src == Src::Head ? (hold.right ? "key right" : "key left")
: hold.grip ? "gaze drag" : "gaze precision",
lost ? "lost" : "released", hold.pressed ? "" : lost ? " (no click)" : " (click)");
if (debug) std::fflush(stdout);
hold = {};
}
// The press, once a hold's source is set: a real press (dragging until the release), or
// held back like gaze mode's mouse press (see the top): the click comes on the release.
void Pointer::StartHold(bool press, Clock::time_point tnow) {
hold.startYaw = hold.lastYaw = yaw, hold.startPitch = hold.lastPitch = pitch;
hold.pressed = press;
if (press) {
gazeBack = gazeOn && gazeOwns;
PressLeft();
} else {
gazeOwns = false;
nudging = gazeOn && haveHead && tnow - gz.at < std::chrono::milliseconds(200);
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
nudgeAt = aimSince = tnow, nudgeMoved = 0;
aimHeld = aimHand = true;
pulseAt = tnow;
}
}
// The hold's source moved to (hy, hp): past the dead zone, the pointer follows at the gain,
// from where it was when it got past.
void Pointer::Steer(double hy, double hp, double deadzone, double gain, Clock::time_point tnow) {
const double dy = std::remainder(hy - hold.refYaw, 360.0), dp = hp - hold.refPitch;
if (!hold.engaged) {
if (std::hypot(dy, dp) <= deadzone) return;
hold.engaged = true;
hold.refYaw = hy, hold.refPitch = hp;
hold.startYaw = hold.lastYaw = yaw, hold.startPitch = hold.lastPitch = pitch;
return;
}
yaw = std::remainder(hold.startYaw + gain * dy, 360.0);
pitch = std::clamp(hold.startPitch + gain * dp, -85.0, 85.0);
// How far the nudge went: for a keyboard click, from where the dot was at the press
// (a head wobbles on the way); otherwise the path, as with the mouse.
if (!hold.pressed && hold.src == Src::Head)
nudgeMoved = std::hypot(std::remainder(yaw - hold.pressYaw, 360.0), pitch - hold.pressPitch);
else if (!hold.pressed)
nudgeMoved += std::hypot(std::remainder(yaw - hold.lastYaw, 360.0), pitch - hold.lastPitch);
hold.lastYaw = yaw, hold.lastPitch = pitch;
lastMove = tnow; // the dot shows while it moves (gaze mode)
}
void Pointer::BeginHold(int side, bool grip, const fh_pinch_t &g, Clock::time_point tnow) {
handUsed = tnow;
if (hold.src != Src::None) {
// A grip takes over a pinch on the way to it (closing the hand passes through
// one); otherwise one hold at a time.
if (hold.src != Src::Hand || !grip || hold.grip) return;
aimHeld = aimHand = false;
hold = {};
}
if (leftHeld || aimHeld || clickPress || clickRelease) return; // the mouse's button is busy
if (!active) {
Wake(tnow); // the first gesture only wakes the pointer, like the first mouse move
return;
}
vr::HmdMatrix34_t head;
if (!poses.At(g.begin_ns, head)) return;
// No pinch just after a key (typing touches thumb to index), and a grip only with
// the hand held up (hands on a desk curl like a loose fist; looking down at them
// puts them straight ahead in the head's frame, where ft-hands can't tell).
const Vec3 at = Position(head) + Rotate(head, Vec3{g.begin_point[0], g.begin_point[1], g.begin_point[2]});
if (!grip && tnow - lastTyping < std::chrono::duration<double>(cfg.typingHold)) {
if (debug) std::printf("hand pinch ignored: typing\n");
return;
}
if (grip && Position(head).y - at.y > cfg.handBelow) {
if (debug) std::printf("hand grip ignored: %.2f m below the eyes\n", Position(head).y - at.y);
return;
}
Hold h;
h.src = Src::Hand, h.side = side, h.grip = grip, h.origin = Position(head);
if (!HandAngles(g.begin_point, g.begin_ns, h.origin, h.refYaw, h.refPitch)) return;
hold = h;
// A grip, or a pinch without gaze mode: a real press where the pointer is (where
// you look, in gaze mode), dragging with the hand until it opens.
StartHold(grip || !gazeOn, tnow);
if (debug) std::printf("hand %s %s began\n", side ? "right" : "left", grip ? "grip" : "pinch");
if (debug) std::fflush(stdout);
}
// Gaze precision and gaze drag buttons (see the top): the mouse steers.
void Pointer::GazePrecisionButtons(const Frame &frame) {
const auto tnow = frame.tnow;
for (const DevicePress &p : devicePresses) {
if (p.source == "left" || p.source == "right") continue; // no controllers in gaze mode
if (!p.down) {
if (hold.src == Src::Mouse) EndHold(false);
continue;
}
if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
Hold h;
h.src = Src::Mouse, h.grip = p.drag;
hold = h;
StartHold(p.drag, tnow);
if (debug) std::printf("hold gaze %s began (%s)\n", p.drag ? "drag" : "precision", p.source.c_str());
if (debug) std::fflush(stdout);
}
devicePresses.clear();
}
// Pinches and grips from ft-hands (see "Hands" at the top), read every frame.
void Pointer::PinchesAndGrips(Frame &frame) {
const auto tnow = frame.tnow;
fh_gestures_t hg;
const bool handOk = cfg.handsOn && handFile.Read(hg);
frame.handOk = handOk;
if (handOk && (hg.seq != handSeq || handFile.opens != handOpens)) {
handSeq = hg.seq;
handPublished = hg.publish_ns;
const fh_pinch_t *slots[2] = {hg.pinch, hg.grip};
// A new file, or a tracker whose counters went back: start counting from here.
bool rebase = !handBaseline || handFile.opens != handOpens;
for (int k = 0; k < 2; ++k)
for (int s = 0; s < 2; ++s)
rebase = rebase || slots[k][s].begins < seenBegins[k][s] || slots[k][s].ends < seenEnds[k][s];
if (rebase) {
if (hold.src == Src::Hand) EndHold(true);
for (int k = 0; k < 2; ++k)
for (int s = 0; s < 2; ++s) seenBegins[k][s] = slots[k][s].begins, seenEnds[k][s] = slots[k][s].ends;
handBaseline = true, handOpens = handFile.opens;
}
// Not over a VR game (unless the dashboard is up), and not with the headset off.
const bool allowed = !headsetOff && (!inGame || overlay->IsDashboardVisible());
for (int k = 1; k >= 0; --k) // grips first: one takes over a pinch
for (int s = 0; s < 2; ++s) {
const fh_pinch_t &g = slots[k][s];
const bool began = g.begins != seenBegins[k][s], ended = g.ends != seenEnds[k][s];
const bool lost = g.flags & FH_PINCH_LOST;
seenBegins[k][s] = g.begins, seenEnds[k][s] = g.ends;
auto holding = [&] { return hold.src == Src::Hand && hold.side == s && hold.grip == (k == 1); };
if (holding() && ended) EndHold(lost); // the one held ended (another may have begun)
if (began && allowed) {
BeginHold(s, k == 1, g, tnow);
// begun and ended since the last read (a quick tap): its release too
if (!(g.flags & FH_PINCH_DOWN) && holding()) EndHold(lost);
}
}
// The held gesture's hand moves the pointer: past the dead zone (a tap's jitter,
// the pinch point shifting as the fingers close), then at the gain, from there.
if (hold.src == Src::Hand) {
const fh_pinch_t &g = hold.grip ? hg.grip[hold.side] : hg.pinch[hold.side];
double hy, hp;
if ((g.flags & FH_PINCH_TRACKED) && HandAngles(g.point, hg.capture_ns, hold.origin, hy, hp))
Steer(hy, hp, hold.grip ? 0.5 : cfg.pinchDeadzone, hold.grip ? cfg.gripGain : cfg.pinchGain, tnow);
}
}
}
// ft-hands stopped (or hung) with a gesture held: it's over, as lost.
void Pointer::HandsStopped(const Frame &frame) {
const auto tnow = frame.tnow;
const bool handOk = frame.handOk;
const uint64_t nowNs =
uint64_t(std::chrono::duration_cast<std::chrono::nanoseconds>(tnow.time_since_epoch()).count());
if (hold.src == Src::Hand && (!handOk || !active || nowNs - handPublished > 1'500'000'000ull)) EndHold(true);
if (hold.src != Src::None && hold.src != Src::Hand && !active) EndHold(true);
}
+390
View File
@@ -0,0 +1,390 @@
// ft-pointer: waking the pointer, its buttons, and the commands from the relay.
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
void Pointer::Wake(Clock::time_point t) {
if (t < noWakeUntil || headsetOff) return;
wokeAt = t;
active = true;
recenter = true;
SendTo(out, "ft_pointer", "role " + cfg.role); // POINTER_ROLE, before it takes it
SendTo(out, "ft_pointer", "show");
claimPending = true; // take the laser without clicking, once SteamVR has bound the device
claimAt = t + std::chrono::milliseconds(300);
}
void Pointer::PressLeft() {
// ft-screens sends the keyboard to the panel clicked last; it sees clicks on
// its own screens, but only we know when one lands on another panel.
SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
// A click after nudging the gaze-placed pointer: the nudge is a lesson, or past
// POINTER_GAZE_NUDGE_MAX, a quick check (see the top).
if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
(confirmLesson || nudgeMoved >= 0.2)) {
const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
const double ty = std::atan2(-d.x, -d.z) * 180 / M_PI, tp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI;
const double off = std::hypot(std::remainder(ty - nudgeRawHy, 360.0), tp - nudgeRawHp);
char msg[160];
if (off <= cfg.gazeNudgeMax)
std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp, ty, tp);
else
std::snprintf(msg, sizeof msg, "recheck %.0f", off);
SendTo(out, "ft_gazed", msg);
if (debug) std::printf("gaze %s (nudged %.2f deg, off %.2f)\n", msg, nudgeMoved, off);
if (debug) std::fflush(stdout);
}
nudging = confirmLesson = false;
leftHeld = true;
dragDistance = lastDistance;
pressKey.clear();
if (FramePanel(lastHit)) {
vr::ETrackingUniverseOrigin uo;
auto it = handles.find(lastHit);
if (it != handles.end() &&
overlay->GetOverlayTransformAbsolute(it->second, &uo, &pressPose) == vr::VROverlayError_None)
pressKey = lastHit;
}
tiltYaw = tiltPitch = 0; // a new drag starts untilted
dropHoldUntil = {};
pulseAt = Clock::now();
heldButton = pressRight ? "b" : "trigger";
pressRight = false;
SendTo(out, "ft_pointer", "btn " + heldButton + " 1");
}
void Pointer::ReleaseLeft() {
leftHeld = false;
tilting = false;
// Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
SendTo(out, "ft_pointer", "btn " + heldButton + " 0");
// ft-screens releases a button held on its screens in KWin even when SteamVR hands
// the release to some other overlay (its catcher usually gets it; this is the backstop).
SendTo(out, "ft_screens", "up");
if (gazeBack) gazeOwns = true, gazeBack = false;
}
// The left button, from the relay's "btn trigger", with gaze mode's held-back press (see
// the top).
bool Pointer::CanAim() {
return gazeOn && cfg.gazeMousePrecision && gazeOwns && !aimHeld && !clickPress && !clickRelease &&
hold.src == Src::None;
}
// Hold the press back: the pointer stops where the gaze put it.
void Pointer::AimStart(bool right) {
gazeOwns = false;
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
aimHeld = true, aimRight = right;
}
// A drag the right button began (chordDrag): it lasts while either button is held.
void Pointer::ChordDrop() {
chordDrag = false;
if (leftHeld) ReleaseLeft();
if (debug) std::printf("mouse drag dropped\n");
if (debug) std::fflush(stdout);
}
void Pointer::LeftButton(bool down) {
leftDown = down;
if (down) {
if (aimHeld && aimRight && !aimHand) {
ignoreLeftUp = true; // the right's press is held back: the left does nothing
return;
}
if (CanAim()) {
AimStart(false);
return;
}
PressLeft();
return;
}
if (ignoreLeftUp) {
ignoreLeftUp = false;
return;
}
if (chordDrag) {
if (!rightDown) ChordDrop(); // otherwise the right holds it (tilting, maybe)
return;
}
if (aimHeld && !aimHand && !aimRight) {
aimHeld = false;
clickPress = true; // after this frame's pose, so it lands where the pointer was moved to
gazeBack = nudgeMoved < 0.2;
return;
}
if (leftHeld) ReleaseLeft();
}
// The right button (see the top); false: not taken here (a tilt, or passed on as it is).
bool Pointer::RightButton(bool down) {
rightDown = down;
if (down) {
if (aimHeld && !aimHand && !aimRight) {
// During the left's held-back press: press the left where the pointer is now (the
// correction is a lesson), and the mouse drags.
aimHeld = false;
chordDrag = gazeBack = true;
PressLeft();
if (debug) std::printf("mouse drag began (right during left)\n");
if (debug) std::fflush(stdout);
return true;
}
if (CanAim() && !leftHeld) {
AimStart(true);
return true;
}
return false;
}
if (chordDrag) {
if (leftDown) return !swallowedRight; // the left holds it; a tilt's release ends the tilt
tilting = swallowedRight = false;
ChordDrop();
return true;
}
if (aimHeld && !aimHand && aimRight) {
aimHeld = aimRight = false;
pressRight = clickPress = true; // the right click, where the pointer was moved to
gazeBack = nudgeMoved < 0.2;
return true;
}
if (leftHeld && heldButton == "b") { // its press, held still into a real one
ReleaseLeft();
return true;
}
return false;
}
// POINTER_GAZE_MOUSE_MOVE=held (see the top): the gaze is fresh and nothing is pressed, so a
// mouse move doesn't move the pointer.
bool Pointer::MouseMoveHeld() {
return gazeOn && cfg.gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) &&
!aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease;
}
// Commands from the relay.
void Pointer::RelayCommands() {
char buf[256];
ssize_t n;
sockaddr_un from{};
socklen_t fromLen = sizeof from;
while ((n = recvfrom(in, buf, sizeof buf - 1, 0, reinterpret_cast<sockaddr *>(&from), &fromLen)) > 0) {
buf[n] = 0;
// Reply to the sender (the layout tool binds an abstract address to get answers).
const sockaddr_un sender = from;
const socklen_t senderLen = fromLen;
fromLen = sizeof from;
Command(buf, sender, senderLen);
}
}
// One command from the relay (or another sender: reply answers it).
void Pointer::Command(const char *buf, const sockaddr_un &sender, socklen_t senderLen) {
auto reply = [&](const std::string &msg) {
if (senderLen > offsetof(sockaddr_un, sun_path))
sendto(out, msg.data(), msg.size(), 0, reinterpret_cast<const sockaddr *>(&sender), senderLen);
};
// The gaze, from ft-gazed: not mouse input, it never wakes the pointer.
double g[4];
if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) {
gz = {g[0], g[1], g[2], g[3], Clock::now()};
return;
}
// The gaze calibration panel (see the top): presses answer it instead of clicking.
{
int on;
if (std::sscanf(buf, "calpanel %d", &on) == 1) {
const bool was = Clock::now() < calPanelUntil;
calPanelUntil = on ? Clock::now() + std::chrono::seconds(3) : Clock::time_point{};
if (on && !was) calOpened = true;
return;
}
}
if (Clock::now() < calPanelUntil) {
const bool accept = !std::strncmp(buf, "btn trigger 1", 13) || !std::strncmp(buf, "gazekey left 1", 14);
const bool quit = !std::strncmp(buf, "btn b 1", 7) || !std::strncmp(buf, "gazekey right 1", 15);
if (accept || quit) {
SendTo(out, "ft_gazed", accept ? "calaccept" : "calquit");
return;
}
if (!std::strncmp(buf, "btn ", 4) || !std::strncmp(buf, "gazekey ", 8) ||
!std::strncmp(buf, "precision ", 10) || !std::strncmp(buf, "gazedrag ", 9))
return; // their releases, and the other buttons: nothing to click now
}
{
char kind[16], source[16];
int v;
if (std::sscanf(buf, "%15s %15s %d", kind, source, &v) == 3 &&
(!std::strcmp(kind, "precision") || !std::strcmp(kind, "gazedrag"))) {
lastMouse = Clock::now();
if (!active) Wake(Clock::now());
devicePresses.push_back({source, !std::strcmp(kind, "gazedrag"), v != 0});
return;
}
if (std::sscanf(buf, "gazekey %15s %d", source, &v) == 2 &&
(!std::strcmp(source, "left") || !std::strcmp(source, "right"))) {
lastMouse = Clock::now();
if (!active) Wake(Clock::now());
keyPresses.push_back({!std::strcmp(source, "right"), v != 0});
return;
}
}
if (std::strcmp(buf, "typing") == 0) { // not mouse input: it never wakes the pointer
lastTyping = Clock::now();
return;
}
if (std::strncmp(buf, "vrbind", 6) == 0) {
std::printf("controller buttons: %s\n", controllerButtons.Bind(buf + 6).c_str());
std::fflush(stdout);
return;
}
if (std::strncmp(buf, "vrglobal", 8) == 0) {
const char *arg = buf + 8;
while (*arg == ' ') ++arg;
ControllerButtons::SetGlobal(std::strncmp(arg, "off", 3) != 0);
reply(controllerButtons.Status());
return;
}
if (std::strncmp(buf, "vrstatus", 8) == 0) {
reply(controllerButtons.Status());
return;
}
if (std::strncmp(buf, "overlays", 8) == 0) {
overlays.Request(sender, senderLen); // answered from the list's thread
return;
}
if (std::strncmp(buf, "gaze", 4) == 0) {
const char *arg = buf + 4;
while (*arg == ' ') ++arg;
if (*arg != '?') { // "gaze ?" only asks
gazeOn = std::strncmp(arg, "on", 2) == 0 ? true
: std::strncmp(arg, "off", 3) == 0 ? false
: !gazeOn;
gazeOwns = true, nudging = false;
std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
std::fflush(stdout);
}
reply(gazeOn ? "ok on" : "ok off");
return;
}
const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 ||
std::strncmp(buf, "scroll", 6) == 0;
// A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
// the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
const bool moveHeldBack = std::strncmp(buf, "move", 4) == 0 && MouseMoveHeld();
if (mouseInput && !moveHeldBack) lastMouse = Clock::now();
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
if (!active && mouseInput) Wake(Clock::now());
if (moveHeldBack) return;
double a, b;
char key[128];
double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1;
if (std::sscanf(buf, "grabprobe %127s", key) == 1) {
GrabProbe(key);
return;
}
if (std::sscanf(buf, "place %127s %lf %lf %lf %lf %lf %lf %lf", key, &px, &py, &pz, &pyaw, &ppitch, &proll,
&pgrab) >= 6) {
reply(Place(key, {px, py, pz}, PanelBasis(pyaw, ppitch, proll), pgrab >= 0 ? pgrab : cfg.grabOffset));
return;
}
if (std::sscanf(buf, "measure %127s", key) == 1) {
Panel p;
Vec3 eye;
const std::string err = FindPanel(key, p, eye);
char msg[400] = "";
if (err.empty())
std::snprintf(msg, sizeof msg, "ok %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f",
p.center.x, p.center.y, p.center.z, p.width, p.height, p.basis.x.x, p.basis.x.y,
p.basis.x.z, p.basis.y.x, p.basis.y.y, p.basis.y.z, p.basis.z.x, p.basis.z.y,
p.basis.z.z);
reply(err.empty() ? msg : "error " + err);
return;
}
if (std::strncmp(buf, "head", 4) == 0) {
vr::TrackedDevicePose_t h;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h, 1);
const Vec3 e = Position(h.mDeviceToAbsoluteTracking);
const Vec3 f = Rotate(h.mDeviceToAbsoluteTracking, {0, 0, -1});
char msg[200];
std::snprintf(msg, sizeof msg, "ok %.4f %.4f %.4f %.2f %.2f", e.x, e.y, e.z,
std::atan2(-f.x, -f.z) * 180 / M_PI, std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI);
reply(h.bPoseIsValid ? msg : "error no head pose (headset off?)");
return;
}
if (std::strncmp(buf, "debug", 5) == 0) {
debug = !debug;
std::printf("debug %s\n", debug ? "on" : "off");
std::fflush(stdout);
return;
}
if (std::strncmp(buf, "btn trigger 1", 13) == 0) {
LeftButton(true);
return;
} else if (std::strncmp(buf, "btn trigger 0", 13) == 0) {
LeftButton(false);
return;
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && RightButton(true)) {
return;
} else if (std::strncmp(buf, "btn b 0", 7) == 0 && RightButton(false)) {
return;
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && leftHeld) {
tilting = tiltStart = swallowedRight = true; // right press while dragging: tilt, no right-click
return;
} else if (std::strncmp(buf, "btn b 0", 7) == 0 && swallowedRight) {
tilting = swallowedRight = false;
return;
}
if (tilting && std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
tiltYaw += a;
tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
return;
}
if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode)
if (gazeOn && gazeOwns) {
// The mouse takes the pointer from the gaze, from where the gaze left it.
gazeOwns = false;
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
nudgeAt = Clock::now(), nudgeMoved = 0;
}
if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
if (!anchored) recenter = true;
yaw += a;
while (yaw > 180) yaw -= 360;
while (yaw < -180) yaw += 360;
pitch = std::clamp(pitch + b, -85.0, 85.0);
} else if (std::strncmp(buf, "recenter", 8) == 0) {
recenter = true;
} else if (std::strncmp(buf, "reload", 6) == 0) {
cfg.Load(ReadConfig());
ApplyConfig();
lastSlow = Clock::now() - std::chrono::seconds(10); // apply POINTER_IGNORE now
std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg, "
"controller pickup %.1fx, %zu ignored\n",
cfg.freeDistance, cfg.cursorDeg, cfg.originFraction, follow ? "on" : "off", cfg.leashDeg, cfg.pickupScale,
cfg.ignore.size());
std::fflush(stdout);
} else if (std::strncmp(buf, "follow", 6) == 0) {
const char *arg = buf + 6;
while (*arg == ' ') ++arg;
const bool was = follow;
follow = std::strncmp(arg, "on", 2) == 0 ? true : std::strncmp(arg, "off", 3) == 0 ? false : !follow;
if (follow && !was) followReset = true;
std::printf("head follow %s (leash %.0f deg)\n", follow ? "on" : "off", cfg.leashDeg);
std::fflush(stdout);
} else if (std::strncmp(buf, "show", 4) == 0) {
if (!active) Wake(Clock::now());
} else if (std::strncmp(buf, "hide", 4) == 0) {
active = false;
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
SendTo(out, "ft_pointer", "hide");
} else {
SendTo(out, "ft_pointer", buf); // btn, scroll
}
}
+113
View File
@@ -0,0 +1,113 @@
// ft-pointer: keyboard clicks (see "Keyboard clicks" under "Gaze precision" at the top).
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
// Where the head faces: yaw and pitch, degrees.
bool Pointer::HeadAngles(const vr::TrackedDevicePose_t &hmd, double &hy, double &hp) {
const auto &hm = hmd.mDeviceToAbsoluteTracking.m;
if (!hmd.bPoseIsValid) return false;
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
hy = std::atan2(-f.x, -f.z) * 180 / M_PI;
hp = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
return true;
}
// Keyboard clicks (see the top): held back at the gaze, steered by the head.
void Pointer::KeyboardClicks(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto tnow = frame.tnow;
for (const KeyPress &k : keyPresses) {
(k.right ? keyRightDown : keyLeftDown) = k.down;
if (!k.down) {
if (keyTilting && k.right) { // the tilt ends; the head drags again from here
keyTilting = tilting = false;
hold.engaged = false;
if (debug) std::printf("hold key left: tilt ended\n");
if (debug) std::fflush(stdout);
}
// The keys holding it: a gaze_left then gaze_right drag, either; otherwise its own.
const bool held = hold.keyDrag ? keyLeftDown || keyRightDown : hold.right ? keyRightDown : keyLeftDown;
if (hold.src == Src::Head && !held) {
if (!hold.pressed && aimHeld && tnow - aimSince < std::chrono::duration<double>(cfg.keyTap)) {
// A quick tap: a click where the dot was at the press, and the gaze was right.
yaw = hold.pressYaw, pitch = hold.pressPitch;
nudgeMoved = 0;
confirmLesson = true;
}
EndHold(false);
keyTilting = false;
}
continue;
}
if (k.right && hold.src == Src::Head && !hold.right && hold.pressed && leftHeld) {
// gaze_right during a gaze_left drag: tilt while it's held (see the top).
tilting = tiltStart = keyTilting = true;
HeadAngles(hmd, keyTiltYaw, keyTiltPitch);
if (debug) std::printf("hold key left: tilt began\n");
if (debug) std::fflush(stdout);
continue;
}
if (k.right && hold.src == Src::Head && !hold.right && !hold.pressed && aimHeld) {
// gaze_right while gaze_left aims: press the left button where the dot is now
// (the correction is a lesson), then the head drags.
aimHeld = aimHand = false;
hold.pressed = hold.grip = hold.keyDrag = true, hold.promote = false, hold.engaged = false;
HeadAngles(hmd, hold.refYaw, hold.refPitch);
gazeBack = gazeOn;
pressRight = false;
PressLeft();
if (debug) std::printf("hold key left: pressed (right key)\n");
if (debug) std::fflush(stdout);
continue;
}
if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
Hold h;
h.src = Src::Head, h.promote = true, h.right = k.right, h.pressYaw = yaw, h.pressPitch = pitch;
if (!HeadAngles(hmd, h.refYaw, h.refPitch)) continue;
hold = h;
StartHold(false, tnow);
if (debug) std::printf("hold key %s began\n", k.right ? "right" : "left");
if (debug) std::fflush(stdout);
}
keyPresses.clear();
}
// A keyboard click's hold: the head steers the pointer, or turns the panel in a tilt.
void Pointer::HeadSteer(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto tnow = frame.tnow;
if (hold.src == Src::Head) {
double hy, hp;
if (HeadAngles(hmd, hy, hp) && keyTilting) {
// The head turns the panel, as the mouse does in a tilt; the pointer stays put.
tiltYaw += std::remainder(hy - keyTiltYaw, 360.0);
tiltPitch = std::clamp(tiltPitch + hp - keyTiltPitch, -80.0, 80.0);
keyTiltYaw = hy, keyTiltPitch = hp;
} else if (HeadAngles(hmd, hy, hp)) {
Steer(hy, hp, hold.pressed ? 0.3 : cfg.headDeadzone, 1.0, tnow);
}
}
}
// A keyboard click held still for POINTER_GAZE_HOLD: a real press, then the head drags
// (from where it is now).
void Pointer::KeyboardClickHold(const Frame &frame) {
const vr::TrackedDevicePose_t &hmd = frame.Hmd();
const auto &hm = hmd.mDeviceToAbsoluteTracking.m;
const auto tnow = frame.tnow;
if (aimHeld && aimHand && hold.promote && !hold.engaged &&
tnow - aimSince >= std::chrono::duration<double>(cfg.gazeHold)) {
aimHeld = aimHand = false;
hold.pressed = hold.grip = true;
if (hmd.bPoseIsValid) {
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
hold.refYaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
hold.refPitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
}
gazeBack = gazeOn;
pressRight = hold.right;
PressLeft();
if (debug) std::printf("hold key %s: pressed (held still)\n", hold.right ? "right" : "left");
if (debug) std::fflush(stdout);
}
}
+168
View File
@@ -0,0 +1,168 @@
// ft-pointer: panel placement for layouts (see "Placement" at the top).
// "The top" in comments here is the header comment of ft-pointer.cpp.
#include "pointer.h"
namespace {
void SleepMs(int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); }
} // namespace
// Placement speeds (see "Placement" at the top).
constexpr double kPlaceDegPerSec = 60; // tested: 40 deg/s is applied exactly
// --- Panel placement (see "Placement" at the top of the file) ---
// Device pose, given in the standing universe, sent to the driver in raw space.
void Pointer::SendPose(Vec3 originStanding, const Basis &b) {
vr::TrackedDevicePose_t s, r;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &s, 1);
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, &r, 1);
const auto &S = s.mDeviceToAbsoluteTracking, &R = r.mDeviceToAbsoluteTracking;
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); };
const Vec3 o = Position(R) + toRaw(originStanding - Position(S));
double q[4];
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
char msg[200];
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", o.x, o.y, o.z, q[0], q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
}
std::pair<bool, Vec3> Pointer::HeadPos() {
vr::TrackedDevicePose_t s;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &s, 1);
return std::make_pair(s.bPoseIsValid, Position(s.mDeviceToAbsoluteTracking));
}
// Borrow the device and the laser for a placement: connect, claim, laser mode on.
void Pointer::Borrow() {
SendTo(out, "ft_pointer", "show");
overlay->ShowOverlay(laserMode);
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
SleepMs(active ? 50 : 400); // a fresh connect needs SteamVR to bind the device
SendTo(out, "ft_pointer", "btn a 1");
SleepMs(60);
SendTo(out, "ft_pointer", "btn a 0");
}
void Pointer::GiveBack() {
if (!active) {
SendTo(out, "ft_pointer", "hide");
overlay->HideOverlay(laserMode);
laserModeShown = false;
}
}
std::string Pointer::FindPanel(const char *key, Panel &p, Vec3 &eye) {
vr::VROverlayHandle_t h;
if (overlay->FindOverlay(key, &h) != vr::VROverlayError_None) return std::string("no overlay ") + key;
bool valid;
std::tie(valid, eye) = HeadPos();
if (!valid) return "no head pose (headset off?)";
p = ScanPanel(h, eye, 0.5);
if (!p.found) return std::string("panel not visible: ") + key;
return "";
}
// grabprobe (maintenance): aim down from the panel's bottom edge, 1 cm a step, and log where
// SteamVR's laser hits something (the hit dot shows) and whether the window controls are up.
void Pointer::GrabProbe(const char *key) {
Panel p;
Vec3 eye;
const std::string err = FindPanel(key, p, eye);
if (!err.empty()) {
std::printf("grabprobe: %s\n", err.c_str());
std::fflush(stdout);
return;
}
Borrow();
vr::VROverlayHandle_t footer = vr::k_ulOverlayHandleInvalid;
overlay->FindOverlay("valve.steam.gamepadui.floatingfooter", &footer);
const Vec3 bottom = p.center - p.basis.y * (p.height / 2);
std::printf("grabprobe %s: center (%.3f %.3f %.3f) %.3f x %.3f m\n", key, p.center.x, p.center.y, p.center.z,
p.width, p.height);
for (int cm = -5; cm <= 45; ++cm) {
const Vec3 target = bottom - p.basis.y * (cm / 100.0);
SendPose(eye, AimBasis(target - eye));
SleepMs(90);
const bool dot = systemPointer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(systemPointer);
const bool foot = footer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(footer);
std::printf(" %+3d cm below the bottom edge: steamvr_dot=%d footer=%d", cm, dot, foot);
if (foot) {
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t t{};
if (overlay->GetOverlayTransformAbsolute(footer, &uo, &t) == vr::VROverlayError_None) {
const Vec3 f = Position(t) - p.center;
std::printf(" footer at panel (%.3f %.3f %.3f)", Dot(f, p.basis.x), Dot(f, p.basis.y),
Dot(f, p.basis.z));
}
}
std::printf("\n");
}
std::fflush(stdout);
GiveBack();
}
// Carry a floating panel so its centre lands on `target` with frame `bt` (see
// "Placement" at the top). Returns "ok ..." or "error ...".
std::string Pointer::Place(const char *key, Vec3 target, const Basis &bt, double grabBelow) {
Panel p;
Vec3 eye;
std::string err = FindPanel(key, p, eye);
if (!err.empty()) return "error " + err;
auto offBy = [&](const Panel &q, double &cm, double &deg) {
cm = Length(q.center - target) * 100;
const double c = (Dot(q.basis.x, bt.x) + Dot(q.basis.y, bt.y) + Dot(q.basis.z, bt.z) - 1) / 2;
deg = std::acos(std::clamp(c, -1.0, 1.0)) * 180 / M_PI;
};
double cm, deg;
int moves = 0;
Borrow();
for (int attempt = 0; attempt < 3; ++attempt) {
offBy(p, cm, deg);
if (cm < 1.5 && deg < 1.0) break;
// The rigid motion that takes the panel to the target: rotate by R, then move.
auto turn = [&](Vec3 v) { return FromBasis(bt, ToBasis(p.basis, v)); };
double q[4];
BasisQuat({turn({1, 0, 0}), turn({0, 1, 0}), turn({0, 0, 1})}, q);
const double angle = 2 * std::acos(std::clamp(q[0], -1.0, 1.0));
const Vec3 axis = std::sin(angle / 2) > 1e-6 ? Normalize({q[1], q[2], q[3]}) : Vec3{0, 1, 0};
const Vec3 grab = p.center - p.basis.y * (p.height / 2 + grabBelow);
const Basis d0 = AimBasis(grab - eye);
const Vec3 o1 = target + turn(eye - p.center); // device origin at the end
++moves;
SendPose(eye, d0);
SleepMs(150); // hover: the window controls come up
SendTo(out, "ft_pointer", "btn trigger 1");
SleepMs(150);
// 1. Rotate about the device origin (the eye): the dashboard applies it exactly.
const int rsteps = std::max(4, int(angle * 180 / M_PI / kPlaceDegPerSec * 60));
for (int i = 1; i <= rsteps; ++i) {
const double a = angle * i / rsteps;
auto r = [&](Vec3 v) { return RotateAbout(v, axis, a); };
SendPose(eye, {r(d0.x), r(d0.y), r(d0.z)});
SleepMs(16);
}
// 2. Slide the device slowly: fast moves are accelerated by the dashboard.
const Basis d1{turn(d0.x), turn(d0.y), turn(d0.z)};
const int tsteps = std::max(4, int(Length(o1 - eye) / cfg.slideSpeed * 60));
for (int i = 1; i <= tsteps; ++i) {
SendPose(eye + (o1 - eye) * (double(i) / tsteps), d1);
SleepMs(16);
}
SleepMs(100);
SendTo(out, "ft_pointer", "btn trigger 0");
SleepMs(600); // the dashboard re-reads the pose up to 150 ms after the release
err = FindPanel(key, p, eye);
if (!err.empty()) break;
}
GiveBack();
if (!err.empty()) return "error after the move: " + err;
offBy(p, cm, deg);
char msg[160];
std::snprintf(msg, sizeof msg, "ok %s off by %.1f cm, %.1f deg after %d move%s", key, cm, deg, moves,
moves == 1 ? "" : "s");
std::printf("place: %s\n", msg);
std::fflush(stdout);
return msg;
}
+2 -4
View File
@@ -14,10 +14,8 @@ case ${1:-status} in
"$root/scripts/sync.sh" >/dev/null "$root/scripts/sync.sh" >/dev/null
fill_template "$root/pointer/helper/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit" fill_template "$root/pointer/helper/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
"$frame" --host "set -e; pkill -x ft-pointer || true "$frame" --host "set -e; pkill -x ft-pointer || true
systemctl --user daemon-reload; systemctl --user enable --now $unit systemctl --user daemon-reload; systemctl --user enable $unit
# It comes up once SteamVR runs (it starts with it); distrobox enter takes a few seconds. $(start_with_steamvr $unit)" ;;
for i in \$(seq 20); do systemctl --user is-active --quiet $unit && break; sleep 1; done
echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 3" ;;
uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;; uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
start|stop|restart) if installed; then "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit"; exit; fi ;;& start|stop|restart) if installed; then "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit"; exit; fi ;;&
log) if installed; then "$frame" --host "journalctl --user -u $unit --no-pager -o cat -n ${2:-30}"; exit; fi ;;& log) if installed; then "$frame" --host "journalctl --user -u $unit --no-pager -o cat -n ${2:-30}"; exit; fi ;;&
+2 -8
View File
@@ -1,15 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Build vrprobe, lasertest, vrsetting and focustest in the dev container on the Frame (pointer/probe/build/vrprobe). # Build vrprobe in the dev container on the Frame (pointer/probe/build/vrprobe).
set -euo pipefail set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null "$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer/probe 'set -e; mkdir -p build exec "$root/scripts/frame.sh" -C pointer/probe 'set -e; mkdir -p build
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \ g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \
-o build/vrprobe vrprobe.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -o build/vrprobe vrprobe.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \ echo "built build/vrprobe"'
-o build/lasertest lasertest.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \
-o build/vrsetting vrsetting.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \
-o build/focustest focustest.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
echo "built build/vrprobe build/lasertest build/vrsetting build/focustest"'
-135
View File
@@ -1,135 +0,0 @@
// Does a dashboard overlay with overlay flag 1 << 4 take SteamVR's input focus for its own process
// (docs/gaze-first.md)? SteamVR's dashboard gives an external overlay's page a "VR client" input
// focus (its owner's PID) when that flag is set, and a Steam Input one with 1 << 30. While a VR
// client has the input focus, as a game does, Steam shouldn't get the controllers.
// Shows a dashboard overlay "Frametop focus test" (a plain colour) with that flag, and prints
// input-focus events, IsInputAvailable(), and the dashboard's visibility as they change.
// Runs as an OpenVR overlay client (in the dev container).
// With --manifest it's a SteamVR input client too (focustest_actions/: Y and the triggers), and
// prints when those actions change, which shows whether it gets them while Steam doesn't.
// Usage: focustest [--seconds N] [--no-flag] [--steam-flag] [--manifest]
#include <openvr.h>
#include <chrono>
#include <csignal>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <limits.h>
#include <stdlib.h>
#include <string>
#include <thread>
#include <unistd.h>
#include <vector>
using Clock = std::chrono::steady_clock;
static volatile std::sig_atomic_t g_stop = 0;
int main(int argc, char **argv) {
double seconds = 0;
bool flag = true, steamFlag = false, manifest = false;
for (int i = 1; i < argc; ++i) {
if (!std::strcmp(argv[i], "--seconds") && i + 1 < argc) seconds = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--no-flag")) flag = false;
else if (!std::strcmp(argv[i], "--steam-flag")) steamFlag = true;
else if (!std::strcmp(argv[i], "--manifest")) manifest = true;
else {
std::fprintf(stderr, "usage: %s [--seconds N] [--no-flag] [--steam-flag] [--manifest]\n", argv[0]);
return 2;
}
}
std::signal(SIGINT, [](int) { g_stop = 1; });
std::signal(SIGTERM, [](int) { g_stop = 1; });
vr::EVRInitError err = vr::VRInitError_None;
vr::IVRSystem *sys = vr::VR_Init(&err, vr::VRApplication_Overlay);
if (err != vr::VRInitError_None) {
std::printf("VR_Init failed: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
return 1;
}
vr::IVROverlay *ov = vr::VROverlay();
vr::VRActionSetHandle_t set = vr::k_ulInvalidActionSetHandle;
vr::VRActionHandle_t actY = vr::k_ulInvalidActionHandle, actTrigger = vr::k_ulInvalidActionHandle;
if (manifest) {
char exe[PATH_MAX] = "";
const ssize_t n = readlink("/proc/self/exe", exe, sizeof exe - 1);
std::string dir = n > 0 ? std::string(exe, n) : std::string(".");
dir = dir.substr(0, dir.rfind('/')); // build/
dir = dir.substr(0, dir.rfind('/')); // pointer/probe
const std::string path = dir + "/focustest_actions/actions.json";
const auto ie = vr::VRInput()->SetActionManifestPath(path.c_str());
vr::VRInput()->GetActionSetHandle("/actions/focus", &set);
vr::VRInput()->GetActionHandle("/actions/focus/in/y", &actY);
vr::VRInput()->GetActionHandle("/actions/focus/in/trigger", &actTrigger);
std::printf("action manifest %s: error %d\n", path.c_str(), int(ie));
}
vr::VROverlayHandle_t main = vr::k_ulOverlayHandleInvalid, thumb = vr::k_ulOverlayHandleInvalid;
auto e = ov->CreateDashboardOverlay("frametop.focustest", "Frametop focus test", &main, &thumb);
if (e != vr::VROverlayError_None) {
std::printf("CreateDashboardOverlay: %s\n", ov->GetOverlayErrorNameFromEnum(e));
return 1;
}
// A plain teal page, and a teal thumbnail.
const uint32_t w = 256, h = 160;
std::vector<uint8_t> px(w * h * 4);
for (uint32_t i = 0; i < w * h; ++i) px[i * 4] = 20, px[i * 4 + 1] = 140, px[i * 4 + 2] = 140, px[i * 4 + 3] = 255;
ov->SetOverlayRaw(main, px.data(), w, h, 4);
ov->SetOverlayRaw(thumb, px.data(), w, h, 4);
ov->SetOverlayWidthInMeters(main, 1.5f);
ov->SetOverlayInputMethod(main, vr::VROverlayInputMethod_Mouse);
if (flag) {
e = ov->SetOverlayFlag(main, vr::VROverlayFlags(1 << 4), true);
std::printf("SetOverlayFlag(1 << 4): %s\n", ov->GetOverlayErrorNameFromEnum(e));
}
if (steamFlag) {
e = ov->SetOverlayFlag(main, vr::VROverlayFlags(1 << 30), true);
std::printf("SetOverlayFlag(1 << 30): %s\n", ov->GetOverlayErrorNameFromEnum(e));
}
uint32_t flags = 0;
ov->GetOverlayFlags(main, &flags);
std::printf("overlay flags now 0x%x; pid %d\n", flags, int(getpid()));
std::fflush(stdout);
const auto start = Clock::now();
auto ms = [&] { return std::chrono::duration<double, std::milli>(Clock::now() - start).count(); };
int lastAvail = -1, lastDash = -1, lastActive = -1, lastY = -1, lastTrigger = -1;
while (!g_stop && (seconds <= 0 || ms() < seconds * 1000)) {
vr::VREvent_t ev;
while (sys->PollNextEvent(&ev, sizeof ev)) {
const char *name = sys->GetEventTypeNameFromEnum(vr::EVREventType(ev.eventType));
if (ev.eventType == vr::VREvent_InputFocusChanged || ev.eventType == vr::VREvent_InputFocusCaptured ||
ev.eventType == vr::VREvent_InputFocusReleased)
std::printf("%9.1f ms %s pid %u (old %u)\n", ms(), name, ev.data.process.pid, ev.data.process.oldPid);
else if (ev.eventType == vr::VREvent_ButtonPress || ev.eventType == vr::VREvent_ButtonUnpress ||
ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated)
std::printf("%9.1f ms %s device %u button %u\n", ms(), name, ev.trackedDeviceIndex, ev.data.controller.button);
}
while (ov->PollNextOverlayEvent(main, &ev, sizeof ev)) {
const char *name = sys->GetEventTypeNameFromEnum(vr::EVREventType(ev.eventType));
if (ev.eventType != vr::VREvent_MouseMove && std::strncmp(name, "Unknown", 7) != 0 &&
std::strcmp(name, "VREvent_OverlayMouseFocusChanged") != 0 && ev.eventType != vr::VREvent_PropertyChanged)
std::printf("%9.1f ms overlay: %s\n", ms(), name);
}
if (manifest) {
vr::VRActiveActionSet_t active{};
active.ulActionSet = set;
vr::VRInput()->UpdateActionState(&active, sizeof active, 1);
vr::InputDigitalActionData_t y{}, t{};
vr::VRInput()->GetDigitalActionData(actY, &y, sizeof y, vr::k_ulInvalidInputValueHandle);
vr::VRInput()->GetDigitalActionData(actTrigger, &t, sizeof t, vr::k_ulInvalidInputValueHandle);
const int yv = y.bActive ? y.bState : -2, tv = t.bActive ? t.bState : -2;
if (yv != lastY || tv != lastTrigger) {
std::printf("%9.1f ms our actions: y %d, trigger %d (-2: inactive)\n", ms(), yv, tv);
lastY = yv, lastTrigger = tv;
}
}
const int avail = sys->IsInputAvailable(), dash = ov->IsDashboardVisible(), active = ov->IsActiveDashboardOverlay(main);
if (avail != lastAvail || dash != lastDash || active != lastActive) {
std::printf("%9.1f ms input available %d, dashboard %d, our page active %d\n", ms(), avail, dash, active);
lastAvail = avail, lastDash = dash, lastActive = active;
}
std::fflush(stdout);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
vr::VR_Shutdown();
return 0;
}
@@ -1,15 +0,0 @@
{
"default_bindings": [
{"controller_type": "frame_controller", "binding_url": "bindings_frame_controller.json"}
],
"actions": [
{"name": "/actions/focus/in/y", "type": "boolean"},
{"name": "/actions/focus/in/trigger", "type": "boolean"}
],
"action_sets": [
{"name": "/actions/focus", "usage": "leftright"}
],
"localization": [
{"language_tag": "en_US", "/actions/focus": "Focus test", "/actions/focus/in/y": "Y", "/actions/focus/in/trigger": "Trigger"}
]
}
@@ -1,15 +0,0 @@
{
"action_manifest_version": 0,
"controller_type": "frame_controller",
"description": "focustest (pointer/probe/focustest.cpp)",
"name": "focustest",
"bindings": {
"/actions/focus": {
"sources": [
{"path": "/user/hand/right/input/y", "mode": "button", "inputs": {"click": {"output": "/actions/focus/in/y"}}},
{"path": "/user/hand/right/input/trigger", "mode": "button", "inputs": {"click": {"output": "/actions/focus/in/trigger"}}},
{"path": "/user/hand/left/input/trigger", "mode": "button", "inputs": {"click": {"output": "/actions/focus/in/trigger"}}}
]
}
}
}
-147
View File
@@ -1,147 +0,0 @@
// Watch who has SteamVR's laser mouse, for the gaze-first tests (docs/gaze-first.md, tests 1-3).
// Prints the dashboard's primary device and every controller's role whenever either changes,
// and the dashboard and role events. With --snapback, when the laser goes to any device but
// ours (the ft_pointer driver's), it presses our /input/a (switchlaserhand) to take it back,
// at most every 100 ms, and prints how long that took. With --reclaim MS it also takes the
// laser back when it has been on no device (gamepad mode) for MS milliseconds.
// Runs as a background OpenVR client (in the dev container).
// Usage: lasertest [--snapback] [--reclaim MS] [--seconds N]
#include <openvr.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <chrono>
#include <csignal>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include <thread>
using Clock = std::chrono::steady_clock;
static volatile std::sig_atomic_t g_stop = 0;
static void Send(const char *msg) {
static int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
sockaddr_un a{};
a.sun_family = AF_UNIX;
const char name[] = "ft_pointer"; // the driver's abstract socket
std::memcpy(a.sun_path + 1, name, sizeof name - 1);
sendto(fd, msg, std::strlen(msg), 0, reinterpret_cast<sockaddr *>(&a),
socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1));
}
static std::string Describe(vr::IVRSystem *sys, vr::TrackedDeviceIndex_t i) {
if (i == vr::k_unTrackedDeviceIndexInvalid) return "none";
static const char *roles[] = {"none", "left", "right", "optout", "treadmill", "stylus"};
char type[64] = "", serial[64] = "";
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
sys->GetStringTrackedDeviceProperty(i, vr::Prop_SerialNumber_String, serial, sizeof serial);
const auto role = sys->GetControllerRoleForTrackedDeviceIndex(i);
const int hint = sys->GetInt32TrackedDeviceProperty(i, vr::Prop_ControllerRoleHint_Int32);
char out[200];
std::snprintf(out, sizeof out, "%u %s %s role=%s hint=%d", i, type, serial, role < 6 ? roles[role] : "?", hint);
return out;
}
int main(int argc, char **argv) {
bool snapback = false;
double seconds = 0, reclaimMs = 0;
for (int i = 1; i < argc; ++i) {
if (!std::strcmp(argv[i], "--snapback")) snapback = true;
else if (!std::strcmp(argv[i], "--seconds") && i + 1 < argc) seconds = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--reclaim") && i + 1 < argc) reclaimMs = std::atof(argv[++i]);
else {
std::fprintf(stderr, "usage: %s [--snapback] [--reclaim MS] [--seconds N]\n", argv[0]);
return 2;
}
}
std::signal(SIGINT, [](int) { g_stop = 1; });
std::signal(SIGTERM, [](int) { g_stop = 1; });
vr::EVRInitError err = vr::VRInitError_None;
vr::IVRSystem *sys = vr::VR_Init(&err, vr::VRApplication_Background);
if (err != vr::VRInitError_None) {
std::printf("VR_Init failed: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
return 1;
}
const auto start = Clock::now();
auto ms = [&] { return std::chrono::duration<double, std::milli>(Clock::now() - start).count(); };
std::string lastPrimary, lastRoles;
Clock::time_point lostAt{}, pressedAt{}, noneSince{};
bool pressed = false;
while (!g_stop && (seconds <= 0 || ms() < seconds * 1000)) {
vr::VREvent_t ev;
while (sys->PollNextEvent(&ev, sizeof ev)) {
const char *name = sys->GetEventTypeNameFromEnum(vr::EVREventType(ev.eventType));
switch (ev.eventType) {
case vr::VREvent_DashboardActivated:
case vr::VREvent_DashboardDeactivated:
case vr::VREvent_TrackedDeviceRoleChanged:
case vr::VREvent_TrackedDeviceActivated:
case vr::VREvent_TrackedDeviceDeactivated:
case vr::VREvent_ButtonPress:
case vr::VREvent_ButtonUnpress:
std::printf("%9.1f ms event %s device %u button %u\n", ms(), name, ev.trackedDeviceIndex,
ev.data.controller.button);
break;
default:
break;
}
}
vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
std::string roles;
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
if (sys->GetTrackedDeviceClass(i) != vr::TrackedDeviceClass_Controller) continue;
char type[64] = "";
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
if (!std::strcmp(type, "ft_pointer")) ours = i;
roles += " [" + Describe(sys, i) + " connected=" + std::to_string(sys->IsTrackedDeviceConnected(i)) + "]";
}
if (roles != lastRoles) {
std::printf("%9.1f ms controllers:%s\n", ms(), roles.c_str());
lastRoles = roles;
}
const auto primary = vr::VROverlay()->GetPrimaryDashboardDevice();
const std::string p = Describe(sys, primary);
if (p != lastPrimary) {
std::printf("%9.1f ms laser: %s%s\n", ms(), p.c_str(), primary == ours ? " (ours)" : "");
if (primary == ours && lostAt != Clock::time_point{}) {
std::printf("%9.1f ms back on ours after %.0f ms\n", ms(),
std::chrono::duration<double, std::milli>(Clock::now() - lostAt).count());
lostAt = {};
}
if (primary != ours && ours != vr::k_unTrackedDeviceIndexInvalid && primary != vr::k_unTrackedDeviceIndexInvalid)
lostAt = Clock::now();
noneSince = primary == vr::k_unTrackedDeviceIndexInvalid ? Clock::now() : Clock::time_point{};
lastPrimary = p;
}
if (snapback || reclaimMs > 0) {
const auto now = Clock::now();
if (pressed && now - pressedAt > std::chrono::milliseconds(40)) {
Send("btn a 0");
pressed = false;
}
if (snapback && !pressed && lostAt != Clock::time_point{} && now - pressedAt > std::chrono::milliseconds(100)) {
Send("btn a 1");
pressed = true;
pressedAt = now;
std::printf("%9.1f ms snapback: pressed our a\n", ms());
}
if (!pressed && reclaimMs > 0 && noneSince != Clock::time_point{} && ours != vr::k_unTrackedDeviceIndexInvalid &&
sys->IsTrackedDeviceConnected(ours) &&
now - noneSince > std::chrono::milliseconds(int(reclaimMs)) && now - pressedAt > std::chrono::milliseconds(100)) {
Send("btn a 1");
pressed = true;
pressedAt = now;
std::printf("%9.1f ms reclaim: pressed our a\n", ms());
}
}
std::fflush(stdout);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
if (pressed) Send("btn a 0");
vr::VR_Shutdown();
return 0;
}
-48
View File
@@ -1,48 +0,0 @@
// Read or change one SteamVR setting through vrserver (IVRSettings). Editing steamvr.vrsettings
// while SteamVR runs doesn't stick: vrserver writes its own copy back on the next change.
// Runs as a background OpenVR client (in the dev container).
// Usage: vrsetting get SECTION KEY
// vrsetting set-bool SECTION KEY true|false
// vrsetting remove SECTION KEY
#include <openvr.h>
#include <cstdio>
#include <cstring>
int main(int argc, char **argv) {
if (argc < 4 || (std::strcmp(argv[1], "set-bool") == 0 && argc < 5)) {
std::fprintf(stderr, "usage: %s get|set-bool|remove SECTION KEY [true|false]\n", argv[0]);
return 2;
}
const char *cmd = argv[1], *section = argv[2], *key = argv[3];
vr::EVRInitError err = vr::VRInitError_None;
vr::VR_Init(&err, vr::VRApplication_Background);
if (err != vr::VRInitError_None) {
std::printf("VR_Init failed: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
return 1;
}
vr::EVRSettingsError serr = vr::VRSettingsError_None;
int rc = 0;
if (!std::strcmp(cmd, "get")) {
char s[512] = "";
vr::VRSettings()->GetString(section, key, s, sizeof s, &serr);
if (serr == vr::VRSettingsError_None) std::printf("%s\n", s);
else {
const bool b = vr::VRSettings()->GetBool(section, key, &serr);
std::printf("%s\n", serr == vr::VRSettingsError_None ? (b ? "true" : "false") : "(unset)");
}
} else if (!std::strcmp(cmd, "set-bool")) {
vr::VRSettings()->SetBool(section, key, !std::strcmp(argv[4], "true"), &serr);
} else if (!std::strcmp(cmd, "remove")) {
vr::VRSettings()->RemoveKeyInSection(section, key, &serr);
} else {
std::fprintf(stderr, "unknown command %s\n", cmd);
rc = 2;
}
if (serr != vr::VRSettingsError_None) {
std::printf("error: %s\n", vr::VRSettings()->GetSettingsErrorNameFromEnum(serr));
rc = 1;
}
vr::VR_Shutdown();
return rc;
}
+2 -4
View File
@@ -13,10 +13,8 @@ case ${1:-status} in
"$root/scripts/sync.sh" >/dev/null "$root/scripts/sync.sh" >/dev/null
fill_template "$root/power/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit" fill_template "$root/power/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
"$frame" --host "set -e; pkill -x ft-powerd || true "$frame" --host "set -e; pkill -x ft-powerd || true
systemctl --user daemon-reload; systemctl --user enable --now $unit systemctl --user daemon-reload; systemctl --user enable $unit
# It comes up once SteamVR runs (it starts with it); distrobox enter takes a few seconds. $(start_with_steamvr $unit)" ;;
for i in \$(seq 20); do systemctl --user is-active --quiet $unit && break; sleep 1; done
echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 3" ;;
uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;; uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;; start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
status) "$frame" --host "systemctl --user is-active $unit; python3 -c 'import socket; s=socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM); s.bind(\"\"); s.settimeout(1); s.sendto(b\"status\", \"\\0ft_powerd\"); print(s.recv(256).decode())' 2>/dev/null || echo 'ft-powerd not answering'" ;; status) "$frame" --host "systemctl --user is-active $unit; python3 -c 'import socket; s=socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM); s.bind(\"\"); s.settimeout(1); s.sendto(b\"status\", \"\\0ft_powerd\"); print(s.recv(256).decode())' 2>/dev/null || echo 'ft-powerd not answering'" ;;
+20 -1
View File
@@ -58,6 +58,7 @@
#include <wlr/util/log.h> #include <wlr/util/log.h>
#include "vr.h" #include "vr.h"
#include "controller-click.h"
#define MAX_SCREENS 24 // screens and spare outputs #define MAX_SCREENS 24 // screens and spare outputs
@@ -104,6 +105,7 @@ struct server {
struct wl_list buffers; // tracked_buffer struct wl_list buffers; // tracked_buffer
struct wl_event_source *tick; struct wl_event_source *tick;
struct screen *pointer_focus; struct screen *pointer_focus;
struct ft_controller_click controller_click;
pid_t child; pid_t child;
// Where typing goes: the screens after a click on one, Steam after a click on another // Where typing goes: the screens after a click on one, Steam after a click on another
// panel. The input relay grabs the keyboards while it's the screens (see keys_update). // panel. The input relay grabs the keyboards while it's the screens (see keys_update).
@@ -292,6 +294,11 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
return; return;
} }
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return; if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
struct ft_event filtered = *e;
if (e->screen < s->n_config &&
!ft_controller_click_filter(&s->controller_click, &filtered, s->scale[e->screen])) return;
e = &filtered;
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
struct screen *sc = s->screens[e->screen]; struct screen *sc = s->screens[e->screen];
struct wlr_surface *surface = sc->toplevel->base->surface; struct wlr_surface *surface = sc->toplevel->base->surface;
const uint32_t t = now_ms(); const uint32_t t = now_ms();
@@ -514,7 +521,18 @@ static int control_readable(int fd, uint32_t mask, void *data) {
unsigned code; unsigned code;
int value, index, w, h; int value, index, w, h;
double scale; double scale;
if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) { char tail;
if (strcmp(buf, "controller-click?") == 0) {
snprintf(reply, sizeof reply,
"{\"supported\":true,\"threshold\":%.3f,\"held\":%s,\"dragging\":%s,\"suppressedMotions\":%lu,\"clicks\":%lu,\"drags\":%lu}",
s->controller_click.threshold, s->controller_click.held ? "true" : "false",
s->controller_click.dragging ? "true" : "false", s->controller_click.suppressed,
s->controller_click.clicks, s->controller_click.drags);
} else if (sscanf(buf, "controller-click %lf %c", &scale, &tail) == 1) {
if (!isfinite(scale) || scale < 0 || scale > 64 || s->controller_click.buttons)
snprintf(reply, sizeof reply, "error threshold or held controller button");
else { s->controller_click.threshold = scale; snprintf(reply, sizeof reply, "ok"); }
} else if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
// A new resolution for a screen, live: KWin resizes the screen to match. (KWin makes // A new resolution for a screen, live: KWin resizes the screen to match. (KWin makes
// it this size times its scale; ft-floatd sends spares' sizes divided by theirs.) // it this size times its scale; ft-floatd sends spares' sizes divided by theirs.)
const int min_w = index - 1 < s->n_config ? 320 : 64, min_h = index - 1 < s->n_config ? 200 : 64; const int min_w = index - 1 < s->n_config ? 320 : 64, min_h = index - 1 < s->n_config ? 200 : 64;
@@ -641,6 +659,7 @@ static bool setup_dmabuf(struct server *s) {
int main(int argc, char **argv) { int main(int argc, char **argv) {
struct server s = {0}; struct server s = {0};
s.controller_click.threshold = 8;
for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1; for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
s.kb_screen = -1; s.kb_screen = -1;
const char *socket_name = "ft-screens-0", *control_name = "ft_screens"; const char *socket_name = "ft-screens-0", *control_name = "ft_screens";
+46
View File
@@ -0,0 +1,46 @@
#ifndef FT_CONTROLLER_CLICK_H
#define FT_CONTROLLER_CLICK_H
#include <math.h>
#include <linux/input-event-codes.h>
#include "vr.h"
struct ft_controller_click {
bool held, dragging;
int screen;
uint32_t buttons;
double x, y, threshold, radius;
unsigned long suppressed, clicks, drags;
};
// Hold the desktop position at press time until movement exceeds a logical-pixel
// radius. No timer or delayed button-down. Only a hand controller's press starts
// it: the 3D mouse drives a laser too, and its short drags must stay drags.
static inline bool ft_controller_click_filter(struct ft_controller_click *c,
struct ft_event *e, double scale) {
if (e->type == FT_BUTTON && e->button >= BTN_LEFT && e->button < BTN_LEFT+8) {
uint32_t bit = 1u << (e->button-BTN_LEFT);
if (e->pressed) c->buttons |= bit; else c->buttons &= ~bit;
}
if (e->type == FT_BUTTON && e->pressed) {
if (e->button == BTN_LEFT && e->controller && !c->held && c->threshold > 0) {
c->held = true; c->dragging = false; c->screen = e->screen;
c->x = e->x; c->y = e->y; c->radius = c->threshold * scale;
} else if (e->button != BTN_LEFT) {
c->held = false; // Multi-button gestures retain their usual semantics.
}
} else if (e->type == FT_MOTION && c->held && !c->dragging) {
if (e->screen == c->screen && hypot(e->x-c->x, e->y-c->y) <= c->radius) {
++c->suppressed;
return false;
}
c->dragging = true; ++c->drags;
} else if (e->type == FT_BUTTON && e->button == BTN_LEFT && !e->pressed && c->held) {
if (!c->dragging) {
e->screen = c->screen; e->x = c->x; e->y = c->y;
++c->clicks;
}
c->held = false; c->dragging = false;
} else if (e->type == FT_LEAVE && c->held) {
return false; // Preserve the implicit grab through tiny edge excursions.
}
return true;
}
#endif
+42
View File
@@ -0,0 +1,42 @@
#include <assert.h>
#include "../controller-click.h"
static struct ft_event event(enum ft_event_type t, bool down, double x, double y) {
return (struct ft_event){.type=t,.screen=0,.button=BTN_LEFT,.pressed=down,.controller=true,.x=x,.y=y};
}
int main(void) {
struct ft_controller_click c={.threshold=8};
struct ft_event e=event(FT_BUTTON,true,100,100);
assert(ft_controller_click_filter(&c,&e,2));
e=event(FT_MOTION,false,112,108);
assert(!ft_controller_click_filter(&c,&e,2));
assert(c.held && !c.dragging && c.suppressed==1);
e=event(FT_BUTTON,false,116,111);
assert(ft_controller_click_filter(&c,&e,2));
assert(e.x==100 && e.y==100 && c.clicks==1 && !c.held);
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
e=event(FT_MOTION,false,109,100);assert(ft_controller_click_filter(&c,&e,1));
assert(c.dragging && c.drags==1);
e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1));
e=event(FT_BUTTON,false,103,100);assert(ft_controller_click_filter(&c,&e,1));
assert(e.x==103 && c.clicks==1 && !c.held && !c.dragging);
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
e=event(FT_LEAVE,false,0,0);assert(!ft_controller_click_filter(&c,&e,1));
e=event(FT_BUTTON,false,800,900);e.screen=1;ft_controller_click_filter(&c,&e,1);
assert(e.screen==0 && e.x==100 && c.clicks==2);
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
e=event(FT_MOTION,false,101,100);e.screen=1;
assert(ft_controller_click_filter(&c,&e,1) && c.dragging);
e=event(FT_BUTTON,false,101,100);ft_controller_click_filter(&c,&e,1);
c.threshold=0;
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1) && !c.held);
c.threshold=8;
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
e=event(FT_BUTTON,true,100,100);e.button=BTN_RIGHT;ft_controller_click_filter(&c,&e,1);
assert(!c.held);
// The 3D mouse's laser: a short drag stays a drag, and the release stays where it was.
e=event(FT_BUTTON,true,100,100);e.controller=false;assert(ft_controller_click_filter(&c,&e,1) && !c.held);
e=event(FT_MOTION,false,104,100);assert(ft_controller_click_filter(&c,&e,1));
e=event(FT_BUTTON,false,106,100);e.controller=false;
assert(ft_controller_click_filter(&c,&e,1) && e.x==106 && c.clicks==2 && c.suppressed==1);
}
+23 -2
View File
@@ -31,6 +31,9 @@
// - visibility modes: always (the hide hotkey toggles), only with the SteamVR dashboard // - visibility modes: always (the hide hotkey toggles), only with the SteamVR dashboard
// open, while you look at a chosen controller (the wrist gesture), or toggle only // open, while you look at a chosen controller (the wrist gesture), or toggle only
// (hidden until the hotkey shows them). // (hidden until the hotkey shows them).
// - a screen hidden on its own ("conceal <screen>", from ft-layout and profiles) stays
// hidden whatever the mode or the hotkey says, until "reveal <screen>". (Not "hide
// <screen>": an older build reads anything starting with "hide" as the hotkey's hide.)
// - controllers on the screens: while visible, the screens can keep SteamVR's laser mouse // - controllers on the screens: while visible, the screens can keep SteamVR's laser mouse
// on (VROverlayFlags_MakeOverlaysInteractiveIfVisible), so controllers use them with // on (VROverlayFlags_MakeOverlaysInteractiveIfVisible), so controllers use them with
// the dashboard closed. That also takes the controllers away from a VR game, so by // the dashboard closed. That also takes the controllers away from a VR game, so by
@@ -245,6 +248,7 @@ struct Screen {
double curve = 0; // cylinder radius in metres; 0 = flat double curve = 0; // cylinder radius in metres; 0 = flat
const void *shown = nullptr; // a frame arrived const void *shown = nullptr; // a frame arrived
bool visible = false; // shown in VR right now bool visible = false; // shown in VR right now
bool alone = false; // hidden on its own (conceal <screen>), whatever the mode
float alpha = 1; float alpha = 1;
vr::TrackedDeviceIndex_t pinned = kNone; // riding on this controller vr::TrackedDeviceIndex_t pinned = kNone; // riding on this controller
Mat pinRel = Identity(); // controller -> screen Mat pinRel = Identity(); // controller -> screen
@@ -788,7 +792,7 @@ void UpdateVisibility() {
Mat head; Mat head;
const bool haveHead = DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head); const bool haveHead = DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head);
for (auto &[i, s] : g_screens) { for (auto &[i, s] : g_screens) {
bool visible = s.shown && (shared || s.drag != Drag::None); bool visible = s.shown && (shared || s.drag != Drag::None) && !s.alone;
// A floating window's panel: while a window floats on it, its output is on, and the // A floating window's panel: while a window floats on it, its output is on, and the
// window isn't minimized (and once it has a crop). // window isn't minimized (and once it has a crop).
if (s.floating) visible = visible && s.floatOn && s.outputOn && !s.minimized && s.cropW > 0; if (s.floating) visible = visible && s.floatOn && s.outputOn && !s.minimized && s.cropW > 0;
@@ -886,6 +890,7 @@ void SendFloat(const std::string &msg) {
std::memcpy(addr.sun_path + 1, name, sizeof name - 1); std::memcpy(addr.sun_path + 1, name, sizeof name - 1);
sendto(fd, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<sockaddr *>(&addr), sendto(fd, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<sockaddr *>(&addr),
socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1)); socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1));
if (msg.rfind("resize ", 0) != 0) // an edge drag sends many resizes a second
std::printf("to ft-floatd: %s\n", msg.c_str()); std::printf("to ft-floatd: %s\n", msg.c_str());
} }
@@ -1325,7 +1330,8 @@ void UpdateCatcher() {
for (int k = 0; k < 3; ++k) params.vSource.v[k] = l.m[k][3], params.vDirection.v[k] = -l.m[k][2]; for (int k = 0; k < 3; ++k) params.vSource.v[k] = l.m[k][3], params.vDirection.v[k] = -l.m[k][2];
for (auto &[i, s] : g_screens) { for (auto &[i, s] : g_screens) {
if (!s.visible) continue; if (!s.visible) continue;
std::vector<vr::VROverlayHandle_t> parts(s.All().begin(), s.All().end()); const auto all = s.All(); // one copy: two calls give two temporaries, not one range
std::vector<vr::VROverlayHandle_t> parts(all.begin(), all.end());
for (const auto &[k, sub] : s.subs) parts.push_back(sub.overlay); for (const auto &[k, sub] : s.subs) parts.push_back(sub.overlay);
for (auto o : parts) { for (auto o : parts) {
vr::VROverlayIntersectionResults_t hit; vr::VROverlayIntersectionResults_t hit;
@@ -1736,6 +1742,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
e.type = FT_BUTTON; e.type = FT_BUTTON;
e.button = LinuxButton(ev.data.mouse.button); e.button = LinuxButton(ev.data.mouse.button);
e.pressed = ev.eventType == vr::VREvent_MouseButtonDown; e.pressed = ev.eventType == vr::VREvent_MouseButtonDown;
e.controller = IsHandController(ev.trackedDeviceIndex);
at(); at();
if (!e.pressed && s.titleCarry) e.x = s.carryX, e.y = s.carryY, s.titleCarry = false; if (!e.pressed && s.titleCarry) e.x = s.carryX, e.y = s.carryY, s.titleCarry = false;
if (e.pressed) { if (e.pressed) {
@@ -1944,6 +1951,8 @@ void ft_vr_keyboard_hide(void) {
// wrist <degrees> a pinned screen shows while you see its front within this // wrist <degrees> a pinned screen shows while you see its front within this
// gesture <left|right> <degrees> the gesture mode: look within this of that controller // gesture <left|right> <degrees> the gesture mode: look within this of that controller
// hide | show | toggle the manual switch (see g_manual) // hide | show | toggle the manual switch (see g_manual)
// conceal <screen|all> | reveal <screen|all> a screen hidden on its own, whatever the mode
// concealed -> "ok [<screen> ...]" the screens hidden on their own
// controllers always|outside_games|dashboard when controllers' lasers work the screens // controllers always|outside_games|dashboard when controllers' lasers work the screens
// ingames hide|visible during a VR game, "always" acts like "only with the dashboard" // ingames hide|visible during a VR game, "always" acts like "only with the dashboard"
// (hide), or stays as it is (visible) // (hide), or stays as it is (visible)
@@ -2076,6 +2085,18 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
g_gestureHand = std::strcmp(hand, "right") == 0 ? "right" : "left"; g_gestureHand = std::strcmp(hand, "right") == 0 ? "right" : "left";
g_gestureAngle = std::clamp(w, 5.0, 90.0); g_gestureAngle = std::clamp(w, 5.0, 90.0);
std::snprintf(reply, size, "ok"); std::snprintf(reply, size, "ok");
} else if (std::strncmp(cmd, "concealed", 9) == 0) {
int len = std::snprintf(reply, size, "ok");
for (auto &[i, s] : g_screens)
if (len < size && !s.floating && s.alone) len += std::snprintf(reply + len, size - len, " %d", i + 1);
} else if (std::sscanf(cmd, "conceal %15s", word) == 1 || std::sscanf(cmd, "reveal %15s", word) == 1) {
const bool hide = cmd[0] == 'c';
const Screen *one = std::strcmp(word, "all") ? Find(std::atoi(word)) : nullptr;
if (std::strcmp(word, "all") && (!one || one->floating))
return (void)std::snprintf(reply, size, "error no screen %s", word);
each(word, [&](Screen &s) { s.alone = hide; });
UpdateVisibility();
std::snprintf(reply, size, "ok");
} else if (!std::strncmp(cmd, "hide", 4) || !std::strncmp(cmd, "show", 4) || !std::strncmp(cmd, "toggle", 6)) { } else if (!std::strncmp(cmd, "hide", 4) || !std::strncmp(cmd, "show", 4) || !std::strncmp(cmd, "toggle", 6)) {
const bool always = EffectiveMode() == Mode::Always; const bool always = EffectiveMode() == Mode::Always;
const bool shownNow = always ? !g_manual : g_manual; const bool shownNow = always ? !g_manual : g_manual;
+1
View File
@@ -25,6 +25,7 @@ struct ft_event {
double x, y; // FT_MOTION: buffer pixels from the top left double x, y; // FT_MOTION: buffer pixels from the top left
uint32_t button; // FT_BUTTON: linux BTN_* uint32_t button; // FT_BUTTON: linux BTN_*
bool pressed; bool pressed;
bool controller; // FT_BUTTON: from a hand controller's laser (not the 3D mouse's)
double dx, dy; // FT_SCROLL: notches (positive dy: scroll down) double dx, dy; // FT_SCROLL: notches (positive dy: scroll down)
uint32_t key; // FT_KEY: linux KEY_* from our keyboard (pressed: down or up) uint32_t key; // FT_KEY: linux KEY_* from our keyboard (pressed: down or up)
}; };
+44
View File
@@ -58,3 +58,47 @@ on_frame_script() {
fill_template() { fill_template() {
sed "s|@REPO@|$FRAME_REPO|g" "$1" sed "s|@REPO@|$FRAME_REPO|g" "$1"
} }
# frame_sudo '<command>': run a shell command as root on the Frame host. sudo asks for the
# password in this terminal: on the Frame directly (it reads the terminal itself, so this
# works when stdin isn't one, as in install.sh's steps), or from a PC through ssh -t.
# SUDO_ASKPASS on the Frame, or steamos_root_pwd in the repo's .env (sent to sudo -S on
# stdin, never on a command line) answer it with no terminal; from a PC, .env comes first.
frame_sudo() {
local tty=0 pw
{ : </dev/tty; } 2>/dev/null && tty=1
if [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; then
sudo -A bash -c "$1"
return
fi
if [ "$FRAME_LOCAL" = 1 ] && [ "$tty" = 1 ]; then
sudo bash -c "$1"
return
fi
pw=$(sed -n 's/^steamos_root_pwd=//p' "$REPO_ROOT/.env" 2>/dev/null)
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
if [ -n "$pw" ]; then
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
elif [ "$FRAME_LOCAL" = 0 ] && [ "$tty" = 1 ]; then
ssh -tt -o BatchMode=yes "$FRAME_HOST" "cd $(printf %q "$FRAME_REPO") && sudo bash -c $(printf %q "$1")" </dev/tty
else
echo "sudo needs a terminal for the password, or steamos_root_pwd in $REPO_ROOT/.env" >&2
return 1
fi
}
# start_with_steamvr UNIT: a host command for an installer. Units that need SteamVR
# (Requisite=steamvr.service) can't start without it, so with SteamVR off (an install over
# SSH, the headset asleep) they're left to start with it. With SteamVR on, the unit restarts,
# so a re-install runs the new code, and the command waits for it and shows its last lines.
start_with_steamvr() {
local unit=$1
printf '%s' "if systemctl --user is-active --quiet steamvr.service; then
systemctl --user restart $unit
# distrobox enter takes a few seconds.
for i in \$(seq 20); do systemctl --user is-active --quiet $unit && break; sleep 1; done
echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 3
else
echo '$unit: enabled; SteamVR is off, so it starts with SteamVR'
fi"
}
+18 -3
View File
@@ -6,7 +6,22 @@
# `distrobox enter`. # `distrobox enter`.
box=${FRAME_BOX:-dev} box=${FRAME_BOX:-dev}
export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)} export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
[ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ] && exit 0 running() { [ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ]; }
running && exit 0
podman container exists "$box" 2>/dev/null || exit 0 # not created yet: setup/dev-container.sh does that podman container exists "$box" 2>/dev/null || exit 0 # not created yet: setup/dev-container.sh does that
exec systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \ since=$(date -u +%FT%T)
podman start "$box" >/dev/null systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \
podman start "$box" >/dev/null || exit
# Every start runs distrobox-init in the container, and `distrobox enter` only waits for it
# when it starts the container itself. The first start installs what distrobox needs and sets
# up passwordless sudo, which takes a minute or more; until then sudo in the container asks
# for a password (issue #9). Later starts take a few seconds.
for i in $(seq 600); do
podman logs --since "$since" "$box" 2>&1 | grep -q '^container_setup_done' && exit 0
running || break
[ "$i" = 10 ] && echo "setting up the $box container (the first start takes a few minutes)" >&2
sleep 1
done
echo "the $box container didn't finish starting; see: podman logs $box" >&2
exit 1
+13 -1
View File
@@ -1,6 +1,11 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Check that the Steam Frame is ready for these projects: reachable (from a PC), the # Check that the Steam Frame is ready for these projects: reachable (from a PC), the
# distrobox tool and the dev container present, and disk space. Works on the Frame too. # distrobox tool and the dev container present, and disk space. Then check what Frametop
# needs from SteamOS, which an update can change (scripts/update-check.py). Works on the
# Frame too.
# Usage: scripts/doctor.sh [--mark-good]
# --mark-good once Frametop works, record the SteamOS, SteamVR, and KWin versions, so a
# later run says what an update changed
set -uo pipefail set -uo pipefail
. "$(dirname "${BASH_SOURCE[0]}")/_env.sh" . "$(dirname "${BASH_SOURCE[0]}")/_env.sh"
@@ -31,4 +36,11 @@ check "distrobox" on_frame 'test -x ~/.local/bin/distrobox && ~/.local/bin/distr
check "container $FRAME_BOX" on_frame "podman ps -a --filter name=^$FRAME_BOX\$ --format '{{.Image}} {{.Status}}' | grep ." check "container $FRAME_BOX" on_frame "podman ps -a --filter name=^$FRAME_BOX\$ --format '{{.Image}} {{.Status}}' | grep ."
check "repo on the Frame" on_frame 'pwd' check "repo on the Frame" on_frame 'pwd'
check "free space in ~" on_frame "df -h ~ | awk 'NR==2{print \$4\" free\"}'" check "free space in ~" on_frame "df -h ~ | awk 'NR==2{print \$4\" free\"}'"
echo "what Frametop needs from SteamOS:"
if [ "$FRAME_LOCAL" = 1 ]; then
python3 "$REPO_ROOT/scripts/update-check.py" "$@" || fail=1
else
ssh -o BatchMode=yes "$FRAME_HOST" "python3 - ${*:+$(printf '%q ' "$@")}" < "$REPO_ROOT/scripts/update-check.py" || fail=1
fi
exit "$fail" exit "$fail"
+1 -1
View File
@@ -112,7 +112,7 @@ def main():
try: try:
with open(os.path.join(HOME, ".config/frametop.conf")) as f: with open(os.path.join(HOME, ".config/frametop.conf")) as f:
conf = [ln.split("#")[0].strip() for ln in f] conf = [ln.split("#")[0].strip() for ln in f]
out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|META_DASHBOARD|SHARE_KEYS|BACKEND)=", c))) out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|SHARE_KEYS|BACKEND)=", c)))
except OSError: except OSError:
out("Settings: no ~/.config/frametop.conf") out("Settings: no ~/.config/frametop.conf")
out("ft-screens state (visibility, manual, wrist, gesture, lasers, game running, in games): " out("ft-screens state (visibility, manual, wrist, gesture, lasers, game running, in games): "
+3
View File
@@ -27,6 +27,9 @@ done
echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)" echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p' LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
section "What Frametop needs from SteamOS (scripts/update-check.py)"
python3 "$repo/scripts/update-check.py" 2>&1 || true
section Settings section Settings
grep -v '^\s*#' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//' | grep . || echo "no ~/.config/frametop.conf" grep -v '^\s*#' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//' | grep . || echo "no ~/.config/frametop.conf"
python3 - <<'PY' 2>/dev/null || echo "no ~/.config/frametop-layout.json" python3 - <<'PY' 2>/dev/null || echo "no ~/.config/frametop-layout.json"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
"$root/scripts/frame.sh" -C screens 'mkdir -p build; gcc -std=c11 -Wall -Wextra -Werror tests/controller-click-test.c -lm -o build/controller-click-test && build/controller-click-test'
+414
View File
@@ -0,0 +1,414 @@
#!/usr/bin/env python3
"""Check what Frametop needs from SteamOS, for after a SteamOS update.
On the Frame, a SteamOS update replaces SteamVR, KWin, gamescope, and the kernel along with
the rest of the OS image. Frametop lives in the home folder and survives it, but it uses
SteamVR and KWin interfaces that an update can change or drop, some of them undocumented.
This checks each one, and compares the versions with the last ones recorded as working.
scripts/update-check.py # check (scripts/doctor.sh runs this)
scripts/update-check.py --mark-good # check, then record these versions as working
From a PC: ssh frame python3 - [--mark-good] < scripts/update-check.py
It only reads. It connects to SteamVR as a background app (which never starts SteamVR),
runs `vrcmd --overlays` as the pointer helper does, and maps the eye tracker's shared
memory read-only. It never starts, stops, or restarts anything.
"""
import json
import math
import mmap
import os
import platform
import re
import struct
import subprocess
import sys
import time
HOME = os.path.expanduser("~")
UID = os.getuid()
KNOWN_GOOD = os.path.join(HOME, ".local/state/frametop/known-good.json")
STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64"
LAUNCHER = os.path.join(HOME, ".local/share/applications/deckard-nested-desktop.desktop")
VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath")
BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness" # as in power/ft-powerd.cpp
EYE_MMAP = "/dev/shm/eye-server.mmap"
HOST_GLIBC = (2, 39) # the newest the pointer driver may need (pointer/driver/build.sh)
# Packages in the OS image that Frametop depends on, and what to try by hand when one changes.
PACKAGES = {
"deckard-steamvr-rel": "the 3D mouse on the dashboard, on SteamVR Settings, and on a SteamVR "
"window's grab bar; picking up a controller; mapped controller buttons; gaze",
"kwin": "clicks near the far edge of a screen whose scale isn't 1; every screen comes back "
"after a desktop restart; floating a window",
"plasma-workspace": "the taskbar and panels after a desktop restart",
"gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked",
"bluez": "a Bluetooth mouse reconnecting after it sleeps",
}
KERNEL_HINT = "display power (ft-powerd) and hand tracking"
# Frametop's user services, and the control socket each binds once it's up.
UNITS = {
"frametop-input-relay": "@frametop_relay",
"frametop-pointer": "@ft_pointer_helper",
"frametop-gaze": "@ft_gazed",
"frametop-power": "@ft_powerd",
"frametop-camd": None,
"frametop-hands": None,
}
# eye-server.mmap offsets, the same as gaze/ft-gaze.cpp's (packed, little-endian).
EYE_COUNTER, EYE_TIME, EYE_OPEN, EYE_NEED = 0x38, 0x157, 0x1CB, 0x1D3
EYE_VECTORS = (0x15F, 0x16B, 0x19B, 0x1A7) # set 1 left, right; set 2 left, right
# Runs in a child process, so a SteamVR that hangs can't hang the check.
OPENVR_PROBE = r"""
import ctypes, sys
lib = ctypes.CDLL(sys.argv[1])
lib.VR_InitInternal2.restype = ctypes.c_uint32
lib.VR_InitInternal2.argtypes = [ctypes.POINTER(ctypes.c_int), ctypes.c_int, ctypes.c_char_p]
lib.VR_GetVRInitErrorAsSymbol.restype = ctypes.c_char_p
lib.VR_IsInterfaceVersionValid.restype = ctypes.c_bool
lib.VR_IsInterfaceVersionValid.argtypes = [ctypes.c_char_p]
err = ctypes.c_int(0)
lib.VR_InitInternal2(ctypes.byref(err), 3, None) # VRApplication_Background
if err.value:
print("init", lib.VR_GetVRInitErrorAsSymbol(err.value).decode())
sys.exit(1)
for name in sys.argv[2:]:
print(name, int(lib.VR_IsInterfaceVersionValid(name.encode())))
lib.VR_ShutdownInternal()
"""
failed = False
def report(state, label, detail=""):
global failed
failed = failed or state == "FAIL"
print(f"{state:<6}{label}{': ' + detail if detail else ''}", flush=True)
def run(*cmd, timeout=15, env=None):
try:
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=env)
return p.returncode, p.stdout
except (OSError, subprocess.TimeoutExpired) as e:
return -1, str(e)
def systemctl(*args):
return run("systemctl", "--user", *args)[1].strip()
def unix_sockets():
with open("/proc/net/unix") as f:
return {parts[7] for parts in map(str.split, f) if len(parts) > 7}
def is_elf(path):
try:
with open(path, "rb") as f:
return f.read(4) == b"\x7fELF"
except OSError:
return False
# --- versions
def current_versions():
versions = {}
with open("/etc/os-release") as f:
for line in f:
if line.startswith("BUILD_ID="):
versions["SteamOS build"] = line.split("=", 1)[1].strip().strip('"')
for line in run("pacman", "-Q", *PACKAGES)[1].splitlines():
name, _, version = line.partition(" ")
if name in PACKAGES:
versions[name] = version
versions["kernel"] = platform.release()
return versions
def check_versions(versions):
try:
with open(KNOWN_GOOD) as f:
good = json.load(f)
date, old = good["date"], good["versions"]
except (OSError, ValueError, KeyError, TypeError):
report("warn", "known-good versions", "none recorded yet; run scripts/doctor.sh --mark-good "
"once Frametop works")
return
changed = [k for k in sorted(set(versions) | set(old)) if old.get(k) != versions.get(k)]
if not changed:
report("ok", "versions", f"the same as when marked good ({date})")
for k in changed:
hint = PACKAGES.get(k) or (KERNEL_HINT if k == "kernel" else "")
report("warn", f"{k} changed since {date}",
f"{old.get(k, 'none')} -> {versions.get(k, 'none')}" + (f"; try {hint}" if hint else ""))
# --- the host, without SteamVR
def check_host():
needed = [
("/usr/bin/kwin_wayland_wrapper", "FAIL", "the desktop can't start KWin"),
("/usr/bin/startplasma-wayland", "FAIL", "the desktop can't start Plasma"),
("/usr/bin/dbus-run-session", "FAIL", "the desktop can't start its session bus"),
(f"{STEAMVR_BIN}/vrcmd", "FAIL", "the 3D mouse can't find panels"),
(f"{STEAMVR_BIN}/vrpathreg", "warn", "the pointer driver can't be installed or removed"),
("/usr/share/deckard/mesavars.sh", "warn", "the desktop starts without SteamOS's Mesa settings"),
("/etc/profile.d/flatpak.sh", "warn", "Flatpak apps may open Discover instead of starting"),
("/usr/share/applications/deckard-nested-desktop.desktop", "warn",
"SteamOS's Desktop launcher entry is gone or renamed, so Frametop's copy may not replace it"),
]
missing = [n for n in needed if not os.path.exists(n[0])]
for path, state, effect in missing:
report(state, f"{path} missing", effect)
if not missing:
report("ok", "host files", f"all {len(needed)} the desktop and pointer use are there")
try:
with open("/usr/bin/kwin_wayland", "rb") as f:
# Qt keeps the option name as a UTF-16 string literal.
output_count = "output-count".encode("utf-16-le") in f.read()
except OSError:
output_count = False
if output_count:
report("ok", "KWin", "has --output-count (one output per screen)")
else:
report("FAIL", "KWin", "no --output-count option: the desktop gets one screen at most")
if systemctl("cat", "steamvr.service"):
report("ok", "steamvr.service", "Frametop's services start and stop with it")
else:
report("FAIL", "steamvr.service", "gone or renamed; Frametop's services are ordered on it")
glibc = os.confstr("CS_GNU_LIBC_VERSION") or ""
have = tuple(int(x) for x in re.findall(r"\d+", glibc)[:2])
want = ".".join(map(str, HOST_GLIBC))
if have >= HOST_GLIBC:
report("ok", "glibc", f"{glibc}, the pointer driver needs {want}")
else:
report("FAIL", "glibc", f"{glibc}, older than the {want} the pointer driver needs")
try:
with open(VRPATHS) as f:
drivers = json.load(f).get("external_drivers") or []
except (OSError, ValueError):
drivers = []
driver = next((d for d in drivers if os.path.basename(d.rstrip("/")) == "ft_pointer"), None)
if driver and os.path.isfile(os.path.join(driver, "bin/linuxarm64/driver_ft_pointer.so")):
report("ok", "pointer driver", f"registered with SteamVR ({driver})")
else:
report("FAIL", "pointer driver", "not registered with SteamVR; run pointer/driver/install.sh install, "
"then restart SteamVR")
session = launcher_session()
if session is None:
report("warn", "launcher", "Launch a program -> Desktop starts the stock desktop "
"(./desktops.sh install brings Frametop back)")
elif os.path.isfile(session):
report("ok", "launcher", f"Desktop starts {session}")
else:
report("FAIL", "launcher", f"Desktop starts {session}, which doesn't exist")
if systemctl("is-enabled", "frametop-power") == "enabled":
if os.access(BACKLIGHT, os.W_OK):
report("ok", "backlight", "ft-powerd can turn the displays off")
else:
report("FAIL", "backlight", f"{BACKLIGHT} isn't writable, so ft-powerd can't turn the displays off")
def launcher_session():
try:
with open(LAUNCHER) as f:
m = re.search(r"^Exec=(\S+)", f.read(), re.M)
except OSError:
return None
return m.group(1) if m else None
# --- with SteamVR running
def installed_binaries():
"""The programs the services and the launcher run: {path: what runs it}."""
found = {}
for unit in UNITS:
argv = re.search(r"argv\[\]=([^;]*)", systemctl("show", "-p", "ExecStart", "--value", unit + ".service"))
for arg in (argv.group(1).split() if argv else []):
if not arg.startswith(HOME) or not os.path.isfile(arg):
continue
if is_elf(arg):
found[arg] = unit
else:
# A script runs the programs built next to it (ft-gazed runs build/ft-gaze).
build = os.path.join(os.path.dirname(arg), "build")
for name in sorted(os.listdir(build)) if os.path.isdir(build) else []:
if is_elf(os.path.join(build, name)):
found[os.path.join(build, name)] = unit
session = launcher_session()
if session:
screens = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(session)),
"../screens/build/ft-screens"))
if os.path.isfile(screens):
found[screens] = "the desktop"
return found
def check_openvr():
needs = {} # interface version: programs built against it
for path in installed_binaries():
with open(path, "rb") as f:
for name in set(re.findall(rb"IVR[A-Za-z]+_\d{3}", f.read())):
needs.setdefault(name.decode(), []).append(os.path.basename(path))
if not needs:
report("skip", "OpenVR interfaces", "no installed Frametop programs found")
return
code, out = run(sys.executable, "-c", OPENVR_PROBE, f"{STEAMVR_BIN}/libopenvr_api.so", *sorted(needs),
timeout=30)
served = dict(line.split() for line in out.splitlines() if len(line.split()) == 2)
if code != 0 or "init" in served:
why = served.get("init") or out.strip() or f"exit {code}"
report("FAIL", "OpenVR", f"can't connect to SteamVR as a background app ({why})")
return
gone = [n for n in sorted(needs) if served.get(n) != "1"]
for name in gone:
report("FAIL", f"OpenVR {name}", f"this SteamVR doesn't serve it; rebuild {', '.join(sorted(needs[name]))} "
"against a newer OpenVR header")
if not gone:
programs = sorted({p for ps in needs.values() for p in ps})
report("ok", "OpenVR interfaces", f"all {len(needs)} that {', '.join(programs)} use are served")
def check_overlays(desktop_up):
code, out = run(f"{STEAMVR_BIN}/vrcmd", "--overlays",
env=dict(os.environ, LD_LIBRARY_PATH=STEAMVR_BIN))
# The format the pointer helper (pointer.h) and ft_layout.py parse: 'key' -- 'name', WxH visible VROverlayType_...
keys = re.findall(r"^'([^']+)' -- '.*VROverlayType_", out, re.M)
if not keys:
report("FAIL", "vrcmd --overlays", "lists no overlays in the expected format, so the 3D mouse can't "
"find panels (pointer/helper/pointer.h and layout/ft_layout.py parse it)")
elif desktop_up and not any(re.fullmatch(r"frametop\.screen\.\d+", k) for k in keys):
report("FAIL", "vrcmd --overlays", f"lists {len(keys)} overlays, but none of the desktop's screens")
else:
report("ok", "vrcmd --overlays", f"{len(keys)} overlays in the format the pointer parses")
def check_services(sockets, desktop_up):
for unit, socket in UNITS.items():
if systemctl("is-enabled", unit) != "enabled":
continue
state = systemctl("is-active", unit)
if state != "active":
report("FAIL", unit, f"{state}; see journalctl --user -u {unit}")
elif socket and socket not in sockets:
report("FAIL", unit, f"runs, but hasn't opened {socket}")
else:
report("ok", unit, "running")
if "@ft_pointer" in sockets:
report("ok", "ft_pointer driver", "SteamVR loaded it")
else:
report("FAIL", "ft_pointer driver", "SteamVR didn't load it; see pointer/driver/install.sh log")
if not desktop_up:
report("skip", "desktop", "not running; start it and check again to cover the screens")
elif "@ft_screens" in sockets:
report("ok", "desktop", "ft-screens is running")
else:
report("FAIL", "desktop", "ft-screens runs, but hasn't opened @ft_screens")
if systemctl("is-enabled", "frametop-camd") == "enabled" and run("pgrep", "-x", "XRService")[0] != 0:
report("FAIL", "XRService", "not running, so hand tracking has no cameras")
def check_steam_ui():
"""Steam's UI calls the Steam menu shortcut uses (steam/ft-steam). They're Steam client
internals, so a Steam update can change them too, not just a SteamOS one."""
# Run from stdin (over SSH) there's no __file__: try the usual checkouts.
repos = [os.path.dirname(os.path.dirname(os.path.abspath(__file__)))] if "__file__" in globals() else []
repos += [os.path.join(HOME, "frametop"), os.path.join(HOME, "dev/frametop")]
tool = next((t for t in (os.path.join(r, "steam/ft-steam") for r in repos) if os.path.exists(t)), None)
if not tool:
report("skip", "Steam menu shortcut", "steam/ft-steam isn't in a checkout here")
return
try:
p = subprocess.run([tool, "check"], capture_output=True, text=True, timeout=15)
except (OSError, subprocess.TimeoutExpired) as e:
report("warn", "Steam menu shortcut", f"ft-steam check didn't run ({e})")
return
detail = (p.stdout + p.stderr).strip()
if p.returncode == 0:
report("ok", "Steam menu shortcut", "Steam's UI still has the calls steam/ft-steam makes")
elif detail.startswith("ft-steam:"):
report("skip", "Steam menu shortcut", detail[len("ft-steam:"):].strip())
else:
report("warn", "Steam menu shortcut", f"{detail}; a Meta tap won't open the Steam menu (steam/ft-steam)")
def check_eye_tracker():
gaze = systemctl("is-enabled", "frametop-gaze") == "enabled"
try:
with open(EYE_MMAP, "rb") as f:
m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
except (OSError, ValueError):
report("FAIL" if gaze else "skip", "eye tracker", f"{EYE_MMAP} isn't there")
return
if len(m) < EYE_NEED:
report("FAIL", "eye tracker", f"{EYE_MMAP} is {len(m)} bytes, smaller than gaze/ft-gaze.cpp reads")
return
first = struct.unpack_from("<I", m, EYE_COUNTER)[0]
time.sleep(0.3)
if struct.unpack_from("<I", m, EYE_COUNTER)[0] == first:
report("skip", "eye tracker", "idle, so its layout wasn't checked")
return
age = time.clock_gettime(time.CLOCK_MONOTONIC_RAW) - struct.unpack_from("<d", m, EYE_TIME)[0]
units = sum(abs(math.sqrt(sum(x * x for x in struct.unpack_from("<3f", m, o))) - 1) < 0.02
for o in EYE_VECTORS)
# A lost eye can zero its vector, so two of the four are enough. The timestamp is the
# strong check: a fresh CLOCK_MONOTONIC_RAW double doesn't land on that offset by chance.
if abs(age) < 1 and units >= 2:
report("ok", "eye tracker", "eye-server.mmap still has the layout gaze/ft-gaze.cpp reads")
else:
report("FAIL" if gaze else "warn", "eye tracker", f"eye-server.mmap layout changed (sample age "
f"{age:.3g} s, {units} of 4 gaze vectors unit length); update the offsets in gaze/ft-gaze.cpp")
def main():
args = sys.argv[1:]
if args not in ([], ["--mark-good"]):
sys.exit("usage: update-check.py [--mark-good]")
# A terminal in the desktop has its session's runtime dir and bus; systemctl needs the real ones.
os.environ["XDG_RUNTIME_DIR"] = f"/run/user/{UID}"
os.environ["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path=/run/user/{UID}/bus"
versions = current_versions()
check_versions(versions)
check_host()
steamvr_up = run("pgrep", "-x", "vrserver")[0] == 0
if not steamvr_up:
report("skip", "SteamVR checks", "SteamVR isn't running")
else:
sockets = unix_sockets()
desktop_up = run("pgrep", "-x", "ft-screens")[0] == 0
check_openvr()
check_overlays(desktop_up)
check_services(sockets, desktop_up)
check_steam_ui()
check_eye_tracker()
if args == ["--mark-good"]:
if failed:
print("Not recording these versions as working: fix the failures first.")
elif not steamvr_up:
print("Not recording these versions as working: start SteamVR, so its checks run too.")
sys.exit(1)
else:
os.makedirs(os.path.dirname(KNOWN_GOOD), exist_ok=True)
with open(KNOWN_GOOD, "w") as f:
json.dump({"date": time.strftime("%Y-%m-%d"), "versions": versions}, f, indent=2)
print(f"Recorded these versions as working in {KNOWN_GOOD}.")
sys.exit(1 if failed else 0)
if __name__ == "__main__":
main()
+45 -3
View File
@@ -64,15 +64,26 @@ if [ "${1:-}" != --inner ]; then
echo "Frametop is already running" >&2 echo "Frametop is already running" >&2
exit 0 exit 0
fi fi
# A SteamOS update could move either of the files sourced here. The desktop still
# starts without them (scripts/update-check.py reports it).
if [ -r /usr/share/deckard/mesavars.sh ]; then
set -a; . /usr/share/deckard/mesavars.sh; set +a set -a; . /usr/share/deckard/mesavars.sh; set +a
else
echo "frametop: no /usr/share/deckard/mesavars.sh, starting without SteamOS's Mesa settings" >&2
fi
# Flatpak apps (Chromium) publish their launcher entries under the Flatpak # Flatpak apps (Chromium) publish their launcher entries under the Flatpak
# exports dirs. SSH and launcher environments may lack XDG_DATA_DIRS, and then # exports dirs. SSH and launcher environments may lack XDG_DATA_DIRS, and then
# Plasma can't find them and opens Discover instead. # Plasma can't find them and opens Discover instead.
export XDG_DATA_DIRS=${XDG_DATA_DIRS:-/usr/local/share:/usr/share} export XDG_DATA_DIRS=${XDG_DATA_DIRS:-/usr/local/share:/usr/share}
if [ -r /etc/profile.d/flatpak.sh ]; then
set +u; . /etc/profile.d/flatpak.sh; set -u set +u; . /etc/profile.d/flatpak.sh; set -u
# Arrange the screens in the saved layout once they're up (layout; skipped else
# when auto-arrange is off). echo "frametop: no /etc/profile.d/flatpak.sh, so Flatpak apps may open Discover instead" >&2
setsid "$here/../layout/ft-layout" apply --wait 90 > /tmp/frametop-layout.log 2>&1 < /dev/null & fi
# Arrange the screens once they're up: in the profile this desktop starts with (FT_PROFILE,
# from a profile's launcher entry, or the default profile), which also opens its apps, or
# else in the saved layout (skipped when auto-arrange is off). docs/profiles.md.
setsid "$here/../layout/ft-layout" start --wait 90 > /tmp/frametop-layout.log 2>&1 < /dev/null &
if [ "$backend" = gamescope ]; then if [ "$backend" = gamescope ]; then
export ENABLE_GAMESCOPE_WSI=1 GAMESCOPE_MANGOAPP_SOCKET_DISABLE=1 export ENABLE_GAMESCOPE_WSI=1 GAMESCOPE_MANGOAPP_SOCKET_DISABLE=1
@@ -196,4 +207,35 @@ else
rm -f "$autostart" rm -f "$autostart"
fi fi
# Launch as Standalone in every app's right-click menu (float/ft_apps.py): copies of the
# apps' desktop files with that action, first in XDG_DATA_DIRS, so only this desktop sees
# them. Written now, before Plasma reads them; ft-floatd keeps them up to date.
if [ "$float_slots" -gt 0 ]; then
python3 "$here/../float/ft_apps.py" >/dev/null 2>&1 || true
export XDG_DATA_DIRS=$HOME/.local/share/frametop/apps:${XDG_DATA_DIRS:-/usr/local/share:/usr/share}
fi
# With floating windows, the session's windows get Frametop's own decoration: Breeze's look
# plus a float button left of Close (decoration/, a QML decoration KWin's Aurorae engine
# loads; docs/floating-windows.md). It's copied, not linked: KPackage doesn't list linked
# packages. Desktop Mode keeps its own kwinrc, so it keeps Breeze.
deco=kwin4_decoration_qml_frametop
deco_dir=${XDG_DATA_HOME:-$HOME/.local/share}/kwin/decorations/$deco
kwinrc=$XDG_CONFIG_HOME/kwinrc
if [ "$float_slots" -gt 0 ]; then
rm -rf "$deco_dir" "$deco_dir"_try* # (decoration/apply.sh's copies)
mkdir -p "$(dirname "$deco_dir")"
cp -r "$here/../decoration" "$deco_dir"
rm -f "$deco_dir/apply.sh"
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library org.kde.kwin.aurorae
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme "$deco"
elif [ "$(kreadconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme)" = "$deco" ]; then
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library --delete
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete
fi
# Profiles reopen apps (docs/profiles.md), so Plasma's own session restore stays off here;
# with both, apps would open twice.
kwriteconfig6 --file "$XDG_CONFIG_HOME/ksmserverrc" --group General --key loginMode emptySession
dbus-run-session startplasma-wayland dbus-run-session startplasma-wayland
+8 -7
View File
@@ -15,6 +15,7 @@ POINTER_IDLE=30 # seconds without mouse use before the controllers get their
POINTER_WAKE_COUNTS=40 # mouse counts within 1 s needed to wake or re-claim the laser (ignores desk jitter) POINTER_WAKE_COUNTS=40 # mouse counts within 1 s needed to wake or re-claim the laser (ignores desk jitter)
POINTER_CONTROLLER_PICKUP=1 # how hard a controller must move (x 0.35 m/s or 2 rad/s, for 100 ms) to take the laser back; raise it if resting controllers do (0.5 to 5) POINTER_CONTROLLER_PICKUP=1 # how hard a controller must move (x 0.35 m/s or 2 rad/s, for 100 ms) to take the laser back; raise it if resting controllers do (0.5 to 5)
POINTER_DISTANCE=1.5 # metres to the cursor when it isn't on a panel POINTER_DISTANCE=1.5 # metres to the cursor when it isn't on a panel
POINTER_ROLE=right # the hand role the pointer's controller takes: right | left | stylus (no hand); with a Frame controller held in that hand, clicks don't land, so pick the other
POINTER_CURSOR_DEG=0.4 # size of the free-space dot, in degrees POINTER_CURSOR_DEG=0.4 # size of the free-space dot, in degrees
POINTER_ORIGIN_FRACTION=0.95 # laser starts this far along eye->cursor: its beam is a few cm, SteamVR's hit dot tiny POINTER_ORIGIN_FRACTION=0.95 # laser starts this far along eye->cursor: its beam is a few cm, SteamVR's hit dot tiny
POINTER_LASER_WIDTH=0.8 # controller beam width (dashboard.laserRayWidthScale), set once at helper start POINTER_LASER_WIDTH=0.8 # controller beam width (dashboard.laserRayWidthScale), set once at helper start
@@ -29,14 +30,15 @@ POINTER_LEASH_RETURN=0.2 # head follow: once it comes along, how quickly (s) t
POINTER_FOLLOW_REACH=70 # head follow: how far (degrees) from the middle of your view the mouse can move the cursor POINTER_FOLLOW_REACH=70 # head follow: how far (degrees) from the middle of your view the mouse can move the cursor
POINTER_GAZE=0 # 1 = the pointer goes where you look, the mouse does the last bit (needs the gaze service: gaze/run.sh install) POINTER_GAZE=0 # 1 = the pointer goes where you look, the mouse does the last bit (needs the gaze service: gaze/run.sh install)
POINTER_GAZE_RETAKE=5 # gaze mode: how far (degrees) you look away from the pointer before the gaze takes it back from the mouse POINTER_GAZE_RETAKE=5 # gaze mode: how far (degrees) you look away from the pointer before the gaze takes it back from the mouse
POINTER_GAZE_NUDGE_MAX=8 # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error POINTER_GAZE_NUDGE_MAX=55 # gaze mode: a correction up to this far (degrees) before a click is learned as the eye tracker's error; a bigger one runs the quick check (55: half the headset's view)
POINTER_GAZE_HOLD=0.5 # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go POINTER_GAZE_HOLD=0.5 # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
POINTER_GAZE_SHOW=1 # gaze mode: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click POINTER_GAZE_MOUSE_MOVE=held # gaze mode: the mouse moves the pointer only while a button is held, as a correction (held) | any time (free)
POINTER_GAZE_DOT=always # gaze mode: the dot shows all the time (always) | only while the mouse moves it (moving)
POINTER_GAZE_SHOW=1 # gaze mode, with POINTER_GAZE_DOT=moving: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct) POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
POINTER_PRECISION_GAIN=0.5 # gaze precision (a button mapped to gaze_precision): the pointer turns this times the controller's turn POINTER_HEAD_DEADZONE=0.5 # keyboard clicks at the gaze (Meta+J, Meta+K): degrees the head turns before the held dot moves with it
POINTER_PRECISION_DEADZONE=0.3 # gaze precision: degrees the controller turns before the pointer does (the press's own jolt) POINTER_KEY_TAP=0.25 # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
POINTER_GAZE_DRAG_GAIN=1 # gaze drag (gaze_drag): the pointer turns this times the controller's turn GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
GAZE_TRACKER=steam # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated in the gaze probe with Own tracker)
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells) HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on
@@ -52,7 +54,6 @@ POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the p
POINTER_GRIP_GAIN=1 # hands: the pointer moves this times the gripping hand's angle POINTER_GRIP_GAIN=1 # hands: the pointer moves this times the gripping hand's angle
POINTER_GRIP_BELOW=0.35 # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk) POINTER_GRIP_BELOW=0.35 # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk)
POINTER_PINCH_TYPING=1 # hands: no pinch within this many seconds of a key (typing touches thumb to index) POINTER_PINCH_TYPING=1 # hands: no pinch within this many seconds of a key (typing touches thumb to index)
META_DASHBOARD=0 # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
SHARE_KEYS=0 # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen) SHARE_KEYS=0 # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
DISPLAY_OFF_MIN=0 # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never) DISPLAY_OFF_MIN=0 # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never)
DISPLAY_MOVE_MM=5 # displays off: the headset or a controller moving this far (mm) within 10 s counts as use DISPLAY_MOVE_MM=5 # displays off: the headset or a controller moving this far (mm) within 10 s counts as use
+1 -1
View File
@@ -40,7 +40,7 @@ The service runs after Bluetooth has started and never makes Bluetooth wait for
The install writes to `/etc`, so it needs `sudo` and the `steamos` user's password. The install writes to `/etc`, so it needs `sudo` and the `steamos` user's password.
- On the headset, `sudo` asks for the password in the terminal. If you've never set one, run `passwd` first. - On the headset, `sudo` asks for the password in the terminal. If you've never set one, run `passwd` first.
- From a PC over SSH, there's no terminal on the Frame to ask in, so put the password in a `.env` file at the repo root: - From a PC over SSH, the scripts ask for it in your terminal (through `ssh -t`). To skip the question, or with no terminal, put the password in a `.env` file at the repo root:
``` ```
steamos_root_pwd="your-password" steamos_root_pwd="your-password"
+3 -14
View File
@@ -1,8 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Install (or remove) the persistent Bluetooth LE workarounds on the Frame. # Install (or remove) the persistent Bluetooth LE workarounds on the Frame.
# Needs host sudo. On the Frame, sudo asks for the password in the terminal. # Needs host sudo: it asks for the password in the terminal, on the Frame or from a PC
# From a PC (or with no terminal), the password comes from steamos_root_pwd in the # (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
# repo's .env and is sent to sudo -S on stdin, never on a command line.
# Usage: setup/bluetooth/install.sh [install|uninstall|run] # Usage: setup/bluetooth/install.sh [install|uninstall|run]
# run re-apply now without restarting bluetooth (after pairing a new device) # run re-apply now without restarting bluetooth (after pairing a new device)
set -euo pipefail set -euo pipefail
@@ -11,17 +10,7 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
. "$root/scripts/_env.sh" . "$root/scripts/_env.sh"
src=$FRAME_REPO/setup/bluetooth src=$FRAME_REPO/setup/bluetooth
sudo_run() { sudo_run() { frame_sudo "$1"; }
if [ "$FRAME_LOCAL" = 1 ] && [ -t 0 ]; then
sudo bash -c "$1" # asks for the password here
return
fi
local pw
pw=$(sed -n 's/^steamos_root_pwd=//p' "$root/.env" 2>/dev/null)
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
[ -n "$pw" ] || { echo "no terminal for sudo, and steamos_root_pwd is missing from $root/.env" >&2; exit 1; }
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
}
case ${1:-install} in case ${1:-install} in
install) install)
+7 -5
View File
@@ -18,9 +18,11 @@ packages=(
mesa-libgbm-devel wayland-devel vulkan-loader-devel vulkan-headers plasma-wayland-protocols wlroots-devel mesa-libgbm-devel wayland-devel vulkan-loader-devel vulkan-headers plasma-wayland-protocols wlroots-devel
# ft_pointer SteamVR driver: static C++ runtime (the host has an older glibc) # ft_pointer SteamVR driver: static C++ runtime (the host has an older glibc)
libstdc++-static libstdc++-static
# hand tracking: ft-hands reads the calibration with jsoncpp; ft-camd runs on the host, linked # hand tracking (deferred: not installed by install.sh; hands/run.sh install builds it): ft-hands
# statically; the Python tools (hands/tools) need NumPy and OpenCV # reads the calibration with jsoncpp; ft-camd runs on the host, linked statically. Its Python
jsoncpp-devel glibc-static python3-numpy python3-opencv # tools (hands/tools) need NumPy and OpenCV, which aren't here: Fedora's python3-opencv pulls
# in over a gigabyte (hands/README.md says how to get them)
jsoncpp-devel glibc-static
# Frametop Input Settings app (Kirigami, PySide6) # Frametop Input Settings app (Kirigami, PySide6)
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland breeze-icon-theme plasma-breeze python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland breeze-icon-theme plasma-breeze
# Frametop remote desktop (VNC bridge through krdp) # Frametop remote desktop (VNC bridge through krdp)
@@ -43,9 +45,9 @@ fi
"$distrobox" enter dev -- bash -c ' "$distrobox" enter dev -- bash -c '
set -euo pipefail set -euo pipefail
echo "installing ${#@} packages (already-installed ones are skipped)" echo "installing ${#@} packages (already-installed ones are skipped)"
sudo dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; } sudo -n dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; }
# OpenVR programs built here (the pointer helper and probe) look for the runtime at /opt/steamvr. # OpenVR programs built here (the pointer helper and probe) look for the runtime at /opt/steamvr.
[ -e /opt/steamvr ] || sudo ln -s /run/host/opt/steamvr /opt/steamvr [ -e /opt/steamvr ] || sudo -n ln -s /run/host/opt/steamvr /opt/steamvr
echo "dev container ready: $(. /etc/os-release; echo $PRETTY_NAME), glibc $(ldd --version | head -1 | grep -oE "[0-9.]+$")" echo "dev container ready: $(. /etc/os-release; echo $PRETTY_NAME), glibc $(ldd --version | head -1 | grep -oE "[0-9.]+$")"
' dev "$@" ' dev "$@"
EOF EOF
Executable
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""ft-steam: Steam's menu and checks, through Steam's own UI (steamui.py).
ft-steam menu open the SteamVR dashboard on Steam's menu, or close the dashboard if it's
open (the Steam button's job, as a shortcut: the input relay's steam_menu)
ft-steam check do Steam's UI calls that the above needs still exist (update-check.py)
No pointer mode needed: it asks Steam to show its dashboard overlay (window.ShowVROverlay, what
Steam itself calls) and to focus the Steam frame's left menu (MenuStore.OpenMainMenu, which in
VR invokes the frame's focus_left_frame_menu_action). SteamClient.OpenVR.VROverlay says
whether the dashboard is up and hides it. These are Steam's UI internals, not an API, so a
Steam client update can rename them: `ft-steam check` says so.
"""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
from steamui import SteamUnreachable, evaluate # noqa: E402
WINDOW = "SteamUIStore.WindowStore.VRGamepadUIMainWindowInstance"
MENU = f"""(async () => {{
const vr = SteamClient.OpenVR.VROverlay;
if (await vr.IsDashboardVisible()) {{ vr.HideDashboard(); return "closed"; }}
const w = {WINDOW};
if (!w) throw new Error("Steam's VR dashboard window isn't open");
ShowVROverlay(w.GetVROverlayKey());
// The frame's menu takes focus only once the dashboard is up.
for (let i = 0; i < 40 && !(await vr.IsDashboardVisible()); i++) await new Promise(r => setTimeout(r, 25));
w.MenuStore.OpenMainMenu();
return "opened";
}})()"""
CHECK = f"""(() => {{
const missing = [];
const vr = SteamClient.OpenVR && SteamClient.OpenVR.VROverlay;
for (const f of ["IsDashboardVisible", "HideDashboard"]) if (!vr || typeof vr[f] !== "function") missing.push("SteamClient.OpenVR.VROverlay." + f);
if (typeof window.ShowVROverlay !== "function") missing.push("ShowVROverlay");
const w = {WINDOW};
if (!w) missing.push("{WINDOW}");
else {{
if (typeof w.GetVROverlayKey !== "function") missing.push("GetVROverlayKey");
if (!w.MenuStore || typeof w.MenuStore.OpenMainMenu !== "function") missing.push("MenuStore.OpenMainMenu");
}}
return missing;
}})()"""
def main():
cmd = sys.argv[1] if len(sys.argv) == 2 else ""
if cmd not in ("menu", "check"):
sys.exit(__doc__)
try:
result = evaluate(MENU if cmd == "menu" else CHECK)
except SteamUnreachable as e:
sys.exit(f"ft-steam: {e}")
if cmd == "check":
if result:
sys.exit("Steam's UI lacks " + ", ".join(result))
print("ok")
else:
print(result)
if __name__ == "__main__":
main()
+108
View File
@@ -0,0 +1,108 @@
"""Run JavaScript in Steam's UI, standard library only.
Steam on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. Steam's
own calls live there: SteamClient.* (settings, SteamVR's dashboard) and the UI's stores
(SteamUIStore). Used by Frametop Display Settings (display-settings/steam_settings.py) and
ft-steam (the Steam menu shortcut).
evaluate(expression) the expression's value (awaited, JSON-able); SteamUnreachable when
Steam or its UI isn't there, or the script threw
"""
import base64
import json
import os
import socket
import struct
import urllib.request
CDP_PORT = 8080
class SteamUnreachable(Exception):
pass
class _WebSocket:
def __init__(self, url, timeout=5):
host_port, path = url[len("ws://"):].split("/", 1)
host, port = host_port.rsplit(":", 1)
self.sock = socket.create_connection((host, int(port)), timeout=timeout)
key = base64.b64encode(os.urandom(16)).decode()
self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
head = b""
while b"\r\n\r\n" not in head:
chunk = self.sock.recv(4096)
if not chunk:
raise SteamUnreachable("Steam closed the connection")
head += chunk
if b" 101 " not in head.split(b"\r\n", 1)[0]:
raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
self.buf = head.split(b"\r\n\r\n", 1)[1]
def send(self, text):
data, mask = text.encode(), os.urandom(4)
n = len(data)
if n < 126:
head = struct.pack(">BB", 0x81, 0x80 | n)
elif n < 65536:
head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
else:
head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
def _take(self, n):
while len(self.buf) < n:
chunk = self.sock.recv(65536)
if not chunk:
raise SteamUnreachable("Steam closed the connection")
self.buf += chunk
out, self.buf = self.buf[:n], self.buf[n:]
return out
def recv(self):
message = b""
while True:
b0, b1 = self._take(2)
n = b1 & 0x7F
if n == 126:
n = struct.unpack(">H", self._take(2))[0]
elif n == 127:
n = struct.unpack(">Q", self._take(8))[0]
message += self._take(n)
if b0 & 0x80:
return message.decode(errors="replace")
def close(self):
self.sock.close()
def evaluate(expression):
"""Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
try:
with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
targets = json.load(r)
except OSError as e:
raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
if not url:
raise SteamUnreachable("Steam's UI isn't running")
try:
ws = _WebSocket(url)
try:
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
"params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
while True:
reply = json.loads(ws.recv())
if reply.get("id") == 1:
break
finally:
ws.close()
except OSError as e:
raise SteamUnreachable(str(e)) from e
result = reply.get("result", {})
if "exceptionDetails" in result:
details = result["exceptionDetails"]
raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
return result.get("result", {}).get("value")