Compare commits

...
92 Commits
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 0821b1013e Settings: HANDS_SWAP_SIDES defaults to auto, and old untouched configs move to it
frametop.conf.example said HANDS_SWAP_SIDES=0, and desktops.sh copies it on a
fresh install, so every new install forced ft-camd's side camera names and
turned the hand tracker's own side check off. Some SteamVR restarts swap those
names, and then hands land beside their cutouts and recordings are mislabelled.

The example now says auto. scripts/conf-migrate.sh, run by install.sh and
hands/rec/install.sh, replaces the old line only where it's still exactly as
the example wrote it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:54:03 -06:00
DeeJanuzandClaude Opus 5.5 3cccad3525 Hands: label side cameras by the hands when a forced HANDS_SWAP_SIDES disagrees
With HANDS_SWAP_SIDES set to 0 or 1, ft-hands kept the forced naming as the
published truth even after the hands showed it was backwards. The hand recorder
took that as the session's decision, so the export labelled slam_left and
slam_right the wrong way round (dataset PR #4: every take swapped).

ft-hands now publishes the hands' answer once they disagree (state "forced,
disagrees"); tracking keeps the forced names. sides.read_live corrects the same
case from an ft-hands built before this, and the recorder's log says to use auto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:54:03 -06:00
DeeJanuzandClaude Opus 5.5 276c1409e8 Pages: the hand recorder's fix ships in Frametop 0.2.1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:46:35 -06:00
DeeJanuzandClaude Opus 5.5 a0fe3bb55e Merge experimental: eye tracker first calibration, accessibility registry, hand recorder without the colour module (#37)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:38:43 -06:00
DeeJanuzandClaude Opus 5.5 13198bc203 Pages: the hand recorder works without the colour module now
The known-issue notice becomes a fixed notice: the fix (98bd6ce, 662919b,
67a3c02) reaches main with this release. Anyone who hit "2 of 4 mono
cameras" is pointed at Update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:34:49 -06:00
DeeJanuzandClaude Opus 5.5 42a26753d3 Merge main into experimental: get.sh --branch and the hand recorder page
PR #29 was squash-merged, so main and experimental last shared history at
1ebf369 and 11 files conflicted. Main's copy of each was the PR #29 head
(ec870d2), which experimental already has, so experimental's side is kept
for all of them. The result is experimental plus main's get.sh and
hand-recorder.md, and main can take experimental with a merge commit again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:34:35 -06:00
DeeJanuzandClaude Opus 5.5 0f0674ff36 Merge branch gaze-double-cal (the first calibration no longer runs twice) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:28:23 -06:00
DeeJanuzandClaude Opus 5.5 3a796b5013 Gaze: the first calibration no longer runs twice
Turning gaze mode on wakes our tracker, so its eyes come back just as the
first calibration opens. DON_DELAY later, "the headset went on" re-armed the
automatic calibration while it was still open, and when it ended, ft-eyes'
status was still a moment old (not calibrated), so a second one opened at
once. Seen live on 2026-10-05: 26 of 27 dots, then 27 more.

The headset going on re-arms it only when no check is open.
first-calibration-test.py now has the headset go on mid-calibration and
checks that only one opens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:28:15 -06:00
DeeJanuz 67a3c02e31 ft-camd: map a camera whose dark frames are all zeros
Through the ISP (no colour module), the near-black exposures come out
all zeros, identical every time, so their dequeues change no buffer and
ft-camd never finished learning the side cameras' buffers ("can't tell
which buffers are this camera's yet"): the side pair published nothing.
Such an index is now left unmapped and its frames counted as dark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 89ec1608d9)
2026-10-05 13:55:18 -06:00
DeeJanuz 662919babf camcheck: no nested parentheses in the missing cameras' message
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 917ef9d29c)
2026-10-05 13:55:17 -06:00
DeeJanuz 98bd6ce21c Hand recorder: headsets without the colour module
Without the Arcturus module, XRService runs the side cameras through the
ISP ("ISP enabled for tracking cameras (main VFE available)"): NV12 on
vfe0 and vfe1, pitch 1152. ft-camd published only grey cameras, so it
dropped them, and the recorder stopped at "ft-camd publishes only 2 of 4
mono cameras" whatever was restarted.

- ft-camd reads a tracking camera's NV12 luma as its grey image.
- ft-hands and check_sides name the cameras by XRService's numbering in
  its log (TrackingCameraInit index 0-3), not by capture pipe, which
  moves with the module; a camera whose size isn't its calibration's is
  left out. sides.json says which device each camera was.
- camcheck reports the camera map, the ISP routing and which cameras
  ft-camd is missing; the recorder says so instead of "restart it",
  restarts its own ft-camd once when that's the only problem, and keeps
  the map in session.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4db5266cf7)
2026-10-05 13:55:17 -06:00
DeeJanuzandClaude Opus 5.5 608ad6034a get.sh: --branch NAME, to test a fix before it's released
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:49:20 -06:00
DeeJanuzandClaude Opus 5.5 336a60ce32 Pages: hand recorder needs the color module until the fix
Headsets without the Arcturus color passthrough module stop at the
camera check (ft-camd publishes only 2 of 4 mono cameras). Say so on
the install page, with the fix due by October 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:09:28 -06:00
DeeJanuz e1f27958ca Merge remote-tracking branch 'origin/experimental' into experimental 2026-10-05 11:12:07 -06:00
DeeJanuzandClaude Opus 5.5 c969963b83 A one-command uninstaller that keeps the headset usable
uninstall.sh replaces the README's thirteen uninstall commands. Run from a
Frametop desktop, those commands stopped the input relay second, which took
the keyboard and mouse away before the rest could be typed. And deleting
~/frametop first left Launch a program -> Desktop pointing at a missing
script.

The script works in two steps. Step 1 stops Frametop from starting: the
launcher entry, the user services (disabled, not stopped), the SteamVR
driver, the menu entries, and the eye grabber and Bluetooth fixes under
/etc (one sudo). Everything running keeps running until the headset
restarts, and the script offers to restart it. Step 2 runs once none of
Frametop's programs run. It deletes the code, and asks before deleting the
settings, recordings, and the dev container.

It doesn't use the rest of the repo, so the Pages one-liner works even when
~/frametop is gone. --dry-run shows each command without running it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 11:11:57 -06:00
DeeJanuzandClaude Opus 5.5 9f2ee5ad68 A one-command uninstaller that keeps the headset usable
uninstall.sh replaces the README's thirteen uninstall commands. Run from a
Frametop desktop, those commands stopped the input relay second, which took
the keyboard and mouse away before the rest could be typed. And deleting
~/frametop first left Launch a program -> Desktop pointing at a missing
script.

The script works in two steps. Step 1 stops Frametop from starting: the
launcher entry, the user services (disabled, not stopped), the SteamVR
driver, the menu entries, and the eye grabber and Bluetooth fixes under
/etc (one sudo). Everything running keeps running until the headset
restarts, and the script offers to restart it. Step 2 runs once none of
Frametop's programs run. It deletes the code, and asks before deleting the
settings, recordings, and the dev container.

It doesn't use the rest of the repo, so the Pages one-liner works even when
~/frametop is gone. --dry-run shows each command without running it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 11:11:38 -06:00
DeeJanuzandClaude Opus 5.5 dfc8e6eb5f Merge branch fresh-install (our eye tracker's first calibration; SteamVR's tools from a terminal in the desktop; gaze in the report) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:55:22 -06:00
DeeJanuzandClaude Opus 5.5 1037d4e125 Report: the gaze service, our eye tracker, and their logs
A gaze problem couldn't be told from a report: it had no gaze section. A user's report of
2026-10-05 showed only that the gaze service ran and our tracker was picked.

The report now has a Gaze section: whether the gaze service and our tracker's frame grabber
are enabled and running, when SteamVR's correction and our tracker's calibration were made
(or "none"), and the service's status (ft-gazectl status, on one line: the tracker in use,
whether it's awake and why not, checks.problem). Then the last 10 checks and calibration dots
(checks.jsonl: each dot's reply and whether it was taken), the gaze service's last 60 lines
without podman's exec lines, and the frame grabber's last 20. Nothing in them is an eye image.

Tested on this Frame: each section fills in, and the status is one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:54:57 -06:00
DeeJanuzandClaude Opus 5.5 307e2b7308 SteamVR's tools find SteamVR from a terminal in the Frametop desktop
The desktop has its own XDG_CONFIG_HOME (~/.config/frametop), and SteamVR's tools and OpenVR
read their path registry from there. So from a terminal in the desktop:

- pointer/driver/install.sh (install.sh step 4, every reinstall or update from Konsole) wrote
  a new registry, ~/.config/frametop/openvr/openvrpaths.vrpath, with only our driver and
  "runtime": null, and never registered the driver with SteamVR. That stray file then hid
  SteamVR from every OpenVR program started in the desktop.
- scripts/update-check.py (and so doctor.sh and report.sh) reported OpenVR "can't connect to
  SteamVR as a background app" (VRInitError_Init_PathRegistryNotFound, or
  InstallationNotFound with the stray file) and "vrcmd --overlays lists no overlays". A user's
  report of 2026-10-05 showed both, with SteamVR and Frametop's services fine.
- ft-layout's vrcmd calls (the gamescope backend) found no overlays.

They now run with XDG_CONFIG_HOME=~/.config: the driver installer (install, uninstall, probe,
aimhere), update-check.py (for all its checks: nothing there reads the desktop's own config),
report.sh's vrpathreg show, and ft-layout's vrcmd. report.sh doesn't set it for the whole
report, since session/fix-panels.py --check reads the desktop's Plasma config through it.

The driver installer also removes the stray registry (only vrpathreg's, with no runtime), and
update-check.py warns about one. And vrpathreg runs `xdg-open vrmonitor://driverinstalled` to
tell SteamVR's desktop monitor, which the Frame doesn't have: in the desktop that showed "could
not read file vrmonitor://driverinstalled" on every install. A stand-in xdg-open on its PATH
takes it, so install.sh no longer filters the step's output.

Tested in a fake HOME with a copy of SteamVR's registry, a stray one, and the desktop's
XDG_CONFIG_HOME: the new installer removes the stray file, registers the driver in the copy,
and never reaches xdg-open; the old one wrote the stray file, left the copy alone, and ran
xdg-open. update-check.py from that environment: OpenVR and vrcmd ok (the old one fails both);
its stray-registry warning fires on a seeded file. ft-layout's vrcmd: 119 overlays (was 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:53:59 -06:00
DeeJanuzandClaude Opus 5.5 d39fbb9146 Gaze: our eye tracker can take its first calibration
A fresh install that chose our tracker (the installer's default since 12ad93d) could never
calibrate it, so gaze mode never worked. ft-eyes maps pupils to a gaze only once it has a
calibration, so before its first one ft-gaze's "own" source is empty. ft-gazed then never
counted a sample, Checks.can_run() stayed false, and the calibration refused to open: "Not
calibrated, and the calibration can't open: the eye tracker isn't sending", or "the headset is
off or the tracker isn't sending" from Calibrate. This Frame never hit it, because its
calibration came from the gaze probe. Reported 2026-10-05 on SteamOS stable.

- With our tracker in use, running, and uncalibrated ("blind"), SteamVR's tracker seeing an
  eye is enough to open the full calibration (ft-eyes answering is what makes calibrated()
  False rather than None).
- Each dot stands in for the gaze, so a click takes the CHECK_WINDOW up to it. ft-eyes already
  checks, in calib-point, that each pupil was seen in enough frames and held still then, and
  its reason for a refused dot shows in the panel's note as before.
- A quick or five-dot check is refused until then ("use Calibrate"): ours can't take a click
  without a calibration.
- With no eyes seen, the reason given is that, not "the eye tracker isn't sending".

gaze/test/first-calibration-test.py runs the service against a fake ft-gaze, ft-eyes and
pointer helper, in a temp HOME: the calibration opens by itself when gaze mode comes on, each
click sends ft-eyes the 0.6 s before it, a refused dot's reason reaches the panel, and
calib-fit leaves the tracker calibrated with gaze mode still on. It passes here and fails on
the previous code. gaze/test/idle-test.py still passes. Not yet tested in the headset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:51:07 -06:00
DeeJanuzandClaude Opus 5.5 7fe5d8c0dd Merge branch nested-atspi (PR #28: the desktop starts an accessibility (AT-SPI) registry, #27) into experimental
JakeGreen2145's PR plus our fix 6df6f97: the watcher checks its buses
every 5 seconds instead of every second (about 1% of a core), and
design.md says the watcher is the only cleanup when the VR launcher
runs the desktop in steam.service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:27:26 -06:00
DeeJanuzandClaude Opus 5.5 6df6f97937 Session: AT-SPI watcher checks its buses every 5 seconds
Each check starts two gdbus processes (about 5.5 ms of CPU each on the
Frame), so checking once a second cost about 1% of a core for as long as
the desktop ran. On SteamOS 0.3.0 native activation fails, so the watcher
always runs. A registry left behind now goes within 5 seconds instead of 1.

design.md also says where the watcher matters: started from the VR
launcher, the desktop runs in steam.service, which doesn't stop with it,
so keep-apps.sh and the unit's stop don't clean up there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:23:51 -06:00
DeeJanuzandClaude Opus 5.5 ca9492be00 Pages: link the Frametop Discord at the top and bottom of the hand recorder page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 19:50:12 -06:00
DeeJanuzandClaude Opus 5.5 41b5b68a41 Pages: hand recorder install page
hand-recorder.md, served at deejanuz.github.io/frametop/hand-recorder.html:
the Konsole commands to install, update and uninstall the Hand Recorder,
who can take part, and what it needs. Frametop comes first for now, since
the recorder runs in its desktop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 19:47:13 -06:00
9d7c8a0b91 Experimental (#29)
* Input relay: typing with the pointer helper down no longer ends the relay

Typing on a pass-through keyboard tells the helper "typing". With the
helper not running (SteamVR off), that send raised ConnectionRefusedError
and the relay exited, dropping every grab until systemd restarted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-steam: open Steam's menu through Steam's own UI

steam/ft-steam menu opens the SteamVR dashboard on Steam's menu, or closes
the dashboard if it's up, without pointer mode: it asks Steam's UI over its
debugging port to show its dashboard overlay (ShowVROverlay, what Steam
calls itself) and focus the Steam frame's left menu (MenuStore.OpenMainMenu).
ft-steam check says whether those calls still exist, and update-check.py
runs it, since a Steam client update can rename them.

The CDP client moves from display-settings/steam_settings.py to
steam/steamui.py, so both use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Shortcuts: Steam menu, commands, and modifier taps

Key combinations (and mouse and controller buttons) get two new actions:
- steam_menu: Open Steam menu / close dashboard (steam/ft-steam menu).
- command:CMD: run CMD with sh -c, as the relay's service, with layout/,
  float/ and steam/ on its PATH. Input Settings offers it for key
  combinations as Run a command...
Both work without pointer mode.

A modifier on its own is now a key combination too: a tap, pressed and
released with no other key, mouse button, or scroll in between. A bound
tap sends the desktop F24 before the release, so Plasma's launcher stays
shut. The defaults gain a Meta tap for the Steam menu; this replaces
META_DASHBOARD, which only worked in pointer mode.

input/test/keys-test.py runs the relay against fake devices with every
outgoing socket renamed, so it's safe next to the live relay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Relay: share a key combination's Meta release with frame-voice

A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release
from the desktop, and the relay skipped share_key for it too. frame-voice
saw Meta go down on @frametop_keys and never come up, so it held all
dictated text back, waiting for that release. Keys of grabbed keyboards
are now shared as pressed, before key_binding() decides what the desktop
gets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-cutouts, the hand cutouts without pinches and grips

hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient
user units with ft-hands' new --no-gestures: hands are published for
ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks
or drags and a closing hand doesn't raise the tracking rate. It needs a
build and ft-camd's capabilities, not hands/run.sh install. Its units
conflict with ft-handsctl's, so each stops the other, and they stop with
SteamVR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-cutouts status: only the current run's tracker lines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: say why gaze mode can't work yet, and open the calibration whenever it's missing

Turning gaze mode on without a calibration opened Calibrate only on the
off-to-on change, and only if it could open right then. With the headset
off, the eye tracker silent, or the panel not built, or with gaze mode
already on when the gaze service started, nothing opened and nothing said
why: the pointer just stayed a mouse.

- The gaze service now checks every second: gaze mode on, no calibration
  for the tracker in use, eyes seen -> the full calibration opens. One
  that closes unfinished opens again only after the headset comes off and
  on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that
  fails retries every 10 s.
- Its status says why gaze mode can't work yet (checks.problem): not
  calibrated and opening, open, closed unfinished, or can't open and why.
- Input Settings shows that under the Gaze pointer switch, along with the
  gaze service not installed or not running and our tracker missing its
  frame grabber.
- ft-gazectl on notes a missing calibration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken

A user on a fresh install got "Calibration failed: only 0 of 21 dots" with
no reason. The installer never installed our tracker, so gaze used
SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an
eye for most of the look. The panel just showed a red ring.

- install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh)
  after gaze mode, yes by default; it needs sudo, so --yes runs it only
  when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker.
  Configs that still say GAZE_TRACKER=steam (the old template) are asked
  whether to switch.
- GAZE_TRACKER=auto, the new default: ours when it's installed (the
  frame grabber, its unit, and ft-eyes' Python), else SteamVR's.
  ft-gazed rechecks every second, so installing it switches over. Input
  Settings lists Own tracker first as recommended, and says how to
  install it when it's missing (checking the host's /etc through
  /run/host from the dev container).
- The calibration panel has a note line, orange over the instructions:
  why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for
  SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply
  for ours), what a click is still waiting for after 1.5 s, and a failed
  calibration's most common reason, which the Gaze page shows too.
  steady_samples keeps the same samples as before (checked on 2037
  windows of recordings); a lost eye is named before a blink, since its
  openness reads 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* README: link the Frametop Discord

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Wait for a new container to finish setting up before entering it

container-up.sh starts the dev container in a scope of its own, so
distrobox enter finds it running and skips its wait for distrobox-init.
On a fresh install, init was still setting up passwordless sudo when
dev-container.sh ran sudo dnf install, and sudo asked for a password
with no terminal to read it from. container-up.sh now waits for
container_setup_done itself, and the container's sudo calls use -n,
so a password prompt fails at once with a clear message.

Fixes #9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Prevent small desktop overlay pointer movements from starting a drag

* Clear reported drag state on controller release

* Click stability: only a hand controller's press starts it

The 3D mouse drives SteamVR's laser through the ft_pointer virtual
controller, so its events reach the screens the same way a controller's
do. The filter held every press, which turned the mouse's short drags
(selecting a character or two, nudging a slider) into clicks. Mark
button events from hand controllers and start the filter only on those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: retry a new device until udev gives it to the input group

A new /dev/input node is root:root 0600 until udev applies GROUP=input. The
scan probed each new node once and marked it seen even when the open failed, so
a node caught in that gap was never opened. Behind a KVM, a switch brings back a
hub of devices at once: on the Frame, four nodes failed with EACCES in one switch,
the keyboard was never grabbed, and its keys went to gamescope instead of the
desktop screens. A node that isn't readable yet now waits for the next scan.

* Screens: take a screen's overlays from one copy in the catcher

While a button pressed on a screen is held, UpdateCatcher checks every tick
whether the laser is still on one of the screen's overlays. It built that list
from s.All().begin() and s.All().end(), but All() returns a std::array by
value: iterators into two different temporaries, which is undefined behaviour.
A clang build of ft-screens got a garbage length, threw std::length_error, and
aborted on the first click, taking KWin and the desktop with it.

* Gaze: leave SteamVR's gaze action alone during VR games

From curiousjtuber's PR #13: with the gaze service running, SteamVR
restarted its eye tracker every 10 to 13 s of Beat Saber, as if the
headset came off, and each restart took input focus from the game.
The PR stopped every read in a game. Only the action path reaches
SteamVR (UpdateActionState on the gaze set at overlay-global priority,
then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are
read-only files. So only the action is skipped while a scene app runs,
and gaze keeps moving the pointer over the dashboard in a game. The
action source is only used with --source action.

Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: --record-hz, and the hand recorder's design (hands/rec/DESIGN.md)

ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10).
DESIGN.md lays out the recorder: the headset panel, the session runner and its script,
the files, review and export, consent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-handpanel, the hand recorder's headset panel

A head-locked SteamVR overlay for the hand recorder (hands/rec/DESIGN.md):
1.2 m ahead, 12 degrees up, 36 degrees wide, drawn with stb_truetype
into three shared DMA-BUFs as ft-gazepanel does. It shows the title,
step, wrapped instruction, note, countdown, hand chips, near/far bar
and a "Paused" cover, driven over @ft_handpanel.

It also places the touch target, a 2 cm dot in its own overlay fixed
in the room where the head was at the first command for that point,
and logs head and controller poses to poses.jsonl at 250 Hz from a
thread of its own. Both threads take one lock around OpenVR calls.

--no-vr prints each picture's state to stdout (and --dump writes the
pictures), for testing without a headset. hands/rec/build.sh builds it
in the dev container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder's worn check goes by the panel's backlight, as frame-job does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the hand recorder's session runner and guided script

hands/rec/session.py runs a recording session from script.json: it starts
ft-camd and a tracking ft-hands as transient units only if they aren't
running, records each section as one take (ft-hands --record-only at
10 sets/s, a new sets-N.bin after each pause), drives ft-handpanel, and
writes session.json, calibration.json (identifying fields removed),
prompts.jsonl and take.json. Feedback comes from the live hands file and,
in the controller sections, from the panel's device poll. It also runs
from the command line (--dry-run, --speed, --ring, --no-start).

hands/rec/script.json: 11 sections, about 9 minutes without the object
and controller sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the hand recorder's window, review and export

hands/rec/ft_handrec.py + main.qml (Kirigami, dev container; host launcher
hands/rec/ft-handrec): consent (CONSENT.md, asked again when its version
changes; profile.json with a random contributor id), the before-you-start
checklist with the lighting and free-space checks, the session controls
(Space pauses, Esc stops), review with a frame-set viewer that deletes
ranges, takes and sessions, export with progress and cancel (warns while
the headset is worn), and the upload page (UPLOAD.md, the
huggingface-cli command; HF_DATASET is a placeholder). --dry-run runs
sessions without processes, for testing.

hands/rec/takes.py (standard library): indexes sets.bin and sets-N.bin
without reading pixels, reads one set's cameras, keeps deleted ranges in
take.json, and exports: deleted sets left out, zstd -10 -T2 at nice 19,
manifest.json and SHA256SUMS, nothing left behind on cancel.

CONSENT.md and UPLOAD.md are drafts pending a legal review; the window
says contributions aren't open yet. The dev container gains zstd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: upload from the hand recorder's window, export checks, a rehearsal

hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+)
checks an export before upload and when it's received: SHA256SUMS, an
allow-list of files, the manifest's schema and keys, the consent version,
a uuid4 contributor, no identifying fields in calibration.json or
device.json, every sets.bin.zst decompressed to its end as a stream with
each FHSET01 header checked against the manifest, jsonl lines, the total
size. It decompresses with compression.zstd, zstandard or the zstd
program. validate.py DIR [--json].

hands/rec/hub.py uploads an export with huggingface_hub, as a pull
request to contributions/<contributor>/<session>: validate first, refuse
a repeat of the same export, check the login (whoami) and access
(auth_check), upload_folder(create_pr=True) with the manifest summary as
the description, then record the PR under "uploads" in session.json.
Errors are explained (terms not accepted, not found, 401/403, network).
--dry-run makes no network calls. FT_HANDREC_DATASET overrides
HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real
upload needs FT_HANDREC_ALLOW_UPLOAD=1.

The Upload page shows the login with "Check again" and how to run
hf auth login in a terminal (the token never enters the window), then
Upload with a phase, progress and Cancel (hub.py as a child process), the
PR link, and a warning for an export uploaded before. The manual
command stays as the fallback. ft-handrec --hub-dry-run.

session.py also saves device.json: cv.cad_from_cal and head from
/persist/device_config.json, the labeller's shape, nothing identifying;
export copies it. Session ids with a -N suffix are accepted everywhere.

hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays
30 s of a capture into a ring, session.py records a short test script
with ft-handpanel --no-vr and a tracker, then export, validate and a
dry-run upload (--repo ID uploads for real). It runs in one frame-job
scope, deletes its data and stops its processes, also on Ctrl+C.

hands/rec/tests/test_validate.py covers good and broken exports and hub.py
without the network. The dev container gains python3-huggingface-hub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* README: Frametop doesn't work on the SteamOS beta yet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 072a294941)

* Hands: step mode and pose pictures for the hand recorder

The first real session moved on every 5 s with text only, too fast to follow.
Each step now waits for Next (Space or the window's button), counts down 3-2-1
while recording, then holds. P pauses, R redoes a step, S skips a section;
"Advance by itself" (--auto) keeps the old timed flow. Nothing records while
a step waits: each step is its own recording part.

The panel and the window show a picture of each pose (hands/rec/poses,
generated by make_poses.py from a parametric hand, MIT) and a diagram of where
to hold the hands and how far out. prompts.jsonl gains ready, wait and redo
events; session.json gains mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the headset button as Next, clearer push steps

The headset's right-side click button (KEY_SELECT on gpio-keys, read without
a grab) now works the session: Next while a step waits, pause during a hold,
resume while paused. With no mouse connected the hints lead with it.

The push sections say plainly to push straight out from the headset and pull
back, with a side-view picture of the head, the headset and the arrow, and
the bar's ends read "At your chest" and "Arm out". The bar labels are sent
as one field, so labels with spaces no longer split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-floatd: a launch for a missing app no longer kills the control socket

Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.

Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-floatd: profiles keep and relaunch Flatpak apps whose window names another app id

An X11 window in a Flatpak can give KWin an app id with no desktop file:
RustDesk's says com.carriez.flutter_hbb (its GTK application id), and the
Flatpak's desktop file is com.rustdesk.RustDesk. A profile recorded that
id, so it couldn't relaunch the app (PR #16 keeps that from killing
ft-floatd's socket). And the window's pid is the sandbox's own, so a
launched window matched neither by process nor by app id, and didn't
float.

desktop_name() finds the desktop file whose StartupWMClass names the
window's class (or app id) when the app id has none. Capture records
that name, a profile claims open windows by it, and a launch's window
matches by it.

Profiles saved before this keep the old id; save them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: check the tracking cameras before recording, and watch for losing them

After the headset wakes, XRService sometimes fails to load the colour module's
VCINT FPGA image; then only the two side cameras run, without the IR light,
and the tracker finds no hands. hands/camcheck.py reads XRService's log, the
video nodes it holds and ft-camd's ring, and says ok, degraded or unknown.

The recorder won't start while degraded (--ignore-cameras overrides it), offers
a confirmed SteamVR restart, and stops the first hand-size step when the
tracker sees no hand at all. ft-camwatch (unit file only, not enabled) follows
the log, notifies, and with CAMWATCH_AUTO_RESTART=1 restarts SteamVR when the
headset isn't worn and nothing else uses VR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: shorter recording sessions with pose sweeps

The second real session took 16 minutes, half of it 36 still poses. Labels
come from the auto-labeller, so what matters is variety, not clean holds.
A sweep step shows a strip of pose pictures and lights one every 4 s while
the hands move slowly near and far; each cue is a prompt event with
"cue": true. The core session is now 15 steps, about 5 minutes recorded.

The pose groups, the one-hand sweeps' groups and the cue order are shuffled
per session, seeded from its id and saved in session.json. A quick round
(--quick, or the checklist's choice) is about 2 minutes for extra lighting.
Touch the dot has 6 dots, the push sections two heights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay test: let the fake devices past the udev permission check

Since the relay leaves a node it can't read yet for the next scan (12f2e84, PR #12), it
checks os.access first, and the test's fake /dev/input paths don't exist, so the relay never
opened them and every key check failed. The fake os now says they're readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pause Frametop for VR games

Frametop kept using the headset during games: with gaze mode off, our eye tracker still took
about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden
screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and
resuming brings back only what pausing stopped. It's also a way to keep the gaze service and
our eye tracker off during games, which PR #13 asked for.

Paused (input/game_pause.py, run by the input relay):
- frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop
  stop if they run
- the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a
  frame callback once a second; or, with pause_desktop "close", the desktop closes and starts
  again on resume
- the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key
  combinations do only pause_toggle, steam_menu and commands

Toggled by both thumbsticks clicked together twice (configurable), read passively from
vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the
new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game
starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input
Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a
paused gaze service as failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-hands works out which side camera is which

ft-camd tells the side cameras apart by XRService's buffer allocation order,
which some XRService starts reverse; both of 2026-10-02's starts did, so the
cutouts missed the hands. HANDS_SWAP_SIDES=auto (the default) has ft-hands
vote from hands seen in both side cameras: the landmark rays meet in front
of both cameras only under the right naming. While undecided it probes the
exchanged naming with the landmark model. It decides in about 2 s of hands
(right on all 7 recordings replayed), swaps the views in place, and publishes
sides.json. 0 and 1 still force it, with a warning when the hands disagree.

Recordings carry each part's naming and the session's decision; review,
export, validate and ft-handreplay put the names right, and takes.py sides
records a decision by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: idle while the gaze isn't used

The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes
still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze
and our tracker run only while gaze mode is on and someone wears the headset, while a check or
the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop
Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber
idles with our tracker.

- The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the
  headset); an older helper answers it as before, and the service then goes by gaze mode alone.
- A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once
  it sends; the automatic calibration waits quietly while it starts.
- Status has "awake" and "idle" (why), and the Gaze page shows it.
- gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* update-check: a still controller isn't a broken web socket

vrserver sends a controller's state only when something on it changes, and one lying still
or asleep may not even send its first one. The check subscribed to one controller and failed
after 3 s of silence. It now subscribes to all, and silence after a good handshake is a skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder measures the light itself

The checklist page measures the light when it opens, starting ft-camd if
nothing runs it (and stopping it on quit), instead of saying the cameras
aren't running. The round's lighting defaults to what the cameras measure:
daylight or indoor, from the mono cameras' ambient infrared. Lamps give off
little infrared, so dim and normal rooms read alike; picking dim, room or
daylight still overrides it. session.json gets source, measured and
ambient_ir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder's host commands run from the home folder

host-spawn starts a host command in the caller's folder. Started from /tmp,
the app's folder in the container is /run/host/tmp, which the host doesn't
have, so starting ft-camd (and every other host command) exited 127.
host_command now runs from home (env -C), and the launcher cds there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input Settings: the Games page is Game optimization

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: push steps say to follow the hollow circle, not the blue dot

The dot is the current tracker's distance guess, often wrong; the ring is
where the hands should be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: one thread for OpenCV and numpy

Nothing called cv2.setNumThreads, so OpenCV kept a pool of one worker per core for
pupil windows of 140 to 240 px. Live, its idle workers spun and yielded about 14,000
times a second each, about a quarter of a core, next to SteamVR's compositor. numpy's
OpenBLAS also started 8 threads that never had work.

eyes_pupil.py now sets OpenCV to one thread, and ft-eyes sets OPENBLAS_NUM_THREADS and
OMP_NUM_THREADS to 1 before numpy loads (a value already in the environment wins).

Replaying fit1 into a scratch share (ft-eyes-replay, 14 s measured, capped at one core
with the replay): threads 13 -> 1, involuntary context switches 3,812/s -> 430/s,
system time 6.9% -> 1.6% of a core. Under that cap the frames it kept up with went from
21-36 to 57-69 a second per eye.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: read the overlay list every 20 s, not every second

The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze
mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about
26 to 30 ms of CPU, so about 3% of a core all the time.

The list is now read every 20 seconds, and at once (at most once a second) when it may have
changed: the pointer waking, the dashboard opening or closing or creating an overlay, the
scene app changing, an "overlays" request, and a left click that hit nothing, which may be
on a panel that came up since. The thread waits on a condition variable instead of waking
every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a
new list is in, rather than at its next 1 s tick. Overlays already on the list still show
and hide within 50 ms, from the IsOverlayVisible poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-eyes and ft-gaze below SteamVR's priority

ft-eyes and ft-gaze run in the dev container through distrobox, so they live in podman's
libpod scope: frametop-gaze.service's limits never reach them, and they ran at nice 0
next to vrcompositor and vrserver, also at nice 0.

- ft-eyes sets itself to nice 10 and SCHED_BATCH at start, before its threads. Batch
  turns off wakeup preemption, so a frame ft-eyes wakes up for can wait out a running
  compositor's turn; a few ms late costs the gaze little.
- ft-gaze sets nice 5 before its threads start, but stays SCHED_OTHER: each sample goes
  on to the pointer, and batch would add the same wait to every one of them.
- Both only ever lower their priority (a higher nice already set wins), and a failure
  is logged and ignored. Checked in the dev container: nice 0 -> 10, policy 0 -> 3
  (SCHED_BATCH) without any capability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: sleep on the command socket while the pointer is off

The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake
or not, and every second it looked up every overlay's handle and read a string property from
all 64 device slots to find its own device.

With the pointer off and hand gestures off, the loop now waits in poll() on its command
socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read
(SteamVR input has no event to wait for). A mouse command ends the wait at once. The
headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep
going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the
pointer is awake, and waking forces both. The device index is looked for only while it's
unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is
kept only with hand gestures on, its one user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote desktop: connect FreeRDP only while a VNC viewer is connected

vnc-bridge.sh kept FreeRDP connected to krdpserver from the moment remote desktop
started, so krdp captured and H.264-encoded every KWin redraw in software (openh264)
with nobody watching: krdpserver 55-78% of a core, xfreerdp 16-27%, Xvnc 6-11%, with 0
clients on :5900. krdp 6.7 creates its screencast session per RDP connection and drops
it when the connection closes, so krdpserver itself idles without one and stays up.

The bridge now counts established connections to Xvnc's port with ss, starts FreeRDP
when a viewer appears (the desktop shows about 3 s later; the VNC screen is black until
then) and stops it 45 s after the last one leaves (VNC_IDLE_SEC). Xvnc has no client
hook, so its log output, which it writes for every connection, wakes the bridge early;
otherwise it looks every 5 s while idle (0.1% of a core measured, against 0.9% for ss
once a second) and every second while FreeRDP runs. The layout check runs only while
FreeRDP runs.

While a viewer is connected the bridge sends "watch 15" to ft-screens (@ft_screens) at
once and every 5 s, so screens at a reduced frame rate (out of view, headset on a
stand) stream at full rate; it lapses by itself if the bridge dies, and an ft-screens
without the command just answers an error. The window search after starting FreeRDP
now ends when FreeRDP exits instead of polling for 30 s.

krdp on 127.0.0.1 with a fresh password, VNC on the tailnet address with VncAuth, and
remote-ctl.sh start/stop (pause and resume) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote desktop: read the layout only after it changes

While FreeRDP ran, vnc-bridge.sh called ft-layout remote-view every 5 s, which scans
all of /proc for plasmashell and runs kscreen-doctor -j: about 4.4% of a core for a
layout that rarely changes.

It now stats the two files the answer depends on, the nested KWin's
~/.config/frametop/kwinoutputconfig.json (positions, scales, primary) and
~/.config/frametop-layout.json (screen sizes), once a second while FreeRDP runs. After
either changes it reads the layout every 2 s for 10 s, since KWin's outputs follow the
file a few seconds later; otherwise once a minute, in case a change touched neither.
With no VNC viewer connected nothing runs (previous commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: ft-eyes sleeps until the next frame is due

ft-eyes looked for new frames about 1,000 times a second: each pass of its loop asked the
control socket with a non-blocking recvfrom (a BlockingIOError nearly every time), read
both cameras' counters, and slept 1 ms. The frames come every 11.1 ms per camera, and only
as counters in ft-eyegrab's shared memory, so there's no fd to wait on.

Now each pass ends in select() on the control socket, with a timeout until 2 ms before the
next frame of either camera is due (from when its last one was seen), then every 1 ms until
it comes. A command wakes it at once. A camera with no frame for 0.1 s (headset off,
grabber idle) isn't waited for, and with both stopped it looks every 20 ms. Waiting for the
frame grabber's file uses the same select, 0.2 s at a time, instead of sleeping through
commands.

A new frame is still seen within about 1 ms of when it lands. On a synthetic share at 90 Hz
per camera, on a heavily loaded headset (load average 23, so ft-eyes rarely sat idle), its
waits went from 178 to 81 a second; unloaded, the old loop's 1 ms sleeps add up to about
1,000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: frame rates by attention, ticks in step with the display

ft-screens gave KWin a frame callback for every committed screen on each tick, and the tick
was an 11 ms timer set again after each run, so it slid through the display's frame and came
about 85 times a second at 90 Hz: the desktop repeated a frame several times a second (judder
in scrolling and video), and KWin drew every screen in one burst at a random point of
vrcompositor's frame. Hidden screens got the same 90 Hz unless Frametop was paused for a game.

- Ticks run on a timerfd at absolute times, once per display frame, 1 ms after the vsync
  (IVRSystem::GetTimeSinceLastVsync and the HMD's display frequency, read once a second), so
  KWin gets its callbacks early in the frame. Measured with --no-vr: 91 wakeups a second
  instead of about 85. On the Frame the vsync times SteamVR reports lie on a 90 Hz grid.
- Each screen's callbacks come at a rate for how much of it you see (vr.cpp,
  UpdateAttention): every frame while focused (within 12 degrees of where your head points,
  a laser or the mouse on it in the last 1.5 s, carried, or typed on), 15 a second for the
  rest of what you see (within 60 degrees), and 1 a second when hidden, behind you, or
  paused. Levels rise at once and fall after 1.5 s (focused) or 0.5 s (in view). KWin draws
  a screen only after its callback and its apps wait for theirs, so this throttles the apps
  too. A screen where nothing changes costs nothing at any rate, as before.
- A video in view keeps every frame: 8 commits in a row that each redraw 6% or more of the
  screen, at 10 a second or more, count as one (from the surface's buffer damage).
- "rates F V H" / --rates set the three rates (default 0 15 1, 0 = every frame), "rates?"
  shows them and each screen's level, "watch S" gives everything full rate for S seconds for
  a remote viewer (vnc-bridge.sh renews it), and "phase MS" moves the ticks for tuning.
- ft-screens' main thread runs at nice -5 after the session starts: SteamOS allows down to
  -8 once the soft RLIMIT_NICE is raised, and KWin waits on these ticks. It had spent nearly
  3 times as long waiting to run as running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: skip unchanged work while the pointer is awake

Every frame (about 116 a second) the helper tested the cursor ray against every visible
overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and
visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the
mouse and the head still.

Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the
eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run
at least every 100 ms, since overlays move on their own (a floating window's controls follow
it), and always while dragging. The dots' setters go to SteamVR only when their value changes:
the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well
under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the
laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and
at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's
pose, a placement, or waking sends the next one regardless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Session: blur, background contrast and animations off by default

The nested kwinrc had no [Plugins] group, so KWin ran its default blur and background
contrast effects, and kdeglobals had no AnimationDurationFactor, so animations ran at
full length. KWin renders through zink on Turnip, on the GPU vrcompositor needs, and
blur re-renders what's behind every translucent panel and menu; each animation frame is
another frame for KWin and ft-screens.

Before KWin starts, the session script now writes [Plugins] blurEnabled=false and
contrastEnabled=false to $XDG_CONFIG_HOME/kwinrc and [KDE] AnimationDurationFactor=0 to
its kdeglobals, each only if the desktop's own file has no value for it. It does this
once and records that in $XDG_CONFIG_HOME/frametoprc ([Defaults] effects=1), because
System Settings deletes a key put back to its default: without the marker, turning blur
back on wouldn't survive a restart. The ids blur and contrast are the built-in effects
of KWin 6.2.5 on SteamOS (both enabled by default in its plugin metadata). Tested
against a temporary XDG_CONFIG_HOME: fresh config, an existing blurEnabled=true kept,
and a deleted key not rewritten.

README and docs/reference.md say how to turn them back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Session: don't autostart Discover's notifier or IBus in the desktop

The nested Plasma session runs the system's XDG autostart entries. Discover's update
notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) started
plasma-discover --mode update inside it, 520-620 MB resident and about 9% of a core,
with flatpak-system-helper and AppStream downloads behind it. IBus started a nested
ibus-daemon with kimpanel and ibus-extension-gtk3, which no app in the desktop can use:
KWin's input method is ft-textinput (zwp_input_method_v1, focus reports only; the VR
keyboard types through ft-screens' seat), and the session already drops QT_IM_MODULE,
GTK_IM_MODULE and XMODIFIERS. Nothing in Frametop talks to IBus.

Before Plasma starts, the session script copies both entries into
$XDG_CONFIG_HOME/autostart with Hidden=true, which plasma-session honours for that
desktop only. It does this once ([Defaults] autostart=1 in frametoprc) and skips a name
the user already has a file for, so deleting the copy brings the program back. The
geoclue demo agent stays (it answers apps' location requests outside GNOME and idles at
0%), and orca's entry is OnlyShowIn GNOME-family desktops, so it never ran. Tested
against a temporary XDG_CONFIG_HOME, including an existing user ibus.desktop left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: never block on the pointer helper's socket

The relay sent to @ft_pointer_helper on a blocking socket. When the helper stalled, a
layout placement or grabprobe holds it for seconds while ft-gazed keeps filling its socket at
90 Hz, the relay's one loop blocked with it: keyboards, the volume keys (which must never
reach gamescope), and pausing all stopped until the helper read again.

The socket is non-blocking now. A command the helper doesn't take (EAGAIN) waits in a queue,
and everything after it queues behind it so the order holds; tick() sends what it can on each
loop, and the select timeout drops to 20 ms while anything waits. Mouse moves add up into one
queued move. A scroll notch is dropped rather than queued, since scrolling seconds late is no
use; its release still goes. Presses, releases, show, hide, and the rest are kept, so no
button stays down. The queue holds at most 512 commands. While paused, the configured
pointer's queue still drains, so the releases and "hide" from standing down arrive. A
"vrbind" that hits a full socket is sent again on the next loop instead of being lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer driver: parse outside the lock, report only changes

Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and
RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands
(about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands
are now parsed into locals first, and the lock is held only to store the result.

RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every
frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to
SteamVR only when they change (all of them on the first frame). The pose still goes out every
frame while the device is connected, as a tracked device's should; the disconnected pose goes
out once. The helper now sends a pose only when it changes, so the comment says the driver
keeps the last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-powerd: ask SteamVR every 100 ms, not on every input event

The loop polled the input devices with a 100 ms timeout and then, on every wake, did
SteamVR's part: PollNextEvent, the headset's activity level and every device's pose, all
IPC calls to vrserver. Input wakes it at once so the displays come on with the first
key or motion, but a moving mouse sends hundreds of events a second, so moving the mouse
meant hundreds of rounds of IPC a second instead of 10.

Every wake still drains the input devices and the control socket and counts input as
use straight away; SteamVR's part, and the backlight read that goes with it, now run
only when 100 ms have passed since the last time, and poll sleeps until then. Built in
the dev container (power/build.sh, no warnings); not run, since the live ft-powerd holds
@ft_powerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: ft-eyegrab checks only the slot each camera writes next

While copying, ft-eyegrab woke every 300 us (about 1,500 to 3,000 times a second) and
fingerprinted all eight slots each time: 8 x 256 strided reads from DMA-BUF memory.

- Each look now checks only the slot each camera writes next. The order is known (camera 0
  3,0,1,2; camera 1 7,5,4,6,5,7,6,4), and the next slot follows from the last two; the table
  starts from those orders and learns from every frame, so a SteamVR update that changes
  them costs a few seconds of full scans, not frames.
- A camera with nothing in its expected slot 1.5 frames after its last one, or with no
  order yet, gets all four slots checked, as before. A frame that turns up in an unexpected
  slot means full scans for that camera for 2 s.
- A slot's fingerprint is taken again when it stops being one of the two in use, so a later
  check sees only a new frame. A frame is still passed on when its camera starts the frame
  after next.
- Between frames it sleeps until 2.5 ms before the next is due, then looks every 1 ms, with
  0.5 ms of timer slack (PR_SET_TIMERSLACK, --share only). With no frames from either
  camera for 0.5 s (headset off) it looks every 4 ms.
- --rec keeps its 0.3 ms polls (and the expected-slot checks), for its timestamps.

Tested offline by building poll_frames against simulated cameras that write each frame in
four bursts, the last after the next frame starts (6 s, both cameras): 1,076 frames passed
on, none torn or skipped, with the known orders and with camera 1 in a different order.
Wakeups 1,486/s -> 207/s, the poller's CPU 3.6% -> 0.8% of a core (in plain memory; the real
DMA-BUF reads cost more), and a frame's start is seen 1.35 ms after it begins on average
instead of 0.76. With a camera stalling 15 ms every 2 s, the old poller passed on 22 torn
frames and the new one 8 or fewer.

Built (glibc 2.38 symbols at most, the host has 2.39), not installed: it runs as root from
/etc/frametop, so it takes effect only after gaze/tracker/install.sh (sudo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: upload from the headset, then plug in and leave it

Export and upload are done in the headset now: the export page only notes
that VR may stutter a little. Upload opens the pull request first (a
draft) and shows its link, telling the person to plug in the headset and
leave it until it says Uploaded; the files then go to refs/pr/N, and the
pull request is marked open at the end. A retry of the same export goes on
in the same pull request. While an export or upload runs, a host unit holds
a logind sleep inhibitor so the Frame stays awake with the headset off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote Access: check the status every 5 s instead of every 2 s

While its window was open, Frametop Remote Access ran remote-ctl.sh status every 2 s,
and each run spawns bash, curl (the tailnet name from tailscaled) and python3 to parse
it. It now checks every 5 s, plus when the window comes to the front and once more 2 s
after turning remote access on or off or changing the password, so a change still
shows within a couple of seconds. A check doesn't start while one is still running.
Doing the check in-process would duplicate remote-ctl.sh's idea of "running", which
the session and the pause code share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* update-check: KWin's blur and contrast effect ids, retest hints

The session now turns KWin's blur and contrast effects off by id (blurEnabled and
contrastEnabled in the desktop's kwinrc), and a KWin that renamed them would quietly
leave them on. The check looks for their built-in factories (KWin::blur_factory,
KWin::contrast_factory) in kwin_wayland, which it already reads for --output-count,
and warns if one is gone. The kwin and plasma-workspace retest hints gain the blur and
the hidden autostart entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pause gesture: look the controllers up every 30 s, not every 3 s

The gesture reader fetched vrserver's /input/getstate.json over HTTP every 3 seconds, the
whole time the relay runs, to notice a controller's root path changing when the 3D mouse
takes or gives back its hand role.

It now looks them up when it connects, when a message comes from a device path it doesn't
know (at most every 3 s; the device is read from the message with two string searches, not
a JSON parse of all 160 a second), 1.5 s after the relay's 3D mouse connects or lets go (the
relay tells it through GamePause.controllers_changed), and otherwise every 30 s. The keys
test's pause stub gets the new method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: send mouse motion at most every 4 ms

The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams
a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf
and strncmp tests in the helper before reaching the move handler. In a 200 ms test at
1000 Hz, 149 reports now make 45 moves with the same total.

flush() on a report now sends only once 4 ms have passed since the last move; tick() sends
the rest when due, and the select timeout shrinks to match. Buttons and the gaze
keys still flush first, unconditionally, so a click lands where the pointer was. In the
helper, "move" is now tested first in the command dispatch, and its handling is one lambda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-gaze prints and reads only the sources in use

ft-gaze computed and printed all six sources for every sample: about 1.3 KB of JSON a line
with our tracker (practice2), 120 KB a second through podman's stdio relay for ft-gazed to
json.loads 90 times a second. It also read SteamVR's gaze action for every sample outside
games (UpdateActionState and GetEyeTrackingDataRelativeToNow, two calls into vrserver, 180 a
second), though with our tracker ft-gazed only uses own and mmap1.

- ft-gaze takes --sources LIST (action, mmap1, mmap2, left, right, own, and eye for the EYE
  object; all by default, so the probe and ft-eyes-session are unchanged), and with
  --watch-stdin a line "sources LIST" on stdin switches them. A source left out isn't read
  and prints as {"ok":0} ("eye" as null), so every line keeps the same keys. An older
  ft-gaze ignores both, and prints everything as before.
- ft-gazed asks for what it reads: own,mmap1 with our tracker; left,right,mmap1 with
  SteamVR's eyes; the source plus mmap1 and mmap2 on the older one-source path. While a
  check or the calibration runs or waits to open, all of them, since checks record every
  source (the calibration fits the action's correction too) and the fit check reads "eye".
  It switches as soon as that changes, well inside the check's 0.45 s settle.
- So the action is read only during checks, or with --source action.

On recorded samples, a line with own and mmap1 is about 700 bytes instead of 1,300
(practice2), and one with left, right and mmap1 about 550 instead of 940 (test1).
gaze/test/idle-test.py now checks that ft-gaze starts with every source for a check and is
then switched to those in use, without the action or own; all its checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: don't put a vanished panel back in the visibility map

The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-gaze's loop runs every 4 ms instead of 2

ft-gaze's loop slept 2 ms, so 500 times a second it read the head pose
(GetDeviceToAbsoluteTrackingPose), checked the eye tracker's counter, and drained SteamVR's
events, for samples that come 90 times a second.

It now sleeps 4 ms. A new sample is printed within 4 ms of appearing, 2 on average (was 1),
and the pose history still has a pose within 2 ms of any sample's time, which keeps the
head-pose error under 0.2 degrees for a head turning 100 degrees a second. Sleeping until
the next sample is due would have thinned the pose history to 11 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: the hidden panel waits for a command instead of waking every 50 ms

The calibration panel runs for as long as the gaze service does, hidden nearly all the time,
and it woke 20 to 30 times a second to look at its socket and SteamVR's events: about 0.9% of
a core, the main cost left with gaze idle.

It now waits in poll() on its command socket: up to a second while hidden, and up to 10 ms
while shown, as before (it still drains SteamVR's events each pass, so a quit is acknowledged
within a second while hidden). A command wakes it at once, so "show" draws sooner than
before. With --watch-stdin, its stdin closing wakes it as well, so stopping it doesn't wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: ft-screens announces new panels to the helper

The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel
made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard
the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens
now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds
it to its list at once (only frametop.* keys). An older helper ignores it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: fix Export doing nothing, and show it's busy at once

af2ea7c put _stay_awake between exportSession and its @Slot, so the
window's Export button called a method QML couldn't see. A new test checks
every backend call in main.qml against Backend's slots and properties.
Export and Upload now say Exporting…/Uploading… with a spinner the moment
they're pressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Lazy susan: Meta+Alt+Tab spins the panels around you

- ft-screens "spin next|prev|<degrees>": every unpinned screen and
  floating window turns together about a vertical axis through your
  head (0.3 s, eased), so the next panel on the right or left comes to
  straight ahead; the arrangement stays as it is. Taps during a spin
  add to it, from where the panels are headed; grabbing a panel or
  placing it (ft-layout, ft-floatd) takes it out of the spin
- when a spin settles, the panel in front gets the pointer (recenter),
  typing (as after a click), and KWin's active window: its floating
  window, or the top window on a screen (ft-floatd "front N", the KWin
  script's activate-output). KWin's outputs follow the screens'
  new places (ft-layout scale), as after a move
- the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and
  Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them.
  Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote
  desktop like RustDesk, and the relay would take the Mac's app
  switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS,
  Windows, and KDE

Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and
three or four floating windows, through RustDesk to a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: login command works from Frametop's Konsole

Frametop's Konsole sets XDG_RUNTIME_DIR=/run/user/UID/frametop, where podman finds
no container state, so 'distrobox enter dev -- hf auth login' failed with a crun
error. The command the Upload page shows now sets the real runtime folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: log in from the Upload page, three-step page, no terminal

The Upload page is now three numbered steps: choose the export, log in to
Hugging Face, upload. Log in runs hub.py login, huggingface_hub's browser
login (OAuth device code, as hf auth login does): the link opens in the
browser and the page shows the code to enter, with Copy code and Cancel.
hub.py saves the token; the window never sees one, and nobody pastes one.

The terminal upload and the login command are gone from the page, and
UPLOAD.md is now a short 'About uploading' under the steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: take.json keeps camera clock samples

sets.bin's capture_ns is CLOCK_MONOTONIC_RAW; poses.jsonl and prompts.jsonl are
CLOCK_MONOTONIC. On 2026-10-03 the two were 0.80 s apart during a session and
1.11 s apart five hours later, so images can't be paired with poses by
capture_ns. take.json now samples RAW minus MONOTONIC as each recording part
starts and stops (as ft-hands' raw_minus_mono_ns), so readers can put each
exposure on the poses' clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder export: no controller poses without controllers, nothing in deleted ranges

When the checklist says no controllers, exported poses.jsonl has left and right
null and feedback lines carry no controller state: controllers left switched on
still get tracked (one wandered 2 m in a real session) and would read as the
hands' ground truth. Poses and live-tracker feedback inside deleted ranges are
left out too, as the images there are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: final consent text (2026-10-03), residency check, installer

Consent 2026-10-03, after a non-lawyer review: who runs this and how to reach
them, the dataset is public (Hugging Face, possibly abroad), the Hugging Face
username shows next to the contributor id, purposes (no identification), safety,
the maintainer grant passes to whoever maintains Frametop next, withdrawal
before and after merge, rights such as the GDPR's, and what a new version means.
Residents of Illinois, Texas and Washington can't take part for now (biometric
privacy laws): a third checkbox, profile consent.region_ok, checked by
validate.py from this consent version on. The DRAFT banners are gone, so uploads
no longer need FT_HANDREC_ALLOW_UPLOAD.

hands/rec/install.sh installs the recorder on a Frame with Frametop: container
packages, hand tracking and panel builds, ft-camd's capabilities, menu entry.

test_qml_backend also checks each call's argument count against the slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: a reset button next to the grab bar, clickable in VR games

Each desktop screen gets a reset button left of its bar (a reticle). It
puts every screen back in its layout around where you are now, like
Meta+Shift+R (ft-layout apply).

In a VR game the screens leave the controllers to the game (the
outside_games and dashboard modes), so a controller couldn't click any
of their controls. Aiming a hand controller at the reset button now sets
MakeOverlaysInteractiveIfVisible on that button's overlay alone, so the
trigger clicks it; the flag clears half a second after the aim leaves a
zone twice as wide, and the game gets the controllers back. The aim
comes from the laser poses ft-screens already reads to show the controls.

The ft-layout spawn is now RunLayout(cmd), shared with the arrange.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Click stability: 32 logical pixels by default, not 8

8 is about 0.2 degrees on a 3.4 m wide 3440-pixel screen 2 m away, so a
trigger press turned into a drag unless the hand was very still. 32
(about 0.9 degrees) felt much better in the headset (2026-10-03).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: in games, pointing a controller at a panel turns its laser on

SteamVR's own floating windows take the laser while a controller points
at them in a game and give it back when it points away. Frametop's
panels didn't: with the controllers left to the game (outside_games, the
default, or dashboard), they couldn't be clicked without the dashboard.

ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or
floating window while a hand controller's laser pose meets it, its
controls, or its popups (UpdateAim; curved screens hit on their
cylinder), and clears it 0.3 s after the aim leaves a wider margin. A
drag or a held button keeps it on. The keyboard, one overlay, uses
ComputeOverlayIntersection and now follows the mode when a game starts
or ends while it's open. This replaces the reset button's own aim zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: find the controllers' laser tip during VR games too

GetComponentStateForDevicePath with no input source handle fails for
every render model component while a VR game runs (checked 2026-10-03
with a game up: all 21 components of frame_controller_right). TipOffset
then fell back to the controller's pose, which aims 40 degrees above the
Frame controller's laser. In games, pointing at a screen's middle missed
it and pointing below it hit, so the new aim-to-laser only worked from
the bottom; the controls' reveal and pin/roll aim were off the same way.
GetComponentState still answers then, with the same tip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* input-relay: Add mute key as volume key

Add KEY_MUTE as volume key. It will be mapped to KEY_MACRO28 and
use wpctl to toggle the mute of the default audio sink. The toggle of
the mute state will be done once when the key is pressed instead of
continuously toggling it when it is held down.
This allows the mute button on keyboards to work properly.

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>

* Session: bring back a taskbar saved on a screen the desktop doesn't have

Plasma 6.2.5 keeps a panel on a screen number and never moves one whose
number is past the screen count, so a taskbar saved on a spare output
(#18, lastScreen=8 with three screens) or on a screen a smaller layout
dropped stayed hidden. Before Plasma starts, session/fix-panels.py moves
such a panel and its tray's containment to screen 0 (the primary),
keeping its widgets, unless screen 0 already has a panel on that edge.
doctor.sh checks the panels' screens, and report.sh lists them with the
live outputs and panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* keys-test: the spin bindings (Meta+Alt+Tab, Meta+Alt+Shift+Tab), not while paused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Menu entries: own programs for Reset Screen Layout and Hide/Show Screens

Reset Screen Layout and Hide/Show Screens both ran ft-layout. Steam lists
entries by program, so Hide/Show launched Reset. Each gets a wrapper.

From PR #17 (only this part of 9618be8; its host_command change is for the
Nix packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: release a held button that can't come up on a screen

Pausing, or hiding the screen a button went down on, took the laser off
it mid-click; the pause gesture's second thumbstick click does that.
SteamVR's laser mouse then forgot the button ("Mouse down count is 1 but
states are all false"), no release came, and the catcher kept showing
whenever the pressing laser was off the panels, even while paused. Being
interactive, it kept the VR game's controllers from it until the
desktop restarted (2026-10-04, Beat Saber).

ft-screens now releases a held button when it's paused, when the screen
it went down on is hidden, or, during VR games, when the pressing hand
controller has held nothing for a second. GetControllerState answers
overlay apps only while a game runs; outside games a hold is never cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <codex@localhost>
Co-authored-by: CuriousJ <curious.j.tuber@gmail.com>
Co-authored-by: Patrick McDavid <fusionjunky@gmail.com>
Co-authored-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
Co-authored-by: John Murray <5672686+JRMurr@users.noreply.github.com>
2026-10-04 19:44:30 -06:00
JakeGreen2145 5e3323c188 [verified] merge experimental into nested AT-SPI fix 2026-10-04 17:38:33 -04:00
JakeGreen2145 25504e0ed4 [verified] fix(session): start nested AT-SPI registry 2026-10-04 16:13:03 -04:00
DeeJanuzandClaude Opus 5.5 ec870d2d7a Merge branch laser-release (release a held button that can't come up on a screen: the game kept losing its controllers) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:37:46 -06:00
DeeJanuzandClaude Opus 5.5 80797c98b9 Screens: release a held button that can't come up on a screen
Pausing, or hiding the screen a button went down on, took the laser off
it mid-click; the pause gesture's second thumbstick click does that.
SteamVR's laser mouse then forgot the button ("Mouse down count is 1 but
states are all false"), no release came, and the catcher kept showing
whenever the pressing laser was off the panels, even while paused. Being
interactive, it kept the VR game's controllers from it until the
desktop restarted (2026-10-04, Beat Saber).

ft-screens now releases a held button when it's paused, when the screen
it went down on is hidden, or, during VR games, when the pressing hand
controller has held nothing for a second. GetControllerState answers
overlay apps only while a game runs; outside games a hold is never cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:31:27 -06:00
DeeJanuzandClaude Opus 5.5 2fadbbe148 Merge branch orphan-panel (bring back a taskbar saved on a screen the desktop doesn't have, #18) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:42:12 -06:00
DeeJanuzandClaude Opus 5.5 5ee07f99a8 Merge branch menu-entry-programs (Hide/Show Screens no longer launches Reset Screen Layout) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:39:44 -06:00
John MurrayandClaude Opus 5.5 f0b93b79d3 Menu entries: own programs for Reset Screen Layout and Hide/Show Screens
Reset Screen Layout and Hide/Show Screens both ran ft-layout. Steam lists
entries by program, so Hide/Show launched Reset. Each gets a wrapper.

From PR #17 (only this part of 9618be8; its host_command change is for the
Nix packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:39:40 -06:00
DeeJanuzandClaude Opus 5.5 13a530d829 Merge branch mute-key (PR #24: the mute key toggles the default output's mute) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:44 -06:00
DeeJanuzandClaude Opus 5.5 f260a6a9e4 Merge branch lazy-susan (PR #22: Meta+Alt+Tab spins the panels around you) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:44 -06:00
DeeJanuzandClaude Opus 5.5 2e09cf9efe Merge PR #24 (SuperTuxii: the mute key toggles the default output's mute) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 9a4b66ff50 keys-test: the spin bindings (Meta+Alt+Tab, Meta+Alt+Shift+Tab), not while paused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 17269134ce Merge PR #22 (ehippy: lazy susan, Meta+Alt+Tab spins the panels around you) into experimental
Conflicts with experimental's pause_toggle, steam_menu and command: actions and its
AnnounceOverlay: both kept. The spin bindings join the Meta tap in the defaults, spinning
doesn't need pointer mode, and like other actions it does nothing while Frametop is paused.
AnnounceOverlay now sends through the PR's SendPointer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 d995b815f8 Session: bring back a taskbar saved on a screen the desktop doesn't have
Plasma 6.2.5 keeps a panel on a screen number and never moves one whose
number is past the screen count, so a taskbar saved on a spare output
(#18, lastScreen=8 with three screens) or on a screen a smaller layout
dropped stayed hidden. Before Plasma starts, session/fix-panels.py moves
such a panel and its tray's containment to screen 0 (the primary),
keeping its widgets, unless screen 0 already has a panel on that edge.
doctor.sh checks the panels' screens, and report.sh lists them with the
live outputs and panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:32:04 -06:00
SuperTuxii f18918f781 input-relay: Add mute key as volume key
Add KEY_MUTE as volume key. It will be mapped to KEY_MACRO28 and
use wpctl to toggle the mute of the default audio sink. The toggle of
the mute state will be done once when the key is pressed instead of
continuously toggling it when it is held down.
This allows the mute button on keyboards to work properly.

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
2026-10-04 16:06:37 +02:00
DeeJanuzandClaude Opus 5.5 e0208687b6 Merge branch tip-in-games (controller laser tip found during VR games) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:33:49 -06:00
DeeJanuzandClaude Opus 5.5 c699e13104 Screens: find the controllers' laser tip during VR games too
GetComponentStateForDevicePath with no input source handle fails for
every render model component while a VR game runs (checked 2026-10-03
with a game up: all 21 components of frame_controller_right). TipOffset
then fell back to the controller's pose, which aims 40 degrees above the
Frame controller's laser. In games, pointing at a screen's middle missed
it and pointing below it hit, so the new aim-to-laser only worked from
the bottom; the controls' reveal and pin/roll aim were off the same way.
GetComponentState still answers then, with the same tip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:33:49 -06:00
DeeJanuzandClaude Opus 5.5 9b12b7d74e Merge branch aim-lasers (in games, pointing a controller at a Frametop panel turns its laser on) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:27:49 -06:00
DeeJanuzandClaude Opus 5.5 c7c7c1f772 Screens: in games, pointing a controller at a panel turns its laser on
SteamVR's own floating windows take the laser while a controller points
at them in a game and give it back when it points away. Frametop's
panels didn't: with the controllers left to the game (outside_games, the
default, or dashboard), they couldn't be clicked without the dashboard.

ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or
floating window while a hand controller's laser pose meets it, its
controls, or its popups (UpdateAim; curved screens hit on their
cylinder), and clears it 0.3 s after the aim leaves a wider margin. A
drag or a held button keeps it on. The keyboard, one overlay, uses
ComputeOverlayIntersection and now follows the mode when a game starts
or ends while it's open. This replaces the reset button's own aim zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:27:49 -06:00
DeeJanuzandClaude Opus 5.5 2486a601e3 Merge branch click-threshold (controller click zone 32 px by default) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:49 -06:00
DeeJanuzandClaude Opus 5.5 de25633f87 Click stability: 32 logical pixels by default, not 8
8 is about 0.2 degrees on a 3.4 m wide 3440-pixel screen 2 m away, so a
trigger press turned into a drag unless the hand was very still. 32
(about 0.9 degrees) felt much better in the headset (2026-10-03).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:49 -06:00
DeeJanuzandClaude Opus 5.5 53a1836dbb Merge branch reset-button (a reset button next to each screen's grab bar, clickable in VR games) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:15:51 -06:00
DeeJanuzandClaude Opus 5.5 73bd0ea28f Screens: a reset button next to the grab bar, clickable in VR games
Each desktop screen gets a reset button left of its bar (a reticle). It
puts every screen back in its layout around where you are now, like
Meta+Shift+R (ft-layout apply).

In a VR game the screens leave the controllers to the game (the
outside_games and dashboard modes), so a controller couldn't click any
of their controls. Aiming a hand controller at the reset button now sets
MakeOverlaysInteractiveIfVisible on that button's overlay alone, so the
trigger clicks it; the flag clears half a second after the aim leaves a
zone twice as wide, and the game gets the controllers back. The aim
comes from the laser poses ft-screens already reads to show the controls.

The ft-layout spawn is now RunLayout(cmd), shared with the arrange.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:15:48 -06:00
DeeJanuzandClaude Opus 5.5 519c623ea9 Merge branch perf (dynamic per-screen frame rates, idle pointer, remote on demand, lighter gaze) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:10:34 -06:00
DeeJanuzandClaude Opus 5.5 8b1327fe1a Merge branch hand-recorder (the hand dataset recorder) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:35:37 -06:00
Patrick McDavidandClaude Opus 5.5 6fb168a8b8 Lazy susan: Meta+Alt+Tab spins the panels around you
- ft-screens "spin next|prev|<degrees>": every unpinned screen and
  floating window turns together about a vertical axis through your
  head (0.3 s, eased), so the next panel on the right or left comes to
  straight ahead; the arrangement stays as it is. Taps during a spin
  add to it, from where the panels are headed; grabbing a panel or
  placing it (ft-layout, ft-floatd) takes it out of the spin
- when a spin settles, the panel in front gets the pointer (recenter),
  typing (as after a click), and KWin's active window: its floating
  window, or the top window on a screen (ft-floatd "front N", the KWin
  script's activate-output). KWin's outputs follow the screens'
  new places (ft-layout scale), as after a move
- the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and
  Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them.
  Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote
  desktop like RustDesk, and the relay would take the Mac's app
  switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS,
  Windows, and KDE

Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and
three or four floating windows, through RustDesk to a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 10:17:22 -06:00
DeeJanuz b67c973f64 Merge branch perf-gaze into perf 2026-10-03 09:28:44 -06:00
DeeJanuzandClaude Opus 5.5 99aec84f62 Pointer: ft-screens announces new panels to the helper
The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel
made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard
the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens
now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds
it to its list at once (only frametop.* keys). An older helper ignores it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:28:10 -06:00
DeeJanuzandClaude Opus 5.5 e44837cee6 Gaze: the hidden panel waits for a command instead of waking every 50 ms
The calibration panel runs for as long as the gaze service does, hidden nearly all the time,
and it woke 20 to 30 times a second to look at its socket and SteamVR's events: about 0.9% of
a core, the main cost left with gaze idle.

It now waits in poll() on its command socket: up to a second while hidden, and up to 10 ms
while shown, as before (it still drains SteamVR's events each pass, so a quit is acknowledged
within a second while hidden). A command wakes it at once, so "show" draws sooner than
before. With --watch-stdin, its stdin closing wakes it as well, so stopping it doesn't wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:28:02 -06:00
DeeJanuzandClaude Opus 5.5 95c5d056cf Gaze: ft-gaze's loop runs every 4 ms instead of 2
ft-gaze's loop slept 2 ms, so 500 times a second it read the head pose
(GetDeviceToAbsoluteTrackingPose), checked the eye tracker's counter, and drained SteamVR's
events, for samples that come 90 times a second.

It now sleeps 4 ms. A new sample is printed within 4 ms of appearing, 2 on average (was 1),
and the pose history still has a pose within 2 ms of any sample's time, which keeps the
head-pose error under 0.2 degrees for a head turning 100 degrees a second. Sleeping until
the next sample is due would have thinned the pose history to 11 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:26:54 -06:00
DeeJanuz e6a931f7ee Merge branch perf-pointer into perf 2026-10-03 09:26:18 -06:00
DeeJanuzandClaude Opus 5.5 60667dba1f Pointer: don't put a vanished panel back in the visibility map
The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:25:16 -06:00
DeeJanuzandClaude Opus 5.5 0c95d1d731 Gaze: ft-gaze prints and reads only the sources in use
ft-gaze computed and printed all six sources for every sample: about 1.3 KB of JSON a line
with our tracker (practice2), 120 KB a second through podman's stdio relay for ft-gazed to
json.loads 90 times a second. It also read SteamVR's gaze action for every sample outside
games (UpdateActionState and GetEyeTrackingDataRelativeToNow, two calls into vrserver, 180 a
second), though with our tracker ft-gazed only uses own and mmap1.

- ft-gaze takes --sources LIST (action, mmap1, mmap2, left, right, own, and eye for the EYE
  object; all by default, so the probe and ft-eyes-session are unchanged), and with
  --watch-stdin a line "sources LIST" on stdin switches them. A source left out isn't read
  and prints as {"ok":0} ("eye" as null), so every line keeps the same keys. An older
  ft-gaze ignores both, and prints everything as before.
- ft-gazed asks for what it reads: own,mmap1 with our tracker; left,right,mmap1 with
  SteamVR's eyes; the source plus mmap1 and mmap2 on the older one-source path. While a
  check or the calibration runs or waits to open, all of them, since checks record every
  source (the calibration fits the action's correction too) and the fit check reads "eye".
  It switches as soon as that changes, well inside the check's 0.45 s settle.
- So the action is read only during checks, or with --source action.

On recorded samples, a line with own and mmap1 is about 700 bytes instead of 1,300
(practice2), and one with left, right and mmap1 about 550 instead of 940 (test1).
gaze/test/idle-test.py now checks that ft-gaze starts with every source for a check and is
then switched to those in use, without the action or own; all its checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:23:53 -06:00
DeeJanuzandClaude Opus 5.5 fcb465a45a Input relay: send mouse motion at most every 4 ms
The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams
a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf
and strncmp tests in the helper before reaching the move handler. In a 200 ms test at
1000 Hz, 149 reports now make 45 moves with the same total.

flush() on a report now sends only once 4 ms have passed since the last move; tick() sends
the rest when due, and the select timeout shrinks to match. Buttons and the gaze
keys still flush first, unconditionally, so a click lands where the pointer was. In the
helper, "move" is now tested first in the command dispatch, and its handling is one lambda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:23:47 -06:00
DeeJanuz d6930a61f5 Merge branch perf-session into perf 2026-10-03 09:22:07 -06:00
DeeJanuz 155deada52 Merge branch perf-screens into perf 2026-10-03 09:22:07 -06:00
DeeJanuzandClaude Opus 5.5 597551cf23 Pause gesture: look the controllers up every 30 s, not every 3 s
The gesture reader fetched vrserver's /input/getstate.json over HTTP every 3 seconds, the
whole time the relay runs, to notice a controller's root path changing when the 3D mouse
takes or gives back its hand role.

It now looks them up when it connects, when a message comes from a device path it doesn't
know (at most every 3 s; the device is read from the message with two string searches, not
a JSON parse of all 160 a second), 1.5 s after the relay's 3D mouse connects or lets go (the
relay tells it through GamePause.controllers_changed), and otherwise every 30 s. The keys
test's pause stub gets the new method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:21:09 -06:00
DeeJanuzandClaude Opus 5.5 33fb3bb611 update-check: KWin's blur and contrast effect ids, retest hints
The session now turns KWin's blur and contrast effects off by id (blurEnabled and
contrastEnabled in the desktop's kwinrc), and a KWin that renamed them would quietly
leave them on. The check looks for their built-in factories (KWin::blur_factory,
KWin::contrast_factory) in kwin_wayland, which it already reads for --output-count,
and warns if one is gone. The kwin and plasma-workspace retest hints gain the blur and
the hidden autostart entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:21:09 -06:00
DeeJanuzandClaude Opus 5.5 6cae2e0afe Remote Access: check the status every 5 s instead of every 2 s
While its window was open, Frametop Remote Access ran remote-ctl.sh status every 2 s,
and each run spawns bash, curl (the tailnet name from tailscaled) and python3 to parse
it. It now checks every 5 s, plus when the window comes to the front and once more 2 s
after turning remote access on or off or changing the password, so a change still
shows within a couple of seconds. A check doesn't start while one is still running.
Doing the check in-process would duplicate remote-ctl.sh's idea of "running", which
the session and the pause code share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:20:32 -06:00
DeeJanuzandClaude Opus 5.5 060b4957f8 Eye tracker: ft-eyegrab checks only the slot each camera writes next
While copying, ft-eyegrab woke every 300 us (about 1,500 to 3,000 times a second) and
fingerprinted all eight slots each time: 8 x 256 strided reads from DMA-BUF memory.

- Each look now checks only the slot each camera writes next. The order is known (camera 0
  3,0,1,2; camera 1 7,5,4,6,5,7,6,4), and the next slot follows from the last two; the table
  starts from those orders and learns from every frame, so a SteamVR update that changes
  them costs a few seconds of full scans, not frames.
- A camera with nothing in its expected slot 1.5 frames after its last one, or with no
  order yet, gets all four slots checked, as before. A frame that turns up in an unexpected
  slot means full scans for that camera for 2 s.
- A slot's fingerprint is taken again when it stops being one of the two in use, so a later
  check sees only a new frame. A frame is still passed on when its camera starts the frame
  after next.
- Between frames it sleeps until 2.5 ms before the next is due, then looks every 1 ms, with
  0.5 ms of timer slack (PR_SET_TIMERSLACK, --share only). With no frames from either
  camera for 0.5 s (headset off) it looks every 4 ms.
- --rec keeps its 0.3 ms polls (and the expected-slot checks), for its timestamps.

Tested offline by building poll_frames against simulated cameras that write each frame in
four bursts, the last after the next frame starts (6 s, both cameras): 1,076 frames passed
on, none torn or skipped, with the known orders and with camera 1 in a different order.
Wakeups 1,486/s -> 207/s, the poller's CPU 3.6% -> 0.8% of a core (in plain memory; the real
DMA-BUF reads cost more), and a frame's start is seen 1.35 ms after it begins on average
instead of 0.76. With a camera stalling 15 ms every 2 s, the old poller passed on 22 torn
frames and the new one 8 or fewer.

Built (glibc 2.38 symbols at most, the host has 2.39), not installed: it runs as root from
/etc/frametop, so it takes effect only after gaze/tracker/install.sh (sudo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:20:00 -06:00
DeeJanuzandClaude Opus 5.5 57617d4a14 ft-powerd: ask SteamVR every 100 ms, not on every input event
The loop polled the input devices with a 100 ms timeout and then, on every wake, did
SteamVR's part: PollNextEvent, the headset's activity level and every device's pose, all
IPC calls to vrserver. Input wakes it at once so the displays come on with the first
key or motion, but a moving mouse sends hundreds of events a second, so moving the mouse
meant hundreds of rounds of IPC a second instead of 10.

Every wake still drains the input devices and the control socket and counts input as
use straight away; SteamVR's part, and the backlight read that goes with it, now run
only when 100 ms have passed since the last time, and poll sleeps until then. Built in
the dev container (power/build.sh, no warnings); not run, since the live ft-powerd holds
@ft_powerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:19:49 -06:00
DeeJanuzandClaude Opus 5.5 83850bb1b7 Pointer driver: parse outside the lock, report only changes
Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and
RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands
(about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands
are now parsed into locals first, and the lock is held only to store the result.

RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every
frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to
SteamVR only when they change (all of them on the first frame). The pose still goes out every
frame while the device is connected, as a tracked device's should; the disconnected pose goes
out once. The helper now sends a pose only when it changes, so the comment says the driver
keeps the last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:19:38 -06:00
DeeJanuzandClaude Opus 5.5 d5a15ebc36 Input relay: never block on the pointer helper's socket
The relay sent to @ft_pointer_helper on a blocking socket. When the helper stalled, a
layout placement or grabprobe holds it for seconds while ft-gazed keeps filling its socket at
90 Hz, the relay's one loop blocked with it: keyboards, the volume keys (which must never
reach gamescope), and pausing all stopped until the helper read again.

The socket is non-blocking now. A command the helper doesn't take (EAGAIN) waits in a queue,
and everything after it queues behind it so the order holds; tick() sends what it can on each
loop, and the select timeout drops to 20 ms while anything waits. Mouse moves add up into one
queued move. A scroll notch is dropped rather than queued, since scrolling seconds late is no
use; its release still goes. Presses, releases, show, hide, and the rest are kept, so no
button stays down. The queue holds at most 512 commands. While paused, the configured
pointer's queue still drains, so the releases and "hide" from standing down arrive. A
"vrbind" that hits a full socket is sent again on the next loop instead of being lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:18:43 -06:00
DeeJanuzandClaude Opus 5.5 b7dfe4759e Session: don't autostart Discover's notifier or IBus in the desktop
The nested Plasma session runs the system's XDG autostart entries. Discover's update
notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) started
plasma-discover --mode update inside it, 520-620 MB resident and about 9% of a core,
with flatpak-system-helper and AppStream downloads behind it. IBus started a nested
ibus-daemon with kimpanel and ibus-extension-gtk3, which no app in the desktop can use:
KWin's input method is ft-textinput (zwp_input_method_v1, focus reports only; the VR
keyboard types through ft-screens' seat), and the session already drops QT_IM_MODULE,
GTK_IM_MODULE and XMODIFIERS. Nothing in Frametop talks to IBus.

Before Plasma starts, the session script copies both entries into
$XDG_CONFIG_HOME/autostart with Hidden=true, which plasma-session honours for that
desktop only. It does this once ([Defaults] autostart=1 in frametoprc) and skips a name
the user already has a file for, so deleting the copy brings the program back. The
geoclue demo agent stays (it answers apps' location requests outside GNOME and idles at
0%), and orca's entry is OnlyShowIn GNOME-family desktops, so it never ran. Tested
against a temporary XDG_CONFIG_HOME, including an existing user ibus.desktop left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:18:31 -06:00
DeeJanuzandClaude Opus 5.5 5b6cfcd746 Session: blur, background contrast and animations off by default
The nested kwinrc had no [Plugins] group, so KWin ran its default blur and background
contrast effects, and kdeglobals had no AnimationDurationFactor, so animations ran at
full length. KWin renders through zink on Turnip, on the GPU vrcompositor needs, and
blur re-renders what's behind every translucent panel and menu; each animation frame is
another frame for KWin and ft-screens.

Before KWin starts, the session script now writes [Plugins] blurEnabled=false and
contrastEnabled=false to $XDG_CONFIG_HOME/kwinrc and [KDE] AnimationDurationFactor=0 to
its kdeglobals, each only if the desktop's own file has no value for it. It does this
once and records that in $XDG_CONFIG_HOME/frametoprc ([Defaults] effects=1), because
System Settings deletes a key put back to its default: without the marker, turning blur
back on wouldn't survive a restart. The ids blur and contrast are the built-in effects
of KWin 6.2.5 on SteamOS (both enabled by default in its plugin metadata). Tested
against a temporary XDG_CONFIG_HOME: fresh config, an existing blurEnabled=true kept,
and a deleted key not rewritten.

README and docs/reference.md say how to turn them back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:17:44 -06:00
DeeJanuzandClaude Opus 5.5 e1f7ccee29 Pointer: skip unchanged work while the pointer is awake
Every frame (about 116 a second) the helper tested the cursor ray against every visible
overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and
visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the
mouse and the head still.

Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the
eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run
at least every 100 ms, since overlays move on their own (a floating window's controls follow
it), and always while dragging. The dots' setters go to SteamVR only when their value changes:
the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well
under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the
laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and
at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's
pose, a placement, or waking sends the next one regardless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:16:57 -06:00
DeeJanuzandClaude Opus 5.5 d8c2ed0c58 Screens: frame rates by attention, ticks in step with the display
ft-screens gave KWin a frame callback for every committed screen on each tick, and the tick
was an 11 ms timer set again after each run, so it slid through the display's frame and came
about 85 times a second at 90 Hz: the desktop repeated a frame several times a second (judder
in scrolling and video), and KWin drew every screen in one burst at a random point of
vrcompositor's frame. Hidden screens got the same 90 Hz unless Frametop was paused for a game.

- Ticks run on a timerfd at absolute times, once per display frame, 1 ms after the vsync
  (IVRSystem::GetTimeSinceLastVsync and the HMD's display frequency, read once a second), so
  KWin gets its callbacks early in the frame. Measured with --no-vr: 91 wakeups a second
  instead of about 85. On the Frame the vsync times SteamVR reports lie on a 90 Hz grid.
- Each screen's callbacks come at a rate for how much of it you see (vr.cpp,
  UpdateAttention): every frame while focused (within 12 degrees of where your head points,
  a laser or the mouse on it in the last 1.5 s, carried, or typed on), 15 a second for the
  rest of what you see (within 60 degrees), and 1 a second when hidden, behind you, or
  paused. Levels rise at once and fall after 1.5 s (focused) or 0.5 s (in view). KWin draws
  a screen only after its callback and its apps wait for theirs, so this throttles the apps
  too. A screen where nothing changes costs nothing at any rate, as before.
- A video in view keeps every frame: 8 commits in a row that each redraw 6% or more of the
  screen, at 10 a second or more, count as one (from the surface's buffer damage).
- "rates F V H" / --rates set the three rates (default 0 15 1, 0 = every frame), "rates?"
  shows them and each screen's level, "watch S" gives everything full rate for S seconds for
  a remote viewer (vnc-bridge.sh renews it), and "phase MS" moves the ticks for tuning.
- ft-screens' main thread runs at nice -5 after the session starts: SteamOS allows down to
  -8 once the soft RLIMIT_NICE is raised, and KWin waits on these ticks. It had spent nearly
  3 times as long waiting to run as running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:16:46 -06:00
DeeJanuzandClaude Opus 5.5 7851ce90cc Eye tracker: ft-eyes sleeps until the next frame is due
ft-eyes looked for new frames about 1,000 times a second: each pass of its loop asked the
control socket with a non-blocking recvfrom (a BlockingIOError nearly every time), read
both cameras' counters, and slept 1 ms. The frames come every 11.1 ms per camera, and only
as counters in ft-eyegrab's shared memory, so there's no fd to wait on.

Now each pass ends in select() on the control socket, with a timeout until 2 ms before the
next frame of either camera is due (from when its last one was seen), then every 1 ms until
it comes. A command wakes it at once. A camera with no frame for 0.1 s (headset off,
grabber idle) isn't waited for, and with both stopped it looks every 20 ms. Waiting for the
frame grabber's file uses the same select, 0.2 s at a time, instead of sleeping through
commands.

A new frame is still seen within about 1 ms of when it lands. On a synthetic share at 90 Hz
per camera, on a heavily loaded headset (load average 23, so ft-eyes rarely sat idle), its
waits went from 178 to 81 a second; unloaded, the old loop's 1 ms sleeps add up to about
1,000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:15:27 -06:00
DeeJanuzandClaude Opus 5.5 e1ee5ef395 Remote desktop: read the layout only after it changes
While FreeRDP ran, vnc-bridge.sh called ft-layout remote-view every 5 s, which scans
all of /proc for plasmashell and runs kscreen-doctor -j: about 4.4% of a core for a
layout that rarely changes.

It now stats the two files the answer depends on, the nested KWin's
~/.config/frametop/kwinoutputconfig.json (positions, scales, primary) and
~/.config/frametop-layout.json (screen sizes), once a second while FreeRDP runs. After
either changes it reads the layout every 2 s for 10 s, since KWin's outputs follow the
file a few seconds later; otherwise once a minute, in case a change touched neither.
With no VNC viewer connected nothing runs (previous commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:14:43 -06:00
DeeJanuzandClaude Opus 5.5 3e7248a04e Remote desktop: connect FreeRDP only while a VNC viewer is connected
vnc-bridge.sh kept FreeRDP connected to krdpserver from the moment remote desktop
started, so krdp captured and H.264-encoded every KWin redraw in software (openh264)
with nobody watching: krdpserver 55-78% of a core, xfreerdp 16-27%, Xvnc 6-11%, with 0
clients on :5900. krdp 6.7 creates its screencast session per RDP connection and drops
it when the connection closes, so krdpserver itself idles without one and stays up.

The bridge now counts established connections to Xvnc's port with ss, starts FreeRDP
when a viewer appears (the desktop shows about 3 s later; the VNC screen is black until
then) and stops it 45 s after the last one leaves (VNC_IDLE_SEC). Xvnc has no client
hook, so its log output, which it writes for every connection, wakes the bridge early;
otherwise it looks every 5 s while idle (0.1% of a core measured, against 0.9% for ss
once a second) and every second while FreeRDP runs. The layout check runs only while
FreeRDP runs.

While a viewer is connected the bridge sends "watch 15" to ft-screens (@ft_screens) at
once and every 5 s, so screens at a reduced frame rate (out of view, headset on a
stand) stream at full rate; it lapses by itself if the bridge dies, and an ft-screens
without the command just answers an error. The window search after starting FreeRDP
now ends when FreeRDP exits instead of polling for 30 s.

krdp on 127.0.0.1 with a fresh password, VNC on the tailnet address with VncAuth, and
remote-ctl.sh start/stop (pause and resume) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:14:25 -06:00
DeeJanuzandClaude Opus 5.5 0680297efa Pointer: sleep on the command socket while the pointer is off
The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake
or not, and every second it looked up every overlay's handle and read a string property from
all 64 device slots to find its own device.

With the pointer off and hand gestures off, the loop now waits in poll() on its command
socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read
(SteamVR input has no event to wait for). A mouse command ends the wait at once. The
headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep
going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the
pointer is awake, and waking forces both. The device index is looked for only while it's
unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is
kept only with hand gestures on, its one user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:12:33 -06:00
DeeJanuzandClaude Opus 5.5 a11cef49ba Gaze: ft-eyes and ft-gaze below SteamVR's priority
ft-eyes and ft-gaze run in the dev container through distrobox, so they live in podman's
libpod scope: frametop-gaze.service's limits never reach them, and they ran at nice 0
next to vrcompositor and vrserver, also at nice 0.

- ft-eyes sets itself to nice 10 and SCHED_BATCH at start, before its threads. Batch
  turns off wakeup preemption, so a frame ft-eyes wakes up for can wait out a running
  compositor's turn; a few ms late costs the gaze little.
- ft-gaze sets nice 5 before its threads start, but stays SCHED_OTHER: each sample goes
  on to the pointer, and batch would add the same wait to every one of them.
- Both only ever lower their priority (a higher nice already set wins), and a failure
  is logged and ignored. Checked in the dev container: nice 0 -> 10, policy 0 -> 3
  (SCHED_BATCH) without any capability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:11:15 -06:00
DeeJanuzandClaude Opus 5.5 19032f8963 Pointer: read the overlay list every 20 s, not every second
The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze
mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about
26 to 30 ms of CPU, so about 3% of a core all the time.

The list is now read every 20 seconds, and at once (at most once a second) when it may have
changed: the pointer waking, the dashboard opening or closing or creating an overlay, the
scene app changing, an "overlays" request, and a left click that hit nothing, which may be
on a panel that came up since. The thread waits on a condition variable instead of waking
every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a
new list is in, rather than at its next 1 s tick. Overlays already on the list still show
and hide within 50 ms, from the IsOverlayVisible poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:10:30 -06:00
DeeJanuzandClaude Opus 5.5 8a02e41b21 Eye tracker: one thread for OpenCV and numpy
Nothing called cv2.setNumThreads, so OpenCV kept a pool of one worker per core for
pupil windows of 140 to 240 px. Live, its idle workers spun and yielded about 14,000
times a second each, about a quarter of a core, next to SteamVR's compositor. numpy's
OpenBLAS also started 8 threads that never had work.

eyes_pupil.py now sets OpenCV to one thread, and ft-eyes sets OPENBLAS_NUM_THREADS and
OMP_NUM_THREADS to 1 before numpy loads (a value already in the environment wins).

Replaying fit1 into a scratch share (ft-eyes-replay, 14 s measured, capped at one core
with the replay): threads 13 -> 1, involuntary context switches 3,812/s -> 430/s,
system time 6.9% -> 1.6% of a core. Under that cap the frames it kept up with went from
21-36 to 57-69 a second per eye.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:08:16 -06:00
DeeJanuzandClaude Opus 5.5 06c4ff9556 Merge branch game-pause (Game optimization page) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:47:44 -06:00
DeeJanuzandClaude Opus 5.5 8760ca3083 Input Settings: the Games page is Game optimization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:47:44 -06:00
DeeJanuzandClaude Opus 5.5 76e5c4932e Merge branch game-pause (update-check: still controllers) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:37:18 -06:00
DeeJanuzandClaude Opus 5.5 ae4a1e30ab update-check: a still controller isn't a broken web socket
vrserver sends a controller's state only when something on it changes, and one lying still
or asleep may not even send its first one. The check subscribed to one controller and failed
after 3 s of silence. It now subscribes to all, and silence after a good handshake is a skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:37:15 -06:00
DeeJanuzandClaude Opus 5.5 b0415cf150 Merge branch game-pause (pause Frametop for VR games, gaze idle) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:34:50 -06:00
DeeJanuzandClaude Opus 5.5 5b08e43a5d Gaze: idle while the gaze isn't used
The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes
still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze
and our tracker run only while gaze mode is on and someone wears the headset, while a check or
the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop
Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber
idles with our tracker.

- The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the
  headset); an older helper answers it as before, and the service then goes by gaze mode alone.
- A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once
  it sends; the automatic calibration waits quietly while it starts.
- Status has "awake" and "idle" (why), and the Gaze page shows it.
- gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:32:49 -06:00
DeeJanuzandClaude Opus 5.5 8aaf7db05a Pause Frametop for VR games
Frametop kept using the headset during games: with gaze mode off, our eye tracker still took
about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden
screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and
resuming brings back only what pausing stopped. It's also a way to keep the gaze service and
our eye tracker off during games, which PR #13 asked for.

Paused (input/game_pause.py, run by the input relay):
- frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop
  stop if they run
- the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a
  frame callback once a second; or, with pause_desktop "close", the desktop closes and starts
  again on resume
- the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key
  combinations do only pause_toggle, steam_menu and commands

Toggled by both thumbsticks clicked together twice (configurable), read passively from
vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the
new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game
starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input
Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a
paused gaze service as failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:58:09 -06:00
DeeJanuzandClaude Opus 5.5 7fbcd177d3 Input relay test: let the fake devices past the udev permission check
Since the relay leaves a node it can't read yet for the next scan (12f2e84, PR #12), it
checks os.access first, and the test's fake /dev/input paths don't exist, so the relay never
opened them and every key check failed. The fake os now says they're readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:58:09 -06:00
Patrick McDavidandClaude Opus 5.5 1ebf3692fb ft-floatd: a launch for a missing app no longer kills the control socket (#16)
Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.

Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:12:49 -06:00
DeeJanuzandClaude Opus 5.5 554820ffc5 ft-floatd: profiles keep and relaunch Flatpak apps whose window names another app id
An X11 window in a Flatpak can give KWin an app id with no desktop file:
RustDesk's says com.carriez.flutter_hbb (its GTK application id), and the
Flatpak's desktop file is com.rustdesk.RustDesk. A profile recorded that
id, so it couldn't relaunch the app (PR #16 keeps that from killing
ft-floatd's socket). And the window's pid is the sandbox's own, so a
launched window matched neither by process nor by app id, and didn't
float.

desktop_name() finds the desktop file whose StartupWMClass names the
window's class (or app id) when the app id has none. Capture records
that name, a profile claims open windows by it, and a launch's window
matches by it.

Profiles saved before this keep the old id; save them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:40:45 -06:00
Patrick McDavidandClaude Opus 5.5 bdc84b24d4 ft-floatd: a launch for a missing app no longer kills the control socket
Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.

Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:39:33 -06:00
DeeJanuzandClaude Opus 5.5 072a294941 README: Frametop doesn't work on the SteamOS beta yet
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 16:12:52 -06:00
DeeJanuz abb05eb68c README: Frametop doesn't work on the SteamOS beta yet
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 072a294941)
2026-10-02 16:12:52 -06:00
DeeJanuzandClaude Opus 5.5 7816633353 README: link the Frametop Discord
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 21:18:09 -06:00
74 changed files with 6376 additions and 576 deletions

No files matched your search

+1 -1
View File
@@ -26,7 +26,7 @@ A Steam Frame is someone's personal headset, and they may be wearing it while yo
- Don't kill or restart `gamescope`, `steam`, `vrserver`, `vrcompositor`, the gamescope session, or the Frametop desktop without asking. Each one ends or disrupts whatever is happening in VR.
- Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Three installers need host `sudo`, and they ask for it: the Bluetooth fixes (`setup/bluetooth/install.sh`), hand tracking (`hands/run.sh install` and `caps`, which set ft-camd's file capabilities with `setcap`), and our own eye tracker's frame grabber (`gaze/tracker/install.sh`).
- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`).
- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`). When an installer starts writing something new outside the repo, add it to `uninstall.sh` too: users uninstall with that script, not with each installer's `uninstall`.
- Never copy `.netrc`, SSH keys, or Steam config off the Frame or into this repo.
## SteamOS updates
+34 -18
View File
@@ -17,6 +17,8 @@ Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas,
## Install on the headset
> **Frametop doesn't work on the SteamOS beta right now.** On the beta (SteamOS 0.4.3), gaze mode can't read the eye tracker, and the desktop has started without its taskbar ([#15](https://github.com/DeeJanuz/frametop/issues/15)). Use the stable SteamOS release until this note is gone.
You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
1. In the launcher, choose Launch a program → Desktop.
@@ -55,10 +57,11 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
| While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off |
| Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
| Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
| Meta+Alt+Tab, or Meta+Alt+Shift+Tab | Spins every screen and floating window around you together, like a lazy susan, so the next one on your right (or left) glides to straight ahead, with the pointer and typing going to it. Their arrangement stays the same: the room turns instead of you. Tap again to keep going; Meta+Shift+R puts the screens back. Pinned screens stay where they are |
| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
| Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
| Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back. With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab |
Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
@@ -93,6 +96,18 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh
| Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes |
| Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings |
### Pause for VR games
Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU, and comes back a few seconds after the game ends. Paused, the screens hide and the desktop nearly stops drawing, but its windows stay open. Gaze mode's eye tracking stops, and so do remote desktop and hand tracking if they run. The mouse works as a plain mouse in SteamVR.
| Do this | To get this |
| --- | --- |
| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too |
| Start a VR game | Frametop pauses, and comes back 5 seconds after the game ends. Bring it back during the game, and it stays on until that game ends |
| Map Pause/resume Frametop to a mouse button, key combination, or controller button | The same, from that button (Frametop Input Settings) |
The Game optimization page of Frametop Input Settings turns the automatic pause off, changes the gesture, closes the desktop instead of hiding it (more for the game, but its windows close), and turns the sound off. From a terminal: `input/ft-pause on`, `off`, or `status`.
### Gaze mode (experimental)
In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
@@ -122,15 +137,16 @@ This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 2026092
- A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
- The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
- During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
- Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
- During a VR game, mapped controller buttons belong to the game, so they can't bring the screens up. The pause gesture still works: Frametop reads it without taking the thumbsticks from the game. With the automatic pause off, open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
- Flatscreen games aren't detected as games, so they don't pause Frametop by themselves: click both thumbsticks twice to pause it. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
- Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back.
- The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing.
- Profiles reopen apps, not what the apps had open. Tabs, files, and folders are left to each app's own restore.
- Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels.
- Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings.
- On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md.
- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart.
- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. It costs almost nothing until a viewer connects; the picture then takes a few seconds to appear.
- The desktop has no blur behind panels and menus, and no window animations, so it leaves the headset's GPU to SteamVR. Turn them back on in the Frametop desktop's System Settings (Desktop Effects, and Animation speed under General Behavior); Frametop won't turn them off again.
- Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power.
## Reporting problems
@@ -155,23 +171,22 @@ Or by hand: `cd ~/frametop && git pull && ./install.sh`.
## Uninstall
In a terminal on the headset, run:
```
./desktops.sh uninstall # the launcher's Desktop entry goes back to the stock desktop
./desktops.sh relay uninstall
pointer/helper/run.sh uninstall
power/run.sh uninstall
pointer/driver/install.sh uninstall # then restart SteamVR
input-settings/install.sh uninstall
display-settings/install.sh uninstall
remote/install.sh uninstall
setup/bluetooth/install.sh uninstall # if you installed the Bluetooth fixes
hands/run.sh uninstall # if you installed hand tracking by hand
gaze/run.sh uninstall # if you installed the gaze service
gaze/tracker/install.sh uninstall # if you installed our own eye tracker's frame grabber
gaze/probe/install.sh uninstall # if you installed the gaze probe
curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
```
Your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, and the gaze calibration in `~/.local/state/frametop/gaze`. So does the desktop's own Plasma setup, in `~/.config/frametop`. Delete them too for a clean slate.
It works in two steps, so it never takes away the keyboard, mouse, or desktop you're using while it runs:
1. It stops Frametop from starting. Launch a program → Desktop opens the stock desktop again, and Frametop's services, its SteamVR driver, its menu entries, and the system files of our eye tracker and the Bluetooth fixes are removed (those need your `sudo` password). Everything running now keeps running until you restart the headset, and it offers to restart it for you.
2. After the restart, run the same command again. It deletes the code in `~/frametop`, and asks whether to delete your settings, any eye or hand recordings, and the build container (1–2 GB) too.
To see what it would do without changing anything, add `-s -- --dry-run` after `bash`. If the code isn't in `~/frametop`, add `-s -- --dir <folder>`. From the repo, the same script is `./uninstall.sh`.
Don't delete `~/frametop` by hand before you uninstall and restart: the desktop and the input relay run from it, and without it Launch a program → Desktop no longer opens anything. If you've already deleted it, the command above still works, since it doesn't need the repo.
Unless you ask for them to go, your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, the gaze calibration in `~/.local/state/frametop`, and the desktop's own Plasma setup in `~/.config/frametop`. A later install picks them up again.
## How it works
@@ -181,6 +196,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
| --- | --- |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
| `install.sh` | The one-step installer. Safe to re-run. |
| `uninstall.sh` | The uninstaller: run it, restart the headset, and run it again. It doesn't need the rest of the repo. |
| `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
| `screens/` | ft-screens, the compositor (wlroots and OpenVR). |
| `session/` | The desktop session script and its config example. |
+1 -1
View File
@@ -3,7 +3,7 @@ Type=Application
Name=Reset Screen Layout
GenericName=Put the VR desktop's screens back in their layout
Comment=Float the screens and arrange them in the layout from Frametop Display Settings
Exec=@REPO@/layout/ft-layout apply
Exec=@REPO@/layout/ft-layout-reset
Icon=view-restore
Categories=Settings;
Keywords=display;screen;layout;arrange;reset;steamvr;frametop;
+1 -1
View File
@@ -3,7 +3,7 @@ Type=Application
Name=Hide/Show Screens
GenericName=Hide or show the VR desktop's screens
Comment=Hide the screens (and SteamVR's laser) for a VR game; press again to bring them back
Exec=@REPO@/layout/ft-layout toggle
Exec=@REPO@/layout/ft-hide-show
Icon=view-visible
Categories=Settings;
Keywords=display;screen;hide;show;steamvr;frametop;
+1 -1
View File
@@ -802,7 +802,7 @@ Kirigami.ApplicationWindow {
Repeater {
model: [
{ value: "hide", text: "Hide them unless the SteamVR dashboard is open", help: "The game has the view to itself; open the dashboard (or press Meta+Shift+H) to see the screens." },
{ value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it." }
{ value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it. Turn off Pause while a VR game runs in Frametop Input Settings (Game optimization), or Frametop pauses and hides them anyway." }
]
delegate: ColumnLayout {
required property var modelData
+4 -4
View File
@@ -1,6 +1,6 @@
# Controller desktop click stability
Trigger presses reach KDE immediately, but controller motion within 8 logical
Trigger presses reach KDE immediately, but controller motion within 32 logical
pixels of the press stays at that position until release. Releasing without a motion outside this
zone delivers the click at the original position, even if the hand moved during
release. Moving outside the zone begins a normal drag immediately; returning to
@@ -13,11 +13,11 @@ floating-app title-bar carrying are unaffected. Multi-button gestures keep their
existing behavior. A motion onto another desktop monitor starts a drag;
cross-monitor motion is not stabilized.
CLI (runtime preferences, reset to 8 on desktop restart):
CLI (runtime preferences, reset to 32 on desktop restart):
```sh
input/ft-clickctl status
input/ft-clickctl threshold 8
input/ft-clickctl threshold 32
input/ft-clickctl threshold 0 # disable without a restart
```
@@ -29,6 +29,6 @@ This is a separate contribution from desktop mouse/controller ownership. Its
hardware validation must check small controls, intentional text selection,
long presses, cross-monitor dragging and simultaneous mouse use. The existing
renderer laser remains tracked; this change stabilizes desktop input rather
than smoothing the visual laser. Default threshold is a starting point to test.
than smoothing the visual laser. The default was 8 at first. That's about 0.2° on a 3.4 m wide 3440-pixel screen 2 m away, and clicking took a very still hand, so it's 32 (about 0.9°) since 2026-10-03.
Run `scripts/test-controller-click.sh` for the isolated gesture-state tests.
+39 -7
View File
@@ -38,7 +38,7 @@ The curved layout chains screens edge to edge, like monitors on a desk: the midd
A resize handle has to be able to shrink a screen from any direction, so the dragged corner follows the laser along the screen's diagonal rather than taking the larger of its horizontal and vertical reach. Pushing and pulling a carried screen moves it along the line from your head, because the 3D mouse's virtual controller sits just in front of the bar, below the screen's centre, so the line from the device points mostly upward.
Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose.
Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose. ft-screens reads the tip with `GetComponentState`: `GetComponentStateForDevicePath` without an input source handle fails for every component while a VR game runs, so in games the rays came from the pose, 40° too high.
`ComputeOverlayIntersection` ignores `SetOverlayIntersectionMask`, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner.
@@ -62,6 +62,8 @@ A profile's screen part is the custom arrangement under a name: each screen's po
`IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting.
In a game, Frametop's panels work like SteamVR's own floating windows: point a controller at one and its laser comes on, point away and the game has the controllers again. ft-screens turns the flag on for a panel while a hand controller's laser pose meets it, its controls, or a floating window's popups. It finds that from the poses it already reads to show the controls, so SteamVR's laser doesn't have to be on first. Leaving takes a margin two control-sizes wide and 0.3 s, a drag or a held button keeps the flag on, and the keyboard, a single overlay, uses SteamVR's `ComputeOverlayIntersection`. The 3D mouse doesn't need any of this: it has its own laser mode.
## Floating windows
[floating-windows.md](floating-windows.md) describes the feature and its parts. Drag and drop and the clipboard only work between windows of one compositor, so a floating window stays a KWin window and gets a KWin output of its own: one of the spare outputs KWin opens after the screens, shown by ft-screens as a panel cropped to the window. What follows is how KWin 6.2.5 behaves underneath that, from its source (`src/backends/wayland/`) and from trying it on the Frametop desktop.
@@ -116,9 +118,9 @@ The driver starts disconnected, because holding the right-hand role while SteamV
### The cursor
Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either.
Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. Both tests ask SteamVR about every visible overlay, so a frame where nothing moved (the mouse, the anchor, the eye by more than 5 mm, which overlays show) reuses the last result, for up to 100 ms, since overlays can also move on their own. The dots' overlay settings go to SteamVR only when they change, and the pose goes to the driver, which keeps the last one, only when the laser would land 0.1 mm or more elsewhere, and at least every 100 ms.
OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately.
OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. Each run is a shell and a new SteamVR client, about 30 ms of CPU, and it ran every second while the pointer was awake, which in gaze mode is all the time. Now it runs every 20 seconds, and at once when the pointer wakes, when the dashboard opens or closes, when a game starts or ends, and when a left click hits nothing (a panel that came up since). An overlay already on the list showing or hiding needs no new list: the helper checks the visibility of the ones it knows every 50 ms.
The laser starts partway along your line of sight to the cursor rather than at your eye. SteamVR sizes its hit dot by distance from the laser's origin, and a laser from the eye still shows a beam in each eye. Starting it close to the target makes the beam and the dot tiny, while `POINTER_ORIGIN_MARGIN` keeps the origin in front of the small window controls, which float a few centimetres in front of their panels. The helper's own white dot is the visible cursor. In empty space it's an interactive overlay that the laser lands on, so SteamVR never draws a laser into nothing.
@@ -131,7 +133,7 @@ A few overlays need special handling:
Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a right click or MetLine truncated
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: the session now starts an AT-SPI registry, but apps still need to expose useful accessibility trees (and may need restarting), and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a rightLine truncated
Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
@@ -141,7 +143,7 @@ Replacing a loaded driver's files, as re-running the installer used to do, leave
The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times `POINTER_CONTROLLER_PICKUP`, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on.
When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off.
When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. With the pointer off it also stops running its loop every 8 ms, about 116 wakeups a second for nothing: it waits up to 250 ms for a command on its socket (20 ms while it reads mapped controller buttons, which SteamVR input only offers by polling), and leaves the overlay lookups until the pointer wakes.
### Moving floating windows
@@ -153,6 +155,8 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle
Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead.
The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. Mouse motion goes to the helper at most every 4 ms, rather than once per report, which from a 1000 Hz mouse was 1000 datagrams a second to a helper that runs every 8 ms; a button sends the motion before it first, so the click lands where the pointer was.
An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key <code> <value> <device name>`). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too.
Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed.
@@ -165,10 +169,22 @@ The Frame controllers can be mapped like mouse buttons, but they aren't input de
## The desktop session
The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly.
The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly. Its own config folder also hides SteamVR's path registry (`~/.config/openvr/openvrpaths.vrpath`) from everything started in it: OpenVR programs there fail with `VRInitError_Init_PathRegistryNotFound`, and `vrpathreg adddriver` writes a new registry under `~/.config/frametop/openvr` that has no SteamVR in it and that SteamVR never reads. So Frametop's scripts run SteamVR's tools with `XDG_CONFIG_HOME=~/.config`.
The VR launcher starts the session from the Steam client, and the client's environment came along: `LD_LIBRARY_PATH` pointing at Steam's own runtime, whose `libavcodec` has no H.264 decoder, so VLC in the desktop couldn't play most videos, plus the client's overlay and launch settings. The session script drops the client's variables before it starts anything. SteamOS's global Mesa settings (`/usr/share/deckard/mesavars.sh`) stay, and the gamescope session's Vulkan layer (`ENABLE_GAMESCOPE_WSI`) is only kept for the gamescope backend.
### Nested accessibility
The session drops an inherited `AT_SPI_BUS_ADDRESS`, so apps cannot accidentally use the host desktop's registry. It autostarts `session/ft-atspi` in Plasma phase 2, after KWin has set the nested display environment. The helper gets the live accessibility address from `org.a11y.Bus` on the private session bus, preserves any existing registry owner, updates the accessibility bus's activation environment, and tries `StartServiceByName` first.
On SteamOS 0.3.0 with at-spi2-core 2.52.0, the native launcher can choose dbus-broker because its process belongs to a systemd user unit. Registry activation then fails: this desktop's private session bus does not have a systemd activation manager. In that case the helper starts only `at-spi2-registryd` on the already-existing accessibility bus. The registry refuses duplicate ownership. Unlike native activation's `--use-gnome-session`, the fallback does not try to register with GNOME's session manager; that flag did not explain the observed native activation failure.
The fallback registry does not exit merely when its bus disconnects in the isolated SteamOS test. Its small watcher checks both private buses every 5 seconds, and terminates and reaps only the child it started when either bus disappears or the watcher is stopped. Each check runs `gdbus` twice; once a second, that cost about 1% of a core. `keep-apps.sh` keeps the watcher in the desktop unit when `desktops.sh start` runs the desktop as `frametop-desktop`. Started from the VR launcher, the desktop runs in steam.service, which doesn't stop with it, so there the watcher is the only thing that stops the registry. There is no second accessibility bus, global systemd environment update, process-name kill, or host registry replacement. Missing accessibility files or bus errors are nonfatal; the desktop still starts. Toolkit-specific accessibility opt-ins and pointer snapping are separate work.
Run the isolated checks on the host with `/usr/bin/python3 session/test/test_accessibility.py`. They use private D-Bus buses, Xvfb and a GTK3 app, never the production display or input. Native activation uses a small `org.a11y.Bus` test provider pointing to a real private dbus-daemon with the installed registry service; the SteamOS fallback uses the installed bus launcher and broker. The tests check real app-tree discovery, existing owners, concurrent starts, session stop/restart, and teardown. They require test-only PyGObject (Gio and GTK3), Xvfb, and at-spi2-core; the runtime helper uses Python's standard library and the existing host `gdbus`. Actual Plasma autostart and VR desktop restart still require an approved hardware test.
### Other session behavior
Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.
Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens Discover instead of launching them, so the session sources `/etc/profile.d/flatpak.sh`.
@@ -177,6 +193,14 @@ The private runtime directory also moves the session's document portal to `$XDG_
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompositor needs to hit its frame time, and on the Frame that costs CPU too. The nested session started with KWin's defaults: blur and background contrast on (no `[Plugins]` group in its kwinrc) and animations at full length. Blur re-renders what's behind every translucent panel and menu each time it changes, and every animated frame is one more frame for KWin and ft-screens to draw and send. They're off by default in the Frametop desktop. The session script writes them before KWin starts, only where the desktop's own config has no value, once: System Settings deletes a setting put back to KDE's default rather than writing it, so without the marker in `frametoprc` a user who turned blur back on would lose it at the next start. The effect ids (`blur`, `contrast`) are the ones built into KWin 6.2.5 on SteamOS; KWin reads `<id>Enabled` from `[Plugins]`.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
Plasma 6.2.5 keeps each panel on a screen number (`lastScreen` in `plasma-org.kde.plasma.desktop-appletsrc`), and the numbers rank the enabled outputs by priority, so 0 is the primary screen. A panel whose number is past the screen count gets no view, and Plasma never moves it: the remap it runs at every start only moves a panel whose number has no desktop, and this desktop keeps a desktop for every output it has seen, spares included. So the taskbar was lost when the number of screens went down, and once it was found saved on a spare output, number 8 of a desktop with three screens ([#18](https://github.com/DeeJanuz/frametop/issues/18)). Before Plasma starts, the session runs `session/fix-panels.py`, which moves any panel numbered past the screen count, with its system tray's containment, to screen 0, keeping its widgets and settings. A panel stays put when screen 0 already has one on that edge, and comes back by itself if the screens do. The file is backed up to `<file>.ft-bak` first. Plasma's scripting can't do this while it runs (`panel.screen` is read-only in 6.2.5), so a lost taskbar comes back at the desktop's next start. `scripts/doctor.sh` and `scripts/report.sh` list the panels and their screens.
Remote desktop is a chain (krdp, then FreeRDP inside Xvnc) because nothing on SteamOS serves KWin over VNC directly. Kept connected all the time, it cost about a core with nobody watching: krdpserver 55 to 78% (it encodes H.264 in software with openh264: VA-API finds no driver for the Frame's GPU in the container), FreeRDP 16 to 27%, Xvnc 6 to 11%, and the bridge's layout check every 5 seconds another 4%. krdp creates its screencast session per RDP connection and drops it when the connection closes (`SessionController::onNewConnection` in krdp 6.7), so an idle krdpserver costs nothing and can stay up; only the RDP connection has to go. The bridge connects FreeRDP when a VNC client appears and disconnects 45 seconds after the last one leaves. Xvnc has no hook for its clients, so the bridge counts established connections to its port with `ss`, woken early by Xvnc's log output; looking with `ss` once a second cost about 0.9% of a core in bash, against about 0.1% this way. `Xvnc -inetd` from a systemd socket would start a server per connection and lose sharing between viewers. The layout check (`ft-layout remote-view`, which scans `/proc` for plasmashell and runs `kscreen-doctor -j`) now runs only while FreeRDP runs, and then only after `kwinoutputconfig.json` or `frametop-layout.json` changes, with one check a minute in case a change touched neither.
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
## Displays off on a stand
@@ -189,6 +213,15 @@ Movement is judged within 10-second windows. On the mount, the head pose jittere
Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values.
## Pausing for VR games
Hiding the screens during a game kept them out of view, but Frametop kept using the headset. Measured on 2026-10-02 with gaze mode off and no game running, in shares of one core: our eye tracker (ft-eyes) about 60%, ft-eyegrab, ft-gaze and ft-gazed about 3 to 4% each; remote desktop (krdpserver, FreeRDP, Xvnc) about 2 cores while it ran; KWin about 13%, ft-screens about 4%. The gaze service ran at full rate whether gaze mode was on or not; now it idles while the gaze isn't used (gaze/README.md), and pausing stops it outright. Reading SteamVR's eye tracking 90 times a second also made it restart every 10 to 13 seconds during Beat Saber, and each restart took input focus from the game, which paused it (PR #13; since then ft-gaze skips SteamVR's gaze action during games, but our own tracker kept running). So pausing stops what costs the most and leaves windows where they are.
- A hidden screen still cost as much as a visible one. ft-screens sent every committed screen its frame callback at 90 Hz whether its overlay showed or not (since then, a hidden screen always gets one a second; see the frame rates in reference.md), so KWin kept drawing, and its apps with it. Paused, ft-screens sends the callbacks once a second. A Wayland client draws again only after its last frame's callback, so KWin's output stalls, KWin's own clients stop getting theirs, and the whole desktop idles, without anything losing its connection. A second's pace, rather than none, keeps any client that waits on a callback from waiting forever. Stopping KWin or the apps with SIGSTOP would free the same, but a Wayland peer that stops reading overflows the other side's 4 KB socket buffer, which ends the connection: that's how the live desktop died once when its KWin stalled (`Data too big for buffer`). They also sit in different cgroups (KWin under steam.service when the VR launcher starts it, ft-screens in the dev container's), so no single freeze stops them together.
- The relay does the pausing because it's the one part that always runs, and the pointer helper keeps running because stopping it leaves its virtual controller connected with its last pose (the driver has no staleness timeout), maybe holding a hand role, with the 3D mouse dead. Releasing it does the job. The helper already checks for a scene app twice a second, so it's what tells the relay a game started.
- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/<serial>` instead of `/user/hand/right`), so the reader looks the controllers up again (an HTTP request to vrserver): when the relay's 3D mouse connects or lets go, when a message comes from a device it doesn't know, and every 30 seconds. It used to be every 3 seconds.
- Resuming starts remote desktop through `systemd-run --scope`: started straight from the relay, it would join the relay's cgroup and end with the next relay restart.
## SteamOS updates
On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.
@@ -205,6 +238,5 @@ On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-stea
- A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens.
- Drawing KWin's cursor on the screens.
- Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back.
- Frame pacing and GPU cost with several busy screens haven't been measured.
- Real standby on a stand, with rendering and tracking paused, not just the backlight off. SteamVR has no call for it, and its activity level follows the proximity sensor.
+2 -2
View File
@@ -46,8 +46,8 @@ So a "Work" profile can put three screens around you with a browser, two termina
## How it works
- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name: single-instance and D-Bus-activated apps open their windows from a process that was already running.
- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. A window whose app id has no desktop file (a Flatpak app's X11 window can give its own: RustDesk's says `com.carriez.flutter_hbb`, its desktop file is `com.rustdesk.RustDesk`) is kept by the desktop file whose `StartupWMClass` names its window class. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running.
- **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
- **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
- **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.
+56 -10
View File
@@ -18,6 +18,18 @@ desktops.sh start | stop | restart | status | log [lines]
When the VR launcher starts the desktop, it inherits the Steam client's environment. The session script drops the client's runtime from it (`LD_LIBRARY_PATH`, the `STEAM_*` settings, and the Steam overlay's Vulkan layer), so apps in the desktop use the system's libraries, including its video codecs, just as they would after a normal login.
The nested session also starts an AT-SPI accessibility registry through `session/ft-atspi` in Plasma's autostart. It discovers the bus from this session, ignores an inherited host accessibility address, and leaves an existing registry alone. Accessibility errors do not stop the desktop. This supplies the registry infrastructure for apps that expose AT-SPI trees; it does not enable gaze snapping or force Chromium/Electron accessibility. After an approved desktop restart, an AT-SPI-aware app should be visible on the nested bus. See [design.md](design.md#nested-accessibility) for native activation, fallback lifecycle, and the isolated test command.
KWin's blur and background contrast effects and its animations are off in this desktop, because KWin draws on the headset's GPU, which SteamVR needs. The session script turns them off once, the first time it starts (it leaves a setting you already have alone, and marks it done in `~/.config/frametop/frametoprc`), so turning them back on sticks. In the Frametop desktop, System Settings → Window Management → Desktop Effects has Blur and Background Contrast, and General Behavior has Animation speed. Or from a terminal, then restart the desktop:
```
kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key blurEnabled true
kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEnabled true
kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1
```
Two of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, and IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`), and skips a name you already have a file for. Delete a copy to start that program again.
Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
Restarting the desktop closes its windows. Before the unit stops, `session/keep-apps.sh` moves every program started in the desktop into a systemd scope of its own, so background work such as servers, tmux, and builds keeps running. An app that shuts down its own helper processes when its window closes will still lose them; run that kind of work outside the desktop, for example as a systemd user service.
@@ -28,12 +40,13 @@ Restarting the desktop closes its windows. Before the unit stops, `session/keep-
Each KWin window is one screen. ft-screens sets its size with an `xdg_toplevel` configure and KWin resizes the output to match, live. Frames arrive as DMA-BUFs and go to SteamVR through OpenVR's `IVRIPCResourceManagerClient::ImportDmabuf`, with no copy and no size limit.
Every screen is an overlay named `frametop.screen.N` with four controls:
Every screen is an overlay named `frametop.screen.N` with five controls:
- `.bar` moves the screen. Drag it with any laser or with the 3D mouse, whose right-drag tilts. Scrolling while you drag pushes the screen away or pulls it closer, along the line from your head.
- `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
- `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
- `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.
- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`).
The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.
@@ -51,7 +64,7 @@ The Visibility & pins tab of Frametop Display Settings decides when the screens
In the last three modes the hotkey shows the screens anyway. A screen can also be hidden on its own (Screens shown on the same tab, or `ft-layout hide N`): it stays hidden whatever the mode or the hotkey says, until it's shown again there. Windows on it stay put, and a new window that would open on it floats instead (ft-floatd). Profiles use this to show only some screens. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:
- During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. Pointing a controller at a screen, a floating window, or the keyboard still turns its laser on, like SteamVR's own floating windows, and pointing away gives the game the controllers back. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.
@@ -66,12 +79,15 @@ place N x y z yaw pitch roll width N metres curve N radius|on|off
pin N|all left|right|head [matrix] unpin N|all size N w h
get N screens head state key code value scale N s vrkeyboard show|hide|toggle|close
visibility always|dashboard|gesture|toggle wrist degrees gesture left|right degrees
hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible
hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible pause on|off|state
conceal N|all reveal N|all concealed cutouts on|off|state cutouts predict on|off cutouts lead ms
float N mpp x y w h title unfloat N pose N matrix sub N k x y w h | sub N k off minimized N 0|1 carry N
rates focused in_view hidden rates? watch seconds phase ms
```
`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
Each screen draws at a frame rate for how much of it you see. KWin draws a screen only after ft-screens gives it a frame callback, and its apps wait for theirs, so the rate of callbacks is the screen's frame rate, for KWin and the apps on it alike. A screen is focused while you look at it (within 12 degrees of where your head points), while a laser or the mouse is on it or was in the last 1.5 seconds, while it's carried, and while you type on it; it gets every display frame. The rest of what you can see (within 60 degrees) gets 15 frames a second, and a hidden screen, one behind you, and everything while paused get one a second. A level goes up at once and comes down after a moment (1.5 s from focused, 0.5 s from in view). A video, or anything moving over a large part of a screen (6% or more of it, redrawn on 8 commits in a row, 10 or more a second), keeps every frame while in view. A floating window is a screen of its own here. Nothing that stands still costs anything at any rate: KWin sends a frame only when something on the screen changed. `rates F V H` sets the three rates in Hz (0: every display frame; default `0 15 1`, also `ft-screens --rates 0,15,1`), and `rates?` shows them, the display's rate, and for each screen its level, its milliseconds between frames, and whether it counts as a video. `watch S` gives every screen full rate for S seconds: remote desktop renews it while a VNC client is connected, since a viewer sees what KWin draws. The ticks (SteamVR events and the callbacks) come once per display frame, 1 ms after the vsync (`phase ms` changes that, for tuning), in step with the display rather than on a timer that drifted through the frame.
`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
## Input relay
@@ -114,12 +130,13 @@ The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `P
## Frametop Input Settings
A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages:
A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has nine pages:
- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|<degrees>`); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
@@ -129,7 +146,7 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta
## Frametop Display Settings and ft-layout
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist or your head come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
@@ -199,9 +216,36 @@ power/run.sh off | on # the displays off now, or back on
power/run.sh log
```
## Pausing for VR games
Paused, Frametop leaves the headset's CPU and GPU to a VR game. The input relay does it (`input/game_pause.py`), since it's the one part that always runs:
- The gaze service stops (`frametop-gaze`: ft-gazed, ft-gaze, our own eye tracker, the gaze panel), so nothing reads SteamVR's eye tracking. Our frame grabber, the root service `ft-eyegrab`, goes idle by itself 3 seconds after our eye tracker stops asking it for frames.
- Hand tracking stops if it runs (`frametop-camd`, `frametop-hands`).
- The desktop, as the Game optimization page of Frametop Input Settings says (`pause_desktop`): hidden (the default) or closed. Hidden, ft-screens hides every screen and floating window whatever the visibility mode, the hotkey, or the dashboard says, and gives KWin a frame callback once a second instead of every display frame. KWin draws a screen only after its frame callback, and its apps wait for theirs, so the desktop hardly draws, but its windows stay open. Remote desktop stops if it runs (`session/remote-ctl.sh`). Closed, `desktops.sh stop` closes the desktop and its windows, and resuming starts it again (about 12 seconds), in its start profile if it has one.
- The relay lets go of the 3D mouse and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, and commands; a key combination that does nothing is typed as usual.
Resuming starts again only what pausing stopped, and plays a second sound. The pointer helper and ft-powerd keep running: they cost little, the helper is what says a game started, and stopping it would leave its virtual controller connected with its last pose.
Ways to pause and resume:
- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none.
- The Pause/resume Frametop action, on a mouse button, a key combination, or a controller button (outside games, like every mapped controller button).
- VR games, with Pause while a VR game runs on (`pause_auto`, the default). The pointer helper tells the relay when a scene app starts and ends (`vrgame 1|0`, repeated every 5 seconds). A game starting pauses Frametop. A pause that starts while a game runs ends 5 seconds after the game does, unless another game starts first. Resumed during a game, Frametop stays on until that game ends. A pause that starts outside a game lasts until you resume. Flatscreen games aren't scene apps, so they don't pause it.
- From a terminal or a script:
```
input/ft-pause on | off | toggle # pause or resume
input/ft-pause status # the state as JSON (the relay's "pause ?")
input/vrws.py 10 # the controllers' buttons from vrserver's web socket, for 10 s
input/test/pause-test.py # the gesture and the automatic pause, offline
```
The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
## Gaze pointer (experimental)
In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. The service idles while the gaze isn't used: its eye tracker reader and our own eye tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration runs, or while the Gaze page of Frametop Input Settings is open, and stop 30 seconds after ([gaze/README.md](../gaze/README.md)). Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
- Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
- The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
@@ -219,7 +263,7 @@ Your hands show over the screens: where a tracked hand is between an eye and a s
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`.
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
- The install leaves both off, and they don't start with SteamVR. `ft-handsctl on` starts them while SteamVR runs, and `ft-handsctl off` stops them; they also stop with SteamVR. The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
@@ -231,6 +275,8 @@ It listens on port 5900 on the Frame's Tailscale address only, not the LAN, so i
No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency. krdp streams every screen; the VNC screen is the primary's size, and the FreeRDP window is shifted so the primary fills it (`ft-layout remote-view` gives the offset). krdp's own `--monitor` would stream just one screen, but it maps the pointer as if that screen sat at 0,0, so clicks would miss. When the layout changes, the VNC screen resizes and FreeRDP reconnects within a few seconds.
FreeRDP runs only while a VNC viewer is connected, because while it's connected krdp captures and encodes every redraw. With no viewer, krdp has no RDP connection and so captures nothing, and Xvnc shows a black screen. When a viewer connects, the bridge starts FreeRDP, and the desktop appears about 3 seconds later; FreeRDP stops 45 seconds after the last viewer leaves (`VNC_IDLE_SEC` in the bridge's environment). The bridge looks for viewers with `ss` whenever Xvnc logs something, as it does for every connection, and every 5 seconds otherwise. While a viewer is connected, the bridge asks ft-screens to draw every screen at full rate (`watch 15` on `@ft_screens`, renewed every 5 seconds), so screens you aren't looking at in the headset, or a headset on a stand, don't stream at a low rate. It reads the primary screen's place again (`ft-layout remote-view`) only while FreeRDP runs, after `~/.config/frametop/kwinoutputconfig.json` or `~/.config/frametop-layout.json` changes, and once a minute.
With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` and `KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screens or inject input. The second one lets scripts take screenshots through KWin's `org.kde.KWin.ScreenShot2` D-Bus interface. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop.
## Limits
+11
View File
@@ -378,6 +378,17 @@ function run(c) {
case "activate":
if (w) workspace.activeWindow = w;
break;
case "activate-output": { // the top window on that output (a spin brought it to the front)
const order = workspace.stackingOrder;
for (let i = order.length - 1; i >= 0; --i) {
const o = order[i];
if (o.deleted || o.minimized || o.hidden || !o.managed || !o.output) continue;
if (o.output.name !== c.output || !o.normalWindow || o.popupWindow) continue;
workspace.activeWindow = o;
break;
}
break;
}
case "minimize":
if (w) w.minimized = c.on;
break;
+38 -6
View File
@@ -18,6 +18,8 @@ command-line side). Replies go to the sender:
("float pointer" is the float key: the window under the pointer, else the active one,
floated or docked; the input relay sends it for float_toggle, and "dock all" for dock_all)
dock N | close N | resize N W H | scale N STEPS (ft-screens, N = its screen)
front N (ft-screens: a spin brought panel N to the front: make its window, or the top one
on a screen, KWin's active window)
Spare outputs are WL-<screens> .. WL-<screens + slots - 1>. A floating window's output is its
frame plus a margin on each side (FLOAT_MARGIN pixels), so menus have room; the panel shows
@@ -229,6 +231,24 @@ def cmdline(pid):
return []
def desktop_name(app, cls):
"""The desktop file name for a window's app id and class. A Flatpak app's window can give an
id with no desktop file: RustDesk's (an X11 window) says com.carriez.flutter_hbb, and its
desktop file is com.rustdesk.RustDesk. That file's StartupWMClass names the window's class
then. The app id itself when nothing matches."""
try:
if not app or Gio.DesktopAppInfo.new(app + ".desktop"):
return app
except TypeError: # PyGObject raises for the NULL a missing desktop file returns
pass
names = {n.lower() for n in (app, cls) if n}
for info in Gio.AppInfo.get_all():
wm = info.get_startup_wm_class() if isinstance(info, Gio.DesktopAppInfo) else None
if wm and wm.lower() in names:
return info.get_id().removesuffix(".desktop")
return app
class Launch:
"""An app we started: its first window floats, or (for a profile) its windows go to the
profile's entries for it, in order."""
@@ -745,7 +765,10 @@ class Daemon:
Returns (reply, pid or None)."""
if app:
app = app.removesuffix(".desktop")
info = Gio.DesktopAppInfo.new(app + ".desktop")
try:
info = Gio.DesktopAppInfo.new(app + ".desktop")
except TypeError: # PyGObject raises for the NULL a missing desktop file returns
info = None
if info is None:
return f"error no app {app}", None
pids = []
@@ -776,8 +799,10 @@ class Daemon:
return None
chain = pid_chain(int(ev.get("pid") or 0))
app = ev.get("app", "")
# (An X11 window in a Flatpak gives its pid in the sandbox: only its app id can match.)
apps = {app, desktop_name(app, ev.get("cls", ""))} if app and any(la.app for la in self.launches) else {app}
for la in self.launches:
if any(p in chain for p in la.pids) or (la.app and app and la.app == app):
if any(p in chain for p in la.pids) or (la.app and app and la.app in apps):
if not la.entries or len(la.entries) <= 1:
self.launches.remove(la)
return la
@@ -841,7 +866,7 @@ class Daemon:
for wid, ev in self.windows.items():
if not recordable(ev):
continue
entry = {"app": ev["app"]} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
entry = {"app": desktop_name(ev["app"], ev.get("cls", ""))} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
if not entry.get("app") and not entry.get("cmd"):
continue
f = self.floats.get(wid)
@@ -868,7 +893,7 @@ class Daemon:
return out
def window_key(self, ev):
return ev.get("app") or json.dumps(cmdline(ev.get("pid")))
return desktop_name(ev.get("app", ""), ev.get("cls", "")) or json.dumps(cmdline(ev.get("pid")))
def open_profile(self, name):
"""Open a profile's apps (additive: nothing closes)."""
@@ -884,9 +909,9 @@ class Daemon:
if key != "[]":
groups.setdefault(key, []).append(e)
claimed = set()
keys = {wid: self.window_key(ev) for wid, ev in self.windows.items() if recordable(ev)}
for key, entries in groups.items():
have = [ev for wid, ev in self.windows.items()
if wid not in claimed and recordable(ev) and self.window_key(ev) == key]
have = [ev for wid, ev in self.windows.items() if wid not in claimed and keys.get(wid) == key]
for e, ev in zip(entries, have):
claimed.add(ev["id"])
self.place_entry(ev, e)
@@ -1040,6 +1065,13 @@ class Daemon:
for f in list(self.floats.values()):
self.dock(f)
return "ok"
if cmd == "front" and len(rest) == 1 and rest[0].isdigit():
f = self.by_panel(int(rest[0]))
if f:
self.command(cmd="activate", id=f.id)
else: # a screen: its output is WL-<N - 1>
self.command(cmd="activate-output", output=f"WL-{int(rest[0]) - 1}")
return "ok"
if cmd in ("dock", "close", "resize", "scale") and rest and rest[0].isdigit():
f = self.by_panel(int(rest[0]))
if not f:
+4 -3
View File
@@ -2,7 +2,7 @@
The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it.
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on. The gaze service asks it for only the sources it uses (`--sources`, and `sources LIST` on its stdin): our tracker and mmap set 1 with Own tracker, about 700 bytes a line instead of 1.3 KB, and every source while a check or the calibration runs. So SteamVR's gaze action, the only source that calls into vrserver (twice a sample), is read only then. The probe gets them all.
- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
@@ -33,8 +33,9 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
@@ -62,7 +63,7 @@ Ground rules, for anyone changing it:
- **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only).
- **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`.
- **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs.
- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. Replays, scoring, and training go to a PC.
- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. ft-eyes keeps OpenCV and numpy to one thread: their pools of one per core spun idle workers at about a quarter of a core, for frames this small. It also runs at nice 10 with SCHED_BATCH, and ft-gaze at nice 5 (not batch, since each sample goes on to the pointer): both run in the dev container's podman scope, out of reach of the gaze service's unit, on the cores vrcompositor and vrserver use at nice 0. Replays, scoring, and training go to a PC.
## Headset fit
+85 -16
View File
@@ -4,7 +4,14 @@
// Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source
// hit-tested against the Frametop screens:
//
// Options: -v (log action errors), --watch-stdin (quit when stdin closes).
// Options: -v (log action errors), --watch-stdin (quit when stdin closes; until then, a line
// "sources LIST" on stdin switches the sources as --sources does), --sources LIST (comma-
// separated: action, mmap1, mmap2, left, right, own, and eye for EYE; or all, the default).
// Sources left out are read not at all and print as {"ok":0} ("eye" as null), so every line
// keeps the same keys. The gaze service asks for the ones it uses (own and mmap1 with our
// tracker), and all of them while a check or the calibration runs. Only the action costs
// SteamVR anything (two calls into vrserver per sample), and a line with every source is
// about 1.5 KB, 130 KB a second through podman's relay.
//
// {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
// "head":{"yaw":..,"pitch":..,"hit":HIT}, head forward ray (for head nudging)
@@ -58,6 +65,7 @@
#include <algorithm>
#include <atomic>
#include <cerrno>
#include <chrono>
#include <climits>
#include <cmath>
@@ -72,6 +80,7 @@
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/un.h>
@@ -389,6 +398,30 @@ std::string SrcJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t
return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}";
}
// Sources (--sources, "sources LIST" on stdin): a bit each.
enum : unsigned { kAction = 1, kMmap1 = 2, kMmap2 = 4, kLeft = 8, kRight = 16, kOwn = 32, kEye = 64, kAll = 127 };
bool ParseSources(const std::string &list, unsigned &mask) {
static const struct {
const char *name;
unsigned bit;
} names[] = {{"action", kAction}, {"mmap1", kMmap1}, {"mmap2", kMmap2}, {"left", kLeft},
{"right", kRight}, {"own", kOwn}, {"eye", kEye}, {"all", kAll}};
unsigned m = 0;
size_t at = 0;
while (at <= list.size()) {
const size_t comma = std::min(list.find(',', at), list.size());
const std::string name = list.substr(at, comma - at);
bool known = false;
for (const auto &n : names)
if (name == n.name) m |= n.bit, known = true;
if (!known) return false;
at = comma + 1;
}
mask = m;
return true;
}
// Head poses of the last half second, so a sample can use the pose at its own time.
class PoseHistory {
public:
@@ -426,17 +459,42 @@ std::string ExeDir() {
int main(int argc, char **argv) {
bool verbose = false, watchStdin = false;
std::atomic<unsigned> sources{kAll};
for (int i = 1; i < argc; ++i) {
if (std::strcmp(argv[i], "-v") == 0) verbose = true;
if (std::strcmp(argv[i], "--watch-stdin") == 0) watchStdin = true;
if (std::strcmp(argv[i], "--sources") == 0 && i + 1 < argc) {
unsigned m;
if (ParseSources(argv[++i], m))
sources = m;
else
std::fprintf(stderr, "ft-gaze: --sources %s: unknown source (all used)\n", argv[i]);
}
}
// Nice 5, before any thread starts (they inherit it): we run in the dev container's podman
// scope, beside vrcompositor and vrserver at nice 0, and the gaze service's unit doesn't
// reach us. Not SCHED_BATCH, as ft-eyes is: that would let each wakeup wait out another
// task's turn, and each sample goes on to the pointer.
errno = 0;
const int nice0 = getpriority(PRIO_PROCESS, 0);
if (errno == 0 && nice0 < 5 && setpriority(PRIO_PROCESS, 0, 5) != 0)
std::fprintf(stderr, "ft-gaze: nice: %s\n", std::strerror(errno));
// --watch-stdin: quit when stdin closes. The probe runs us through distrobox, which
// passes neither its signals nor a closed stdout on to us, but does pass stdin's end.
// Lines on stdin until then: "sources LIST".
std::atomic<bool> stdinClosed{false};
if (watchStdin)
std::thread([&stdinClosed] {
std::thread([&stdinClosed, &sources] {
char c[256];
while (read(0, c, sizeof c) > 0) {
std::string line;
ssize_t n;
while ((n = read(0, c, sizeof c)) > 0) {
line.append(c, size_t(n));
for (size_t nl; (nl = line.find('\n')) != std::string::npos; line.erase(0, nl + 1)) {
unsigned m;
if (line.compare(0, 8, "sources ") == 0 && ParseSources(line.substr(8, nl - 8), m)) sources = m;
}
if (line.size() > 4096) line.clear(); // no newline in sight: not ours
}
stdinClosed = true;
}).detach();
@@ -509,6 +567,7 @@ int main(int argc, char **argv) {
if (fresh && hp.bPoseIsValid) {
lastEmit = now;
const unsigned want = sources;
const auto list = screens.Get();
const vr::HmdMatrix34_t &headNow = hp.mDeviceToAbsoluteTracking;
vr::HmdMatrix34_t headThen = headNow;
@@ -517,7 +576,7 @@ int main(int argc, char **argv) {
// SteamVR's action: a room-space origin and fixation point, turned into the head
// frame so every source reports the same kind of angles.
std::string action = "{\"ok\":0}";
if (!inGame) {
if (!inGame && (want & kAction)) {
vr::VRActiveActionSet_t active{};
active.ulActionSet = set;
active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
@@ -560,26 +619,33 @@ int main(int argc, char **argv) {
return std::string(b);
};
char extra[256];
std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
s.open[0], s.open[1], lr(s.left1, s.right1));
m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1));
std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
left = SrcJson(list, headThen, s.left2);
right = SrcJson(list, headThen, s.right2);
if (want & kMmap1) {
std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
s.open[0], s.open[1], lr(s.left1, s.right1));
m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1));
}
if (want & kMmap2) {
std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
}
if (want & kLeft) left = SrcJson(list, headThen, s.left2);
if (want & kRight) right = SrcJson(list, headThen, s.right2);
// "new" compares with the last sample, so the last measurement is kept either way.
const float *m = s.meas;
const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1];
const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3];
std::memcpy(lastMeas, m, sizeof lastMeas);
std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}",
(m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR));
eye = extra;
if (want & kEye) {
std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}",
(m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR));
eye = extra;
}
}
// Our tracker: its own sample time picks the head pose, like the mmap's.
std::string own = "{\"ok\":0}";
OwnSample o;
if (ownFile.Read(o) && now - o.t < 0.1) {
if ((want & kOwn) && ownFile.Read(o) && now - o.t < 0.1) {
vr::HmdMatrix34_t headOwn = headNow;
history.At(o.t, headOwn);
auto pair = [](bool ok, float a, float b) {
@@ -624,7 +690,10 @@ int main(int argc, char **argv) {
break;
}
if (stdinClosed) break;
std::this_thread::sleep_for(std::chrono::milliseconds(2));
// 250 passes a second: a new sample is printed within 4 ms (2 on average) of appearing,
// and the pose history has a pose within 2 ms of any sample's time (a 0.2 degree head
// turn at 100 degrees a second). Every 2 ms read the pose and the events twice as often.
std::this_thread::sleep_for(std::chrono::milliseconds(4));
}
screens.Stop();
vr::VR_Shutdown();
+111 -8
View File
@@ -63,9 +63,24 @@ and the error moves): lessons from before count less then, so the first few afte
relearn the offset.
Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the probe renews). It
reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
which copies them only while ft-eyes runs.
GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS",
a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab
copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs.
Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone
wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a
check or calibration open or asked for, or a "wake" lease (the Gaze page of Frametop Input
Settings renews one while it's open). IDLE_AFTER after the last use they stop, and the frame
grabber goes idle with our tracker; with our tracker, that was over half a core with gaze mode
off. A check asked for while idle waits for the tracker to start (at most WAKE_SETTLE). Our
tracker's next click after it starts again re-seats it, as after the headset was off, so turning
gaze mode on after a while opens the quick check.
ft-gaze prints only the sources this uses ("sources LIST" on its stdin, see wanted_sources):
own and mmap1 with our tracker, left, right and mmap1 with SteamVR's eyes, and every source
while a check or the calibration runs or is asked for, since those record them all. So
SteamVR's gaze action, the one source that calls into vrserver, is read only then (or with
--source action).
Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
our tracker asks for a click, and the full calibration when gaze mode comes on without one,
@@ -83,6 +98,8 @@ Control socket: abstract unix datagram "@ft_gazed":
reload read calibration.json and the settings again
eyes <seconds> keep our own tracker running that much longer (at most 120),
whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
wake <seconds> keep the gaze running (not idle) that much longer (at most 120),
whatever gaze mode says: Input Settings' Gaze page. Reply: "ok"
quickcal the one-dot check now (the calibration if there's none)
calibrate the full calibration in the panel
fitcheck the headset fit check in the panel
@@ -140,6 +157,9 @@ RETRY = 3.0 # seconds before starting ft-gaze again
EYES_RETRY = 10.0 # seconds before starting ft-eyes again after it stopped on its own
EYES_LEASE_MAX = 120.0
SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it
IDLE_AFTER = 30.0 # seconds after the gaze was last in use before ft-gaze and our tracker stop
WAKE_SETTLE = 15.0 # seconds after waking that the tracker may take to start sending
WAKE_MAX = 120.0
KEYBOARD_PITCH = -20.0 # degrees: gaze under this, on no screen, is a look at the keyboard
@@ -230,6 +250,10 @@ class Service:
self.eyes_proc = None # ft-eyes, while it runs
self.eyes_until = 0.0 # the probe's lease (monotonic time)
self.eyes_restart_at = 0.0
self.awake = False # ft-gaze (and our tracker) run: the gaze is in use (see the top)
self.idle_at = 0.0 # idle from then, unless it's in use again first
self.woke_at = 0.0
self.wake_until = 0.0 # a "wake" lease
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
self.sock.bind(ME)
@@ -242,6 +266,7 @@ class Service:
self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own")
self.checks = Checks(self, self.sel)
self.proc = None
self.proc_sources = None # what ft-gaze was last told to print
self.buf = b""
self.restart_at = 0.0
self.running = True
@@ -399,8 +424,73 @@ class Service:
log(f"lesson log: {e}")
return rec
# --- Idle (see the top) ---
def use_reason(self):
"""Why the gaze is in use now, or None."""
c = self.checks
if c.gaze_on and c.headset is not False:
return "gaze mode is on"
if c.active or c.pending:
return "a check"
if time.monotonic() < self.wake_until:
return "asked to stay awake"
return None
def idle_reason(self):
c = self.checks
if c.gaze_on is None:
return "the pointer helper isn't answering (SteamVR not running?)"
if c.gaze_on and c.headset is False:
return "nobody is wearing the headset"
return "gaze mode is off"
def waking(self):
"""Idle, or awake too briefly for the tracker to be sending yet."""
return not self.awake or time.monotonic() - self.woke_at < WAKE_SETTLE
def update_awake(self):
now = time.monotonic()
why = self.use_reason()
if why:
self.idle_at = now + IDLE_AFTER
if why and not self.awake:
self.awake, self.woke_at, self.restart_at = True, now, 0.0
log(f"awake: {why}")
elif not why and self.awake and now >= self.idle_at:
self.awake = False
log(f"idle: {self.idle_reason()}; ft-gaze and our tracker stop until the gaze is used again")
self.stop_helper()
if self.eyes_proc and not self.eyes_wanted():
self.stop_eyes()
# --- ft-gaze ---
def wanted_sources(self):
"""The sources ft-gaze should print (its --sources): what on_sample and the checks read.
mmap1 always (blinks, lost eyes, and the headset going on, from its openness and
variances); everything while a check runs or waits, since they record every source."""
c = self.checks
if c.active or c.pending:
return "all"
kind = self.kind
if kind == "own":
return "own,mmap1"
if kind == "eyes":
return "left,right,mmap1"
return ",".join(dict.fromkeys((self.source, "mmap1", "mmap2"))) # mmap2: each eye, for the fallback
def sync_sources(self):
want = self.wanted_sources()
if not self.proc or want == self.proc_sources:
return
try:
self.proc.stdin.write(f"sources {want}\n".encode())
self.proc.stdin.flush()
except (OSError, ValueError):
return # it's stopping: read_stdout notices
self.proc_sources = want
def start_helper(self):
if not HELPER.exists():
log(f"ft-gaze isn't built: run {REPO}/gaze/build.sh")
@@ -411,9 +501,11 @@ class Service:
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
# ft-gaze quits when its stdin closes: the one thing distrobox passes on.
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
start_new_session=True)
sources = self.wanted_sources()
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin",
"--sources", sources], env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, start_new_session=True)
self.proc_sources = sources
os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
@@ -441,7 +533,7 @@ class Service:
self.proc = None
def eyes_wanted(self):
return (self.tracker == "own" and not self.override) or time.monotonic() < self.eyes_until
return (self.tracker == "own" and not self.override and self.awake) or time.monotonic() < self.eyes_until
def start_eyes(self):
"""Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
@@ -700,6 +792,14 @@ class Service:
reply = "ok"
except ValueError:
reply = "error bad seconds"
elif words[:1] == ["wake"] and len(words) == 2:
try:
secs = min(max(float(words[1]), 0.0), WAKE_MAX)
self.wake_until = max(self.wake_until, time.monotonic() + secs)
self.update_awake()
reply = "ok"
except ValueError:
reply = "error bad seconds"
elif words[:1] == ["reload"]:
self.load_settings()
self.load_calibration()
@@ -750,6 +850,7 @@ class Service:
st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
lessons=max(len(m) for m in w.misses), own_running=bool(own),
own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
st.update(awake=self.awake, idle=None if self.awake else self.idle_reason())
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
tracker_setting=self.tracker_setting, own_installed=own_installed())
st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
@@ -770,6 +871,7 @@ class Service:
if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
and own_installed() != (self.tracker == "own")):
self.load_settings()
self.update_awake()
want = self.eyes_wanted()
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
self.start_eyes()
@@ -790,7 +892,7 @@ class Service:
next_verbose = time.monotonic() + 5
while self.running:
now = time.monotonic()
if not self.proc and now >= self.restart_at:
if self.awake and not self.proc and now >= self.restart_at:
self.start_helper()
for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5):
if key.data == "control":
@@ -808,6 +910,7 @@ class Service:
elif key.data == "stderr" and self.proc:
self.read_stderr()
self.checks.tick()
self.sync_sources()
if now >= next_periodic:
self.periodic()
next_periodic = now + 1.0
+84 -16
View File
@@ -10,7 +10,9 @@ directions: look at each one.
once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse
button or key combination mapped to Gaze quick check), and when a click's correction
was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is
far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing.
far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing. The headset going
on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see
ft-gazed), so it also opens when gaze mode comes on after the service idled.
five the middle and four around it, when the first FIVE_COUNT lessons after a quick check
were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
@@ -30,6 +32,9 @@ directions: look at each one.
adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
A check asked for while the gaze service idles (quickcal, calibrate, fitcheck) wakes it and
waits until the tracker sends, at most ft-gazed's WAKE_SETTLE; then it opens, or logs why not.
The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
@@ -43,6 +48,13 @@ click with nothing taken after ACCEPT_WAIT, and a failed calibration names its m
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
renewed every second; the helper shows it again by itself when that stops).
Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (it maps pupils
to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs
anyway ("blind"): someone in the headset (SteamVR's tracker sees an eye) and ft-eyes answering
are enough to start it, each dot stands in for the gaze, and a click takes the CHECK_WINDOW up
to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point)
and says why not. Without this, a fresh install could never calibrate our tracker.
What a capture teaches:
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
calib-start, a calib-point for each dot, calib-fit (its calibration)
@@ -209,7 +221,9 @@ class Checks:
sel.register(self.out, selectors.EVENT_READ, "checks")
self.check = None
self.gaze_on = None
self.headset = None # someone wears it (the helper's "worn"), False "away", None unknown
self.gaze_heard = 0.0
self.pending = None # (command words, when): asked for while the gaze service idled
self.full_armed = True # gaze mode on without a calibration opens the full one (need_full)
self.full_blocked = None # why it can't open now
self.full_retry_at = 0.0
@@ -294,17 +308,22 @@ class Checks:
pass
def on_readable(self):
"""Replies on our socket: the helper's "ok on|off" to "gaze ?"; the panel's are dropped."""
"""Replies on our socket: the helper's "ok on|off [worn|away]" to "gaze ? headset" (an
older helper leaves the headset out); the panel's are dropped."""
while True:
try:
data = self.out.recv(4096).decode("utf-8", "replace")
words = self.out.recv(4096).decode("utf-8", "replace").split()
except (BlockingIOError, OSError):
return
if data in ("ok on", "ok off"):
on = data == "ok on"
was, self.gaze_on, self.gaze_heard = self.gaze_on, on, time.monotonic()
if on and was is False:
if words[:1] == ["ok"] and len(words) in (2, 3) and words[1] in ("on", "off"):
on = words[1] == "on"
headset = None if len(words) == 2 else words[2] == "worn"
was = (self.gaze_on, self.headset)
self.gaze_on, self.headset, self.gaze_heard = on, headset, time.monotonic()
if on and was[0] is False:
self.on_gaze_on()
if was != (on, headset):
self.svc.update_awake()
# --- State ---
@@ -324,9 +343,17 @@ class Checks:
return time.monotonic() - self.seen_at < within
def can_run(self):
"""Someone's in the headset and the tracker is sending."""
"""Someone's in the headset and the tracker is sending. Our tracker sends no gaze before
its first calibration: then ft-eyes answering (calibrated() is False only once it has)
is enough, since the calibration is what it needs (see "first calibration" at the top)."""
if self.blind():
return self.eyes_seen()
return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
def blind(self):
"""Our tracker is in use, running, and not calibrated yet: it has no gaze to send."""
return self.svc.kind == "own" and self.calibrated() is False
def on_gaze_on(self):
self.full_armed = True
self.need_full("gaze mode came on without a calibration")
@@ -341,8 +368,10 @@ class Checks:
self.full_blocked = None
return
now = time.monotonic()
if not self.can_run() and self.svc.waking():
return # the tracker is still starting (the service idled)
if not self.can_run():
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 and not self.blind()
else "no eyes seen (is the headset on?)")
elif now < self.full_retry_at:
return
@@ -394,6 +423,9 @@ class Checks:
if not self.can_run():
return "error the headset is off or the tracker isn't sending"
own = svc.kind == "own"
blind = self.blind()
if blind and kind != "full":
return "error our tracker isn't calibrated yet: use Calibrate"
if kind == "full" and own:
reply = ask(EYES, "calib-start", 3.0)
if not reply.startswith("ok"):
@@ -402,8 +434,10 @@ class Checks:
now = time.monotonic()
self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
"started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
log(f"{kind} check: {reason}")
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": "",
"blind": blind}
log(f"{kind} check: {reason}" + (" (our tracker's first: no gaze yet, so each click takes the look "
"up to it)" if blind else ""))
if kind == "full":
st = ask(SCREENS, "state", 0.5).split()
if len(st) >= 3 and st[0] == "ok":
@@ -490,6 +524,8 @@ class Checks:
def on_sample(self, s):
self.sample_at = time.monotonic()
if self.pending:
self.run_pending()
unc = (s["src"].get("mmap1") or {}).get("unc")
if unc and min(unc) <= EYE_LOST:
self.seen_at = time.monotonic()
@@ -503,6 +539,12 @@ class Checks:
return
if c["own"]:
src = s["src"].get("own") or {}
if c["blind"]:
# Our tracker's first calibration: no gaze yet, so the dot stands in for it (the
# gaze can't seem to move) and a click takes the look up to it. ft-eyes checks
# the pupils held still (see the top).
yaw, pitch, _ = c["dots"][c["i"]]
src = {"hy": yaw, "hp": pitch}
else:
src = s["src"].get("mmap1") or {}
if "hy" not in src:
@@ -742,9 +784,28 @@ class Checks:
# --- From the service ---
def command(self, words):
def run_pending(self):
"""A check asked for while the service idled: once the tracker sends (and our tracker has
said whether it's calibrated), or WAKE_SETTLE after waking, when its error is the real one."""
svc = self.svc
words, _ = self.pending
ready = (time.monotonic() - svc.last_sample < 2 if words[0] == "fitcheck" else self.can_run()) \
and (svc.kind != "own" or self.calibrated() is not None)
if not ready and svc.waking():
return
self.pending = None
reply = self.command(words, queue=False)
if reply != "ok":
log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}")
def command(self, words, queue=True):
"""quickcal, calibrate, calaccept, calquit -> a reply."""
cmd = words[0]
if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check:
# The tracker isn't running (or only just started): wake it, and do this once it sends.
self.pending = (words, time.monotonic())
self.svc.update_awake()
return "ok waking the eye tracker first"
if cmd == "quickcal":
return self.start("full" if self.calibrated() is False else "quick", "asked for")
if cmd == "calibrate":
@@ -820,9 +881,11 @@ class Checks:
now = time.monotonic()
if not self.panel_proc and now >= self.panel_restart_at:
self.start_panel()
self.to_helper("gaze ?")
self.to_helper("gaze ? headset")
if now - self.gaze_heard > 5:
self.gaze_on = None # the helper isn't answering
self.gaze_on = self.headset = None # the helper isn't answering
if self.pending:
self.run_pending()
if self.check:
self.to_helper("calpanel 1")
if not self.eyes_seen(AWAY_MIN):
@@ -831,7 +894,11 @@ class Checks:
self.back_since = None # gone again before DON_DELAY
elif self.back_since is not None and now - self.back_since >= DON_DELAY:
self.away, self.back_since = False, None
self.full_armed = True # a calibration that closed unfinished opens again
if not self.check:
# A calibration that closed unfinished opens again. Not one still open: gaze mode
# coming on wakes our tracker, so its eyes come back just as the calibration
# opens, and re-arming then opened a second one when the first ended.
self.full_armed = True
self.auto_quick("the headset went on")
if svc.kind == "own" and now - svc.own_at < 5:
reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
@@ -844,7 +911,8 @@ class Checks:
def status(self):
c = self.check
st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
st = {"check": None, "gaze_mode": self.gaze_on, "headset_worn": self.headset, "pending": self.pending[0][0]
if self.pending else None, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
"problem": self.problem(),
"panel": self.panel_proc is not None,
"last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
+6 -1
View File
@@ -47,6 +47,7 @@
#include <drm_fourcc.h>
#include <fcntl.h>
#include <gbm.h>
#include <poll.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
@@ -524,7 +525,11 @@ int main(int argc, char **argv) {
if (!shown) ov->ShowOverlay(h), shown = true;
dirty = false;
}
std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
// Until a command comes, or 10 ms while shown (SteamVR's events). Hidden, it waits up to a
// second: it woke 20 to 30 times a second for nothing, the main cost left with gaze idle.
// A closed stdin (--watch-stdin) wakes it too, so quitting doesn't wait.
pollfd fds[2] = {{sock, POLLIN, 0}, {0, POLLIN, 0}};
poll(fds, watchStdin ? 2 : 1, visible ? 10 : 1000);
}
ov->DestroyOverlay(h);
buffers.Drop();
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""Offline test of our own eye tracker's first calibration (gaze/gazecheck.py, "blind"): before
it has a calibration, ft-eyes publishes no gaze, and the calibration must still open and take
its dots. Until 2026-10-05 it couldn't: a fresh install that chose our tracker never calibrated.
Runs ft-gazed's Service with its sockets renamed and HOME in a temp folder (state and settings
go there), a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR sees both
eyes; "own" is {"ok":0}, as with an uncalibrated ft-eyes), a fake ft-eyes control socket, and no
panel (a stand-in process). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
or SteamVR, so it's safe next to them.
gaze/test/first-calibration-test.py
"""
import os
import tempfile
HOME = tempfile.mkdtemp(prefix="ft-gaze-first-cal-test-")
os.environ["HOME"] = HOME # before gazecal: its STATE, and frametop.conf, follow HOME
import importlib.machinery # noqa: E402
import importlib.util # noqa: E402
import json # noqa: E402
import selectors # noqa: E402
import shutil # noqa: E402
import socket # noqa: E402
import subprocess # noqa: E402
import sys # noqa: E402
import threading # noqa: E402
import time # noqa: E402
HERE = os.path.dirname(os.path.abspath(__file__))
GAZE = os.path.join(HERE, "..")
sys.path.insert(0, GAZE)
loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
loader.exec_module(gazed)
import gazecheck # noqa: E402 (the module ft-gazed imported)
tag = f"ft_gaze_first_cal_test_{os.getpid()}"
gazed.ME = f"\0{tag}_gazed"
gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL = f"\0{tag}_panel"
gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes"
gazed.read_settings = lambda: ("own", "auto", "auto", 55.0)
DOTS = 3
real_dots = gazecheck.check_dots
gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS] # a short calibration
logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg)
# The fake ft-gaze: 90 samples a second, SteamVR sees both eyes, no gaze from ours.
FAKE = os.path.join(HOME, "ft-gaze")
with open(FAKE, "w") as f:
f.write('''import json, os, select, sys, time
while True:
if select.select([sys.stdin], [], [], 1 / 90)[0]:
if not os.read(0, 4096):
break
print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
"open": [0.8, 0.8]}, "own": {"ok": 0}}}), flush=True)
''')
SLEEPER = [sys.executable, "-c", "import sys; sys.stdin.read()"] # quits when its stdin closes
def start_helper(self):
"""ft-gaze, straight from here instead of the dev container."""
self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
self.proc_sources = self.wanted_sources()
os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
self.buf = b""
def start_eyes(self):
"""ft-eyes' process: a stand-in. Its control socket is the fake below."""
self.eyes_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
os.set_blocking(self.eyes_proc.stderr.fileno(), False)
self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
def start_panel(self):
self.panel_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
os.set_blocking(self.panel_proc.stderr.fileno(), False)
self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
gazed.Service.start_helper = start_helper
gazed.Service.start_eyes = start_eyes
gazecheck.Checks.start_panel = start_panel
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that.
eyes_state = {"cal": None, "points": [], "fail": None}
eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
eyes.bind(gazed.EYES_SOCKET)
eyes.settimeout(0.2)
def eyes_answer():
while True:
try:
data, addr = eyes.recvfrom(512)
except socket.timeout:
continue
except OSError:
return
w = data.decode().split()
if w[0] == "status":
reply = json.dumps({"calibration": eyes_state["cal"], "calibrating": False, "dots": 0,
"eyes": {"right": {"reseat": False}, "left": {"reseat": False}}})
elif w[0] == "calib-start":
eyes_state["points"] = []
reply = "ok"
elif w[0] == "calib-point":
eyes_state["points"].append(tuple(map(float, w[1:5])))
reply, eyes_state["fail"] = eyes_state["fail"] or "ok 50 50 1.00 1.00", None
elif w[0] == "calib-fit":
eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])}
reply = f"ok {len(eyes_state['points'])} dots"
else:
reply = f"fail unknown command {w[0]}"
if addr:
eyes.sendto(reply.encode(), addr)
threading.Thread(target=eyes_answer, daemon=True).start()
helper_state = {"reply": "ok off worn", "heard": []}
helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
helper.bind(gazed.POINTER)
helper.settimeout(0.2)
def helper_answer():
while True:
try:
data, addr = helper.recvfrom(512)
except socket.timeout:
continue
except OSError:
return
if data.startswith(b"gaze ?") and addr:
helper.sendto(helper_state["reply"].encode(), addr)
else:
helper_state["heard"].append(data.decode())
threading.Thread(target=helper_answer, daemon=True).start()
svc = gazed.Service(None, False, gazed.POINTER)
threading.Thread(target=svc.run, daemon=True).start()
ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
ctl.bind("")
ctl.settimeout(3)
def ask(cmd):
ctl.sendto(cmd.encode(), gazed.ME)
return ctl.recv(65536).decode()
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
def wait(cond, seconds):
end = time.monotonic() + seconds
while time.monotonic() < end:
if cond():
return True
time.sleep(0.05)
return cond()
def check_state():
return svc.checks.check or {}
def take_dot(i):
"""Wait for dot i to settle, then click: is it taken?"""
wait(lambda: check_state().get("i") == i and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
before = check_state().get("captured", 0)
ask("calaccept")
return wait(lambda: check_state().get("captured", 0) > before, 2)
check("gaze mode off, Gaze page open (wake): ours runs, uncalibrated", ask("wake 60"), "ok")
check("the service knows ours has no calibration", wait(lambda: svc.checks.calibrated() is False, 6), True)
check("a quick check is refused while ours has no calibration", svc.checks.start("quick", "test"),
"error our tracker isn't calibrated yet: use Calibrate")
helper_state["reply"] = "ok on worn"
check("gaze mode on: the calibration opens by itself", wait(lambda: check_state().get("kind") == "full", 6), True)
check("it runs blind (no gaze from ours yet)", check_state().get("blind"), True)
check("nothing says it can't open", any("can't open" in m for m in logs), False)
# Live 2026-10-05: turning gaze mode on woke our tracker, and the calibration opened before
# DON_DELAY of eyes had passed, so "the headset went on" came while it was open. That re-armed
# it, and a second calibration opened as soon as the first ended.
svc.checks.away, svc.checks.back_since = True, None
check("dot 1: a click takes it", take_dot(0), True)
check("the headset went on while it was open",
wait(lambda: not svc.checks.away, gazecheck.DON_DELAY + 2) and bool(svc.checks.check), True)
t0, t1, yaw, pitch = eyes_state["points"][0]
dot = gazecheck.check_dots("full", True)[0]
check("its window is the look up to the click (about CHECK_WINDOW)",
abs((t1 - t0) - gazecheck.CHECK_WINDOW) < 0.15, True)
check("ft-eyes got that dot's direction", (round(yaw, 3), round(pitch, 3)), (round(dot[0], 3), round(dot[1], 3)))
eyes_state["fail"] = "fail the left eye was seen in only 3 frames"
wait(lambda: check_state().get("i") == 1 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
ask("calaccept")
check("ft-eyes refusing a dot: its reason reaches the panel's note",
wait(lambda: "left eye in only 3 frames" in check_state().get("note", ""), 2), True)
check("dot 2, second try: taken", take_dot(1), True)
check("dot 3: taken", take_dot(2), True)
check("all dots: ours fits its calibration (calib-fit)",
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 4), True)
check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is True, 4), True)
check("and gaze mode stays on", "gaze off" in helper_state["heard"], False)
check("and no second calibration opens", wait(lambda: svc.checks.check is not None, 3), False)
check("one calibration in the log", sum(m.startswith("full check:") for m in logs), 1)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
svc.running = False
time.sleep(0.7)
shutil.rmtree(HOME, ignore_errors=True)
os._exit(1 if failures else 0)
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Offline test of the gaze service idling (gaze/ft-gazed, gaze/gazecheck.py): ft-gaze runs only
while the gaze is in use, and a check asked for while it idles waits for the tracker.
Runs ft-gazed's Service with its sockets renamed, a fake pointer helper (answers "gaze ?
headset" as the test says), and a fake ft-gaze (prints samples, quits when its stdin closes).
SteamVR's tracker is the one in use, so our own isn't started; the panel isn't either. Nothing
reaches the live gaze service, the pointer helper, or SteamVR, so it's safe next to them.
gaze/test/idle-test.py
"""
import importlib.machinery
import importlib.util
import os
import socket
import subprocess
import sys
import tempfile
import threading
import time
HERE = os.path.dirname(os.path.abspath(__file__))
GAZE = os.path.join(HERE, "..")
sys.path.insert(0, GAZE)
loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
loader.exec_module(gazed)
import gazecheck # noqa: E402 (the module ft-gazed imported)
tag = f"ft_gaze_idle_test_{os.getpid()}"
gazed.ME = f"\0{tag}_gazed"
gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel" # "isn't built": no panel
gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0
gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg)
# The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes. It logs the
# sources it was asked for: "argv LIST" (--sources), then "line LIST" for each "sources LIST".
tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-")
FAKE = os.path.join(tmp, "ft-gaze")
SOURCES_LOG = os.path.join(tmp, "sources.log")
with open(FAKE, "w") as f:
f.write('''import json, os, select, sys, time
log = open(sys.argv[1], "a", buffering=1)
log.write("argv " + (sys.argv[sys.argv.index("--sources") + 1] if "--sources" in sys.argv else "-") + "\\n")
buf = b""
while True:
if select.select([sys.stdin], [], [], 1 / 90)[0]:
data = os.read(0, 4096)
if not data:
break
buf += data
while b"\\n" in buf:
line, buf = buf.split(b"\\n", 1)
if line.startswith(b"sources "):
log.write("line " + line[8:].decode() + "\\n")
eye = {"hy": 1.0, "hp": 2.0}
print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
"open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True)
''')
started = []
def start_helper(self):
"""ft-gaze, straight from here instead of the dev container."""
import selectors
self.proc = subprocess.Popen([sys.executable, FAKE, SOURCES_LOG, "--sources", self.wanted_sources()],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
self.proc_sources = self.wanted_sources()
os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
self.buf = b""
started.append(time.monotonic())
gazed.Service.start_helper = start_helper
# The fake pointer helper.
helper_state = {"reply": "ok off worn"}
helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
helper.bind(gazed.POINTER)
helper.settimeout(0.2)
def answer():
while True:
try:
data, addr = helper.recvfrom(512)
except socket.timeout:
continue
except OSError:
return
if data.startswith(b"gaze ?") and addr:
helper.sendto(helper_state["reply"].encode(), addr)
threading.Thread(target=answer, daemon=True).start()
svc = gazed.Service(None, False, gazed.POINTER)
threading.Thread(target=svc.run, daemon=True).start()
ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
ctl.bind("")
ctl.settimeout(2)
def ask(cmd):
ctl.sendto(cmd.encode(), gazed.ME)
return ctl.recv(4096).decode()
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
def wait(cond, seconds):
end = time.monotonic() + seconds
while time.monotonic() < end:
if cond():
return True
time.sleep(0.05)
return cond()
time.sleep(2.5)
check("gaze mode off: idle, no ft-gaze", (svc.awake, svc.proc is None), (False, True))
check("status says why", ask("status").count('"idle": "gaze mode is off"'), 1)
helper_state["reply"] = "ok on worn"
check("gaze mode on, headset worn: awake within 2 s", wait(lambda: svc.awake and svc.proc is not None, 2.5), True)
check("samples come in", wait(lambda: svc.counts["samples"] > 20, 2), True)
helper_state["reply"] = "ok on away"
check("headset off: still awake for IDLE_AFTER", wait(lambda: not svc.awake, 0.5), False)
check("then idle, ft-gaze stopped", wait(lambda: not svc.awake and svc.proc is None, 3.5), True)
check("why: nobody wears it", ask("status").count('"idle": "nobody is wearing the headset"'), 1)
helper_state["reply"] = "ok on"
check("an older helper (no headset word): awake with gaze mode on", wait(lambda: svc.awake, 2.5), True)
helper_state["reply"] = "ok off worn"
check("gaze mode off again: idle", wait(lambda: not svc.awake and svc.proc is None, 4.5), True)
check("wake lease", ask("wake 2"), "ok")
check("wake: awake at once", (svc.awake, wait(lambda: svc.proc is not None, 1)), (True, True))
check("lease over (2 s + IDLE_AFTER): idle", wait(lambda: not svc.awake, 4.5), True)
time.sleep(0.5)
logs.clear()
open(SOURCES_LOG, "w").close()
count = len(started)
check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first")
check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True)
check("once the tracker sends, it ran (and says why it couldn't open)",
wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True)
check("nothing left pending", svc.checks.pending, None)
sources = open(SOURCES_LOG).read().split("\n")
check("ft-gaze started with every source for the check", sources[0], "argv all")
in_use = svc.wanted_sources()
check("then only those in use (SteamVR's tracker: not the action, not own)",
(f"line {in_use}" in sources, in_use != "all", "action" in in_use, "own" in in_use), (True, True, False, False))
check("idle again after", wait(lambda: not svc.awake, 3), True)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
svc.running = False
time.sleep(0.7)
os.remove(FAKE)
os.remove(SOURCES_LOG)
os.rmdir(tmp)
os._exit(1 if failures else 0)
+6 -1
View File
@@ -9,7 +9,12 @@ the search runs again with a smaller closing.
import cv2
import numpy as np
DARK = 30 # pupil pixels are below this (the face around it is 40-180)
# One thread: OpenCV's pool of one per core costs more than it saves on a frame this small
# (a 140-240 px window while it follows the pupil). Its idle workers spun and yielded about
# 14,000 times a second each, a quarter of a core, beside SteamVR's compositor.
cv2.setNumThreads(1)
DARK = 30 # pupil pixels are below this (the face around it is 40-180)
MIN_AREA = 150 # pupil area range in pixels
MAX_AREA = 20000
MIN_FILL = 0.75 # blob area / fitted-ellipse area
+7
View File
@@ -63,6 +63,13 @@ reading only.
frame when its camera starts the frame after next (no slot is rewritten sooner than three
frames), ignores late writes to the frame just finished, and saves from a separate
thread. fit1 may hold a few percent of torn frames.
- `--share` (2026-10-03) checks only the slot each camera writes next, from the two before
(the orders above, learned again if they change; all four slots for 2 s after a frame turns
up elsewhere), sleeps until 2.5 ms before the next frame is due, then looks every 1 ms with
0.5 ms of timer slack. Against the 0.3 ms poll of all eight slots, on simulated cameras:
207 wakeups a second instead of 1,486, 0.8% of a core instead of 3.6% (more on the real
DMA-BUF memory), no torn or skipped frames, and a frame's start seen 1.35 ms late on
average instead of 0.76. `--rec` still polls all slots every 0.3 ms, for its times.
- Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so
recordings are empty then.
- **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time):
+73 -16
View File
@@ -46,6 +46,7 @@
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <time.h>
@@ -288,20 +289,53 @@ static int eye_buffer(void) {
// Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass
// (forever if negative), a stop signal comes, the tracker process goes away, or keep()
// (checked about every 0.25 s, when given) says to stop.
// (checked about every 0.25 s, when given) says to stop. Looks every `poll_us`.
//
// A frame lands over several milliseconds, in bursts, and its last bursts can come after
// the camera has started its next frame. A slot isn't rewritten until at least three frames
// later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when
// its camera starts the frame after next. Changes to the slot just finished are late bursts,
// not a new frame. A frame's time is when its slot first changed.
//
// Each look checks only the slot each camera writes next: the one that followed the last two
// before (after[][], seeded with the orders above and learned as frames come). Every check
// reads 256 words spread over a frame in DMA-BUF memory, so all eight slots each time was
// most of the cost. A camera with nothing in its expected slot for 1.5 frames, or with no
// order known yet, has all its slots checked until a frame comes. When that finds a frame
// somewhere else (the order changed, or a frame was missed), all its slots are checked for
// SCAN_AFTER_MISS, as before, while after[][] learns the new order. A slot's fingerprint is
// taken again when it stops being one of the two in use, so a check later sees only a new frame.
// Between frames it sleeps until FRAME_EARLY before the next one is due (the cameras run at
// 90 fps, within a ms of each other), then looks every `poll_us`.
#define FRAME_DUE (1.5 / 90) // s: a camera that hasn't started a frame by then gets all its slots checked
#define SCAN_AFTER_MISS 2.0 // s of checking all slots after a frame came in an unexpected one
#define CAMS_IDLE 0.5 // s without a frame from either camera: look 4 times less often
#define FRAME_EARLY 0.0025 // s before a frame is due to start looking for it
// When to start looking for the next frame: FRAME_EARLY before the first camera's is due. A
// camera already late (it lost its order, or stopped) means now.
static double next_due(const double last[2], double t) {
double due = 1e300;
for (int cam = 0; cam < 2; cam++) {
double d = last[cam] + 1.0 / 90 - FRAME_EARLY;
if (t - last[cam] > CAMS_IDLE) continue; // stopped: the other one sets the pace
if (d < due) due = d;
}
return due < 1e300 ? due : t;
}
static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double),
int (*keep)(void)) {
int (*keep)(void), unsigned poll_us) {
static const int order[2][8] = {{3, 0, 1, 2, 3, 0, 1, 2}, {7, 5, 4, 6, 5, 7, 6, 4}};
int after[EYE_SLOTS][EYE_SLOTS];
memset(after, -1, sizeof after);
for (int c = 0; c < 2; c++)
for (int i = 0; i < 8; i++) after[order[c][i]][order[c][(i + 1) % 8]] = order[c][(i + 2) % 8];
uint64_t sig[EYE_SLOTS];
double first[EYE_SLOTS];
int cur[2] = {-1, -1}, prev[2] = {-1, -1};
for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0;
double start = now(), checked = start, kept = start;
double start = now(), checked = start, kept = start, last[2] = {start, start}, scan_until[2] = {0, 0};
char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid);
while ((seconds < 0 || now() - start < seconds) && !stop_rec) {
@@ -315,18 +349,36 @@ static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8
if (!keep()) return;
kept = t;
}
for (int k = 0; k < EYE_SLOTS; k++) {
uint64_t s = frame_sig(bufs[b].p + slot_start(k));
if (s == sig[k]) continue;
sig[k] = s;
int cam = k >= 4;
if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst
if (prev[cam] >= 0) done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
prev[cam] = cur[cam];
cur[cam] = k;
first[k] = t;
for (int cam = 0; cam < 2; cam++) {
int next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1;
int all = next < 0 || t - last[cam] > FRAME_DUE || t < scan_until[cam];
int from = all ? cam * 4 : next, to = all ? cam * 4 + 4 : next + 1;
for (int k = from; k < to; k++) {
uint64_t s = frame_sig(bufs[b].p + slot_start(k));
if (s == sig[k]) continue;
sig[k] = s;
if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst
if (next >= 0 && k != next) scan_until[cam] = t + SCAN_AFTER_MISS;
if (prev[cam] >= 0) {
done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
after[prev[cam]][cur[cam]] = k;
// Out of use from now on: later changes are a new frame.
sig[prev[cam]] = frame_sig(bufs[b].p + slot_start(prev[cam]));
}
prev[cam] = cur[cam];
cur[cam] = k;
first[k] = t;
last[cam] = t;
next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1;
}
}
usleep(300);
double wait = poll_us * 1e-6;
if (t - last[0] > CAMS_IDLE && t - last[1] > CAMS_IDLE) {
wait *= 4;
} else if (next_due(last, t) - t > wait) {
wait = next_due(last, t) - t;
}
usleep((useconds_t)(wait * 1e6));
}
}
@@ -356,7 +408,7 @@ static int rec(double seconds, const char *dir, int pid) {
pthread_t writer;
pthread_create(&writer, NULL, ring_writer, NULL);
double start = now();
poll_frames(b, seconds, pid, rec_frame, NULL);
poll_frames(b, seconds, pid, rec_frame, NULL, 300); // 0.3 ms: recordings' times
pthread_mutex_lock(&ring.mu);
ring.done = 1;
pthread_cond_signal(&ring.cv);
@@ -383,6 +435,10 @@ static int rec(double seconds, const char *dir, int pid) {
#define SHARE_SLOTS 8
#define SHARE_MAGIC 0x31434546u // "FEC1"
#define WANT_FRESH 3.0 // seconds a touch of the --want file lasts
// How often --share looks for frames, with a timer slack that lets the kernel group the
// wakeups: a frame reaches ft-eyes 1 to 1.5 ms after its camera starts the next, not 0.3.
#define SHARE_POLL_US 1000
#define SHARE_SLACK_NS 500000
typedef struct {
uint32_t magic, version, width, height, slots, entry_size;
@@ -462,6 +518,7 @@ static int share_loop(const char *path) {
.slots = SHARE_SLOTS, .entry_size = (uint32_t)esize};
signal(SIGINT, on_stop);
signal(SIGTERM, on_stop);
if (prctl(PR_SET_TIMERSLACK, SHARE_SLACK_NS, 0, 0, 0) != 0) perror("PR_SET_TIMERSLACK");
fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "",
want_path ? want_path : "");
int pid = -1, idle = -1, missing = 0;
@@ -486,7 +543,7 @@ static int share_loop(const char *path) {
if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid);
idle = 0, missing = 0;
h->tracker_pid = pid;
poll_frames(eye_buffer(), -1, pid, share_frame, wanted);
poll_frames(eye_buffer(), -1, pid, share_frame, wanted, SHARE_POLL_US);
struct stat st;
char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid);
+42 -4
View File
@@ -50,6 +50,7 @@ import json
import math
import mmap
import os
import select
import socket
import struct
import sys
@@ -58,7 +59,12 @@ import time
from collections import deque
from pathlib import Path
import numpy as np
# One thread for numpy's BLAS and OpenMP, set before numpy loads: it would start one per core
# (8 here) for small arrays that never need them. eyes_pupil keeps OpenCV to one as well.
for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS"):
os.environ.setdefault(_var, "1")
import numpy as np # noqa: E402
sys.path.insert(0, str(Path(__file__).resolve().parent))
import eyes_model # noqa: E402
@@ -82,6 +88,15 @@ GAP = 3.0 # s without frames: the headset was off, and may sit diffe
CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation)
CALIB_MIN = 15 # frames an eye needs in a calibration dot's window
CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot
# Waiting for frames. They come only as a counter in shared memory, so there's nothing to block
# on: sleep until a camera's next frame is due, then look every POLL. ft-eyegrab passes each one
# on within a ms or two of when its camera starts the one after, so they arrive 11.1 ms apart,
# give or take that.
PERIOD = 1 / 90 # s between a camera's frames
EARLY = 0.002 # start looking this long before a frame is due
POLL = 0.001 # then this often until it comes
STALLED = 0.1 # s without a frame: that camera stopped (headset off), and isn't waited for
IDLE_POLL = 0.02 # how often to look while both are stopped
class Cams:
@@ -352,7 +367,7 @@ def wait_for_cams(sock, tracker, want):
return Cams()
except (OSError, ValueError, RuntimeError):
serve(sock, tracker)
time.sleep(0.2)
select.select([sock], [], [], 0.2)
def serve(sock, tracker):
@@ -372,7 +387,24 @@ def serve(sock, tracker):
pass
def below_steamvr():
"""Nice 10 and SCHED_BATCH, for this thread and those it starts. ft-eyes runs in the dev
container's podman scope, where the gaze service's unit doesn't reach it, so it ran at
nice 0 on the cores vrcompositor and vrserver use. Batch also lets a waking ft-eyes wait
for the running task's turn instead of taking the core: a frame a few ms late costs the
gaze little, a late compositor frame costs a dropped frame in the headset."""
try:
os.setpriority(os.PRIO_PROCESS, 0, max(os.getpriority(os.PRIO_PROCESS, 0), 10))
except OSError as e:
print(f"ft-eyes: nice: {e}", file=sys.stderr, flush=True)
try:
os.sched_setscheduler(0, os.SCHED_BATCH, os.sched_param(0))
except (OSError, AttributeError) as e:
print(f"ft-eyes: SCHED_BATCH: {e}", file=sys.stderr, flush=True)
def main():
below_steamvr()
verbose = "-v" in sys.argv
if "--watch-stdin" in sys.argv:
# Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our
@@ -398,6 +430,7 @@ def main():
print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True)
seen = [cams.count(0), cams.count(1)]
report = time.monotonic()
arrived = [0.0, 0.0] # when each camera's newest frame was seen (monotonic)
while True:
serve(sock, tracker)
want()
@@ -407,6 +440,7 @@ def main():
if n == seen[c]:
continue
seen[c] = n
arrived[c] = time.monotonic()
got = cams.frame(c, n - 1) # only the newest: never fall behind
if got:
eyes[c].feed(*got)
@@ -425,8 +459,6 @@ def main():
shifts += [float(v) for v in e.shift.value]
pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan]
out.write(latest, (yaw, pitch), flags, per, shifts, pupils)
else:
time.sleep(0.001)
now = time.monotonic()
if now - report >= 5:
if verbose:
@@ -444,6 +476,12 @@ def main():
print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True)
cams = wait_for_cams(sock, tracker, want)
seen = [cams.count(0), cams.count(1)]
# Until the next frame is due (a command on the socket wakes us sooner).
wait = IDLE_POLL
for c in (0, 1):
if now - arrived[c] < STALLED:
wait = min(wait, max(arrived[c] + PERIOD - EARLY - now, POLL))
select.select([sock], [], [], wait)
if __name__ == "__main__":
+8 -1
View File
@@ -9,6 +9,7 @@
# Options (piped, they go after "bash -s --"):
# --stable the main branch: tested releases (the default for a new install)
# --experimental the experimental branch: the newest features, less tested
# --branch NAME another branch, such as a fix to test before it's released
# --dir DIR where the repo goes (default ~/frametop)
# --clone-only get or update the repo, but don't run install.sh
# --yes, --no-bluetooth passed to install.sh (--yes also answers this script's question:
@@ -17,7 +18,7 @@ set -euo pipefail
usage() {
cat <<'EOF'
usage: get.sh [--stable | --experimental] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
EOF
}
@@ -31,6 +32,7 @@ main() {
case $1 in
--stable) branch=main ;;
--experimental) branch=experimental ;;
--branch) branch=${2:?--branch needs a branch name}; shift ;;
--dir) dir=${2:?--dir needs a folder}; shift ;;
--clone-only) clone_only=1 ;;
--yes) yes=1; pass+=("$1") ;;
@@ -80,6 +82,11 @@ main() {
fi
fi
if ! git ls-remote --exit-code --heads "$repo" "$branch" >/dev/null; then
echo "Frametop has no branch called $branch (or GitHub can't be reached)." >&2
return 1
fi
if [ ! -e "$dir" ]; then
echo "Cloning Frametop ($branch) into $dir"
git clone --branch "$branch" "$repo" "$dir"
+68
View File
@@ -0,0 +1,68 @@
---
title: Install the Frametop Hand Recorder
---
# Install the Frametop Hand Recorder
The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face). These are the Konsole commands to install it.
> **Fixed in Frametop 0.2.1 (October 5, 2026):** the recorder now works on headsets without the Arcturus color passthrough module too. If you installed it earlier and the camera check stopped with "Not all of the headset's tracking cameras are running (ft-camd publishes only 2 of 4 mono cameras ...)", run the commands under [Update](#update).
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions and help with recording.
For now the recorder runs inside Frametop's desktop, so these steps install Frametop first. A standalone recorder that runs from the SteamVR dashboard without Frametop is planned.
## Before you start
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/DeeJanuz/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
## Open Konsole
1. In the launcher, choose Launch a program → Desktop.
2. In the application menu, open System → Konsole.
## 1. Install Frametop
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
```
This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
## 2. Install the Hand Recorder
After SteamVR restarts, choose Launch a program → Desktop again, open Konsole, and run:
```
~/frametop/hands/rec/install.sh
```
This builds the camera broker, the hand tracker and the headset panel (the first build takes a few minutes), asks for your `sudo` password once to let the camera broker read the cameras, and adds Frametop Hand Recorder to the application menu.
## 3. Record
Open Frametop Hand Recorder from the desktop's application menu. It walks you through the consent text, a short checklist, the recording, a review of what you recorded, and the upload to Hugging Face. Nothing leaves the headset until you press Upload.
## Update
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
~/frametop/hands/rec/install.sh
```
Run the second command after SteamVR has restarted, as in the install.
## Uninstall
```
~/frametop/hands/rec/install.sh uninstall
```
This removes the menu entry. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/DeeJanuz/frametop#uninstall) in the README.
## Help
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/DeeJanuz/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
+1 -1
View File
@@ -36,7 +36,7 @@ hands/run.sh uninstall
Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them), read when ft-camd and ft-hands start:
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks and logs a warning if the hands disagree.
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
+42 -5
View File
@@ -56,13 +56,19 @@ ANSI = re.compile(r"\x1b\[[0-9;]*m")
STAMP = re.compile(r"^\w{3} \w{3} \d{2} \d{4} (\d{2}:\d{2}:\d{2})\.\d+ (\w+): ?(.*)$")
# Lines worth reading; anything else is skipped before the regexes (the log grows by MBs a day).
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Closing tracking camera", "Streaming",
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService")
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService", "ISP ")
# XRService's numbering of its tracking cameras (the TrackingCameraInit index): ft-hands names
# them this way too (track/main.cpp, cameras_from_xrservice_log).
NAMES = ("slam_left", "slam_right", "upper_left", "upper_right")
RE_PASSTHRU = re.compile(r"Passthrough connected but FPGA is (\S+) - loading VCINT")
RE_INTERLEAVE = re.compile(r"Upper cameras FPGA interleaving support: (\d)")
RE_TASKS = re.compile(r"Created (\d+) tasks \((\d+) tracking, (\d+) passthrough\)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)")
RE_STREAM = re.compile(r"Streaming resumed \(FPGA: (\S+), VC interleaving: (\w+)\)")
RE_STATE = re.compile(r"FPGA state check: (\S+)")
# Without the colour module XRService runs the side cameras through the ISP, as NV12 on other
# capture pipes (vfe0 and vfe1), and the upper pair on vfe3 and vfe4.
RE_ISP = re.compile(r"ISP (enabled|disabled) for tracking cameras")
class LogState:
@@ -85,7 +91,7 @@ class LogState:
def _new_episode(self, t):
self.closed = False
self.episode = {"start": t, "fpga_before": "", "vcint": "", "interleave": None, "tasks": None,
"inits": {}, "stream": "", "failure": "", "evidence": []}
"inits": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
if self.closed_at:
self.episode["evidence"].append(self.closed_at)
@@ -146,6 +152,12 @@ class LogState:
ep["interleave"] = int(m.group(1))
ep["evidence"].append(short)
return
m = RE_ISP.search(text)
if m:
ep = self._ep(t)
ep["isp"] = m.group(1) == "enabled"
ep["evidence"].append(short)
return
m = RE_TASKS.search(text)
if m:
ep = self._ep(t)
@@ -184,6 +196,12 @@ class LogState:
got = tuple(self.nodes[i] for i in (2, 3) if i in self.nodes)
return got if len(got) == 2 else UPPER_NODES
def camera_map(self):
"""{calibration name: /dev/videoN's N} from the latest camera start's TrackingCameraInit
lines (the whole log's when that start has none yet)."""
inits = (self.episode or {}).get("inits") or self.nodes
return {NAMES[i]: node for i, node in sorted(inits.items()) if 0 <= i < len(NAMES)}
def tracking_nodes(self):
got = tuple(self.nodes[i] for i in range(TRACKING) if i in self.nodes)
return got if len(got) == TRACKING else SIDE_NODES + UPPER_NODES
@@ -353,7 +371,9 @@ def check(log=None, proc=True, ring=True, ring_path=None):
status, reason = "unknown", "no XRService log in %s" % LOG_DIR
out = {"log": path, "xrservice": None, "ring": None,
"episode": state.snapshot()["episode"] if state else {},
"failure": state.snapshot()["failure"] if state else ""}
"failure": state.snapshot()["failure"] if state else "",
"map": {name: {"node": n, "pipe": pipe_name(n)} for name, n in state.camera_map().items()} if state else {},
"ring_missing": []}
if proc:
if pid is None:
@@ -389,13 +409,30 @@ def check(log=None, proc=True, ring=True, ring_path=None):
evidence.append("ft-camd (pid %d, %s): %d mono cameras: %s" % (
r["writer_pid"], "running" if r["alive"] else "stale ring", len(r["mono"]), names))
if r["alive"] and len(r["mono"]) < TRACKING and status == "ok":
status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras (it started while "
"they were missing: restart it)" % (len(r["mono"]), TRACKING))
have = {c["node"] for c in r["mono"]}
want = state.tracking_nodes() if state else SIDE_NODES + UPPER_NODES
out["ring_missing"] = [n for n in want if n not in have]
status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras, missing %s" % (
len(r["mono"]), TRACKING, " ".join("video%d" % n for n in out["ring_missing"]) or "?"))
out.update(status=status, reason=reason, evidence=evidence,
summary="ok" if status == "ok" else "%s: %s" % (status, reason))
return out
def pipe_name(node):
"""The capture pipe behind /dev/videoN (msm_vfe3_video0 and so on), or ""."""
try:
with open("/sys/class/video4linux/video%d/name" % node) as f:
return f.read().strip()
except OSError:
return ""
def is_ring_short(result):
"""XRService runs all the tracking cameras, but ft-camd doesn't publish them all."""
return bool(result) and result.get("status") == "degraded" and bool(result.get("ring_missing"))
def is_vcint_failure(result):
return bool(result) and result.get("status") == "degraded" and result.get("reason") == VCINT_REASON
+25 -2
View File
@@ -379,9 +379,15 @@ static bool resolve_block(cam_t *c)
return true;
}
/*
* Through the ISP (no colour module), the near-black exposures come out all zeros,
* the same every time, so their dequeues change no buffer and get no votes. Such
* an index is left unmapped (slot -1): on_frame counts its frames as dark without
* reading them. Most of a camera's indices silent means it isn't streaming yet.
*/
static bool resolve_each(cam_t *c)
{
int depth = c->maxindex + 1;
int depth = c->maxindex + 1, silent = 0;
bool taken[MAX_SLOTS] = { false };
for (int i = 0; i < depth; i++) {
@@ -396,6 +402,12 @@ static bool resolve_each(cam_t *c)
}
}
if (c->nobs[i] >= 3 && v1 <= 0.2 * c->nobs[i]) {
c->slot_of[i] = -1;
silent++;
continue;
}
if (c->nobs[i] < 3 || best < 0 || taken[best] || v1 < 0.8 * c->nobs[i] || v2 > 0.3 * c->nobs[i])
return false;
@@ -403,9 +415,14 @@ static bool resolve_each(cam_t *c)
c->slot_of[i] = best;
}
if (silent * 2 > depth)
return false;
printf("%s: queue depth %d, buffers mapped one by one:", c->slug, depth);
for (int i = 0; i < depth; i++)
printf(" %d", c->slot_of[i]);
c->slot_of[i] < 0 ? printf(" -") : printf(" %d", c->slot_of[i]);
if (silent)
printf(" (- : %d indices whose frames are all zeros, skipped as dark)", silent);
printf("\n");
return true;
}
@@ -677,6 +694,12 @@ static void on_frame(cam_t *c, int64_t index, uint32_t seq, uint64_t ts, uint64_
int slot = c->slot_of[index];
if (slot < 0) { /* an index whose frames are all zeros (resolve_each): a dark one */
c->dark++;
c->rc->dropped++;
return;
}
/* color runs at 60 fps: skip frames early enough that the asked rate holds, before any sync */
if (c->color && color_fps > 0 && evtime - c->last_pub_ns < (uint64_t)(1e9 / color_fps) - 3000000) {
c->paced++;
+24 -7
View File
@@ -544,22 +544,26 @@ static void probe_cameras(xr_state_t *st)
/*
* qcom-camss can report bytesperline as the visible width while the VFE
* writes a larger aligned pitch. sizeimage is right, so derive the pitch.
* For NV12, plane 0 normally holds the chroma rows after the luma (the side
* cameras through the ISP: 1056 wide, 1152 bytes a row); if it's too small for
* that, it holds the luma alone.
*/
unsigned xr_camera_stride(const xr_camera_t *c)
{
if (!c->height || !c->planesize[0])
return c->bytesperline ? c->bytesperline : c->width;
double bpp = 1.0;
if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21)
bpp = 1.5;
unsigned s = (unsigned)((double)c->planesize[0] / ((double)c->height * bpp));
bool yuv = c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21;
unsigned s = (unsigned)((double)c->planesize[0] / ((double)c->height * (yuv ? 1.5 : 1.0)));
if (s >= c->width && s <= c->width * 4)
return s;
s = (unsigned)(c->planesize[0] / c->height);
if (yuv && s >= c->width && s <= c->width * 4)
return s;
return c->bytesperline ? c->bytesperline : c->width;
}
@@ -591,7 +595,20 @@ void xr_camera_layout(const xr_camera_t *c, xr_layout_t *l)
l->pitch = xr_camera_stride(c);
l->width = c->width < l->pitch ? c->width : l->pitch;
if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21) {
/*
* Without the colour module, XRService runs the side cameras through the
* ISP ("ISP enabled for tracking cameras (main VFE available)" in its log),
* and they come out NV12. They're mono sensors, so the luma is the image.
*/
bool yuv = c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21;
if (yuv && c->role && !strcmp(c->role, "tracking")) {
l->fmt = XR_FMT_GREY8;
l->rows = c->height;
return;
}
if (yuv) {
l->fmt = XR_FMT_NV12;
l->rows = c->height + c->height / 2;
} else {
+1 -1
View File
@@ -29,7 +29,7 @@ Every part runs in the dev container, as ft-hands and Input Settings do. The hos
- **A tracking ft-hands** gives feedback through the hands file: which hands are seen, the palm's distance, the index tip. If none is running, the session starts `ft-hands --no-gestures --status 0` (unit `frametop-handrec-hands.service`). An ft-hands already running is used as it is.
- **A recording ft-hands** runs once per recording part: `ft-hands --record-only --record DIR --record-for SECONDS --record-hz 10 --status 0 --sides auto|0|1` (below, "Side cameras"). It runs as a plain child process of the session, ended with SIGTERM when the part ends. SIGTERM ends ft-hands' loop, and `Recorder` writes out its queue when it's destroyed. In step mode (below) a part is one step's countdown and hold, so a take has one part per step (about 40 in the hand poses); in auto mode a take is one part, plus one more after each pause. `--record-for` is only a safety net.
- Why a process per part rather than one kept alive and paused: measured in the dev container with `ft-ringplay`'s ring (2026-10-02), `ft-hands --record-only` writes its first set 16-27 ms after it starts and ends 4-6 ms after SIGTERM, so a new part costs nothing the 3 s countdown doesn't cover. Every reader already takes parts in order (`takes.py`, `validate.py` through the export's single stream, the labeller's `fhl_io.py`, numbering `sets-10.bin` after `sets-9.bin`), ft-hands needs no new control, and nothing is written while a step waits. Before the hold starts the session checks that the part has written a set (`Recorder.has_data`, up to 3 s more), so the hold is recorded from its first frame.
- **Side cameras.** ft-camd can name the two side cameras the wrong way round (hands/README.md, "Which camera is which"). The tracking ft-hands decides from the hands within about 2 s of them being in view (`HANDS_SWAP_SIDES=auto`, hands/track/sides.h) and publishes that in `/run/user/UID/frametop-hands/sides.json`. The session reads it (`sides.py`, `read_live`) and stores it in session.json `"sides"`. Each later part is recorded named right (`--sides 1` or `0`). Parts recorded before the decision use ft-camd's names (`--sides auto`; a record-only ft-hands can't tell), and readers rename them (`sides.py`). Each part's `names_swapped` goes into take.json `"parts"`. Without a tracking ft-hands nothing decides: `"swapped": null`, the names stay as recorded, and the maintainer's check (`hub_review check`, check_sides on a few sets per take) tells. `takes.py sides SESSION --set swapped|named` records a decision by hand.
- **Side cameras.** ft-camd can name the two side cameras the wrong way round (hands/README.md, "Which camera is which"). The tracking ft-hands decides from the hands within about 2 s of them being in view (`HANDS_SWAP_SIDES=auto`, hands/track/sides.h) and publishes that in `/run/user/UID/frametop-hands/sides.json`. With `HANDS_SWAP_SIDES` forced to `0` or `1`, the hands still decide what's published once they disagree (state `"forced, disagrees"`; `read_live` also corrects an ft-hands built before that). The session reads it (`sides.py`, `read_live`) and stores it in session.json `"sides"`. Each later part is recorded named right (`--sides 1` or `0`). Parts recorded before the decision use ft-camd's names (`--sides auto`; a record-only ft-hands can't tell), and readers rename them (`sides.py`). Each part's `names_swapped` goes into take.json `"parts"`. Without a tracking ft-hands nothing decides: `"swapped": null`, the names stay as recorded, and the maintainer's check (`hub_review check`, check_sides on a few sets per take) tells. `takes.py sides SESSION --set swapped|named` records a decision by hand.
- **ft-handpanel** runs as a child process with `--watch-stdin`. It shows the panel and logs poses during each take.
- **The headset button's reader** is a thread of the session (`ButtonReader`, below), not a process.
+5 -4
View File
@@ -457,11 +457,12 @@ class Backend(QObject):
"""Start stays off: the check found the cameras degraded (unless --ignore-cameras)."""
return not self._ignore_cameras and self._camera.get("status") == "degraded"
def _check_now(self):
def _check_now(self, repair=False):
"""repair (off this thread only: it may restart ft-camd, up to 15 s): see session.camera_check."""
mod = self._runner()
if not mod or self._session_options.get("dry_run"):
return {"status": "unknown", "summary": "not checked (dry run)", "reason": "dry run", "evidence": []}
return mod.camera_check()
return mod.camera_check(repair=repair)
@Slot()
def checkCameras(self):
@@ -470,7 +471,7 @@ class Backend(QObject):
return
self._camera_busy = True
self.cameraChanged.emit()
self._thread(lambda: self._cameraArrived.emit(self._check_now()))
self._thread(lambda: self._cameraArrived.emit(self._check_now(repair=True)))
def _on_camera(self, result):
self._camera = result
@@ -499,7 +500,7 @@ class Backend(QObject):
end = time.monotonic() + RECHECK_FOR_S
time.sleep(RECHECK_S * 2)
while time.monotonic() < end:
result = self._check_now()
result = self._check_now(repair=True)
# Done once a new XRService (a new log) has opened its cameras, ok or not.
if result.get("status") in ("ok", "degraded") and result.get("log") != before:
break
+1
View File
@@ -23,6 +23,7 @@ case ${1:-install} in
"$root/hands/rec/build.sh"
echo "== 4/4 ft-camd's capabilities (asks for your password) and the menu entry"
"$root/hands/run.sh" caps
"$root/scripts/conf-migrate.sh" # HANDS_SWAP_SIDES=0, the old default, becomes auto
fill_template "$root/hands/rec/ft-handrec.desktop" |
on_frame "mkdir -p ~/.local/share/applications && cat > $entry"
echo
+37 -4
View File
@@ -1153,11 +1153,33 @@ def _steamvr_version():
return ""
def camera_check():
"""camcheck.check() (the XRService log, its open cameras where readable, ft-camd's ring);
never raises: a failure is "unknown"."""
_camd_restarted = set() # ft-camd pids camera_check(repair=True) has restarted: once each
def _unit_pid(unit):
r = subprocess.run(host_command("systemctl", "--user", "show", "-p", "MainPID", "--value", unit),
capture_output=True, text=True, timeout=30)
try:
return camcheck.check()
return int(r.stdout.strip() or 0)
except ValueError:
return 0
def camera_check(repair=False):
"""camcheck.check() (the XRService log, its open cameras where readable, ft-camd's ring);
never raises: a failure is "unknown". repair (the window, never during a session): when
XRService runs every tracking camera but the recorder's own ft-camd doesn't publish them all
(it started while some were missing), restart it, once per ft-camd, and check again."""
try:
r = camcheck.check()
pid = (r.get("ring") or {}).get("writer_pid")
if repair and camcheck.is_ring_short(r) and pid not in _camd_restarted and _unit_pid(CAMD_UNIT) == pid:
_camd_restarted.add(pid)
stop_unit(CAMD_UNIT)
start_camd()
r = camcheck.check()
r["evidence"].append("restarted ft-camd (pid %d) because it published only some of the cameras" % pid)
return r
except Exception as e:
return {"status": "unknown", "summary": "unknown: the camera check failed (%s)" % e,
"reason": str(e), "evidence": []}
@@ -1169,6 +1191,10 @@ def camera_text(result):
return ""
if camcheck.is_vcint_failure(result):
return camcheck.USER_TEXT
if camcheck.is_ring_short(result):
return ("The headset's tracking cameras are all running, but the recorder can't read some of them (%s). "
"Close the Hand Recorder, run ~/frametop/hands/rec/install.sh again, and open it again. If that "
"doesn't help, ask in the Frametop Discord." % result.get("reason", ""))
return ("Not all of the headset's tracking cameras are running (%s). Restart SteamVR, or restart the "
"headset if that doesn't fix it." % result.get("reason", ""))
@@ -1451,6 +1477,10 @@ class Session:
cam = self._status.get("camera")
if cam:
self._session_json["camera"] = {"status": cam.get("status"), "reason": cam.get("reason", "")}
# which device each calibrated camera was (XRService's numbering) and whether XRService
# ran the side cameras through the ISP (no colour module): to check the names later
self._session_json["device"]["camera_map"] = cam.get("map") or {}
self._session_json["device"]["isp"] = (cam.get("episode") or {}).get("isp")
if self.dry_run:
self._session_json["dry_run"] = True
if self.speed != 1:
@@ -1545,6 +1575,9 @@ class Session:
self._save_session()
self._log("side cameras: %s (%s, %s)" % ("SWAPPED" if swapped else "as named", new["decided_by"],
json.dumps(new["evidence"])))
if new["state"] == "forced, disagrees":
self._log("side cameras: HANDS_SWAP_SIDES in ~/.config/frametop.conf forces names the hands say are "
"backwards; the recording goes by the hands. Set HANDS_SWAP_SIDES=auto.")
def _sides_swapped(self):
"""session.json's decision: True, False, or None (not known yet)."""
+5
View File
@@ -125,4 +125,9 @@ def read_live(path=None, ring_path=None, now_ns=None):
return None
except OSError:
return None
if (s.get("state") == "forced, disagrees" and s.get("swapped") is not None
and bool(s["swapped"]) == bool(s.get("names_swapped"))):
# An ft-hands built before 2026-10-06 kept a forced HANDS_SWAP_SIDES as the truth even
# when the hands disagreed. The hands are right: the truth is the other way round.
s = dict(s, swapped=not s["swapped"], decided_by="auto")
return s
+39
View File
@@ -91,6 +91,29 @@ class RulesTest(unittest.TestCase):
finally:
shutil.rmtree(d)
def test_read_live_forced(self):
"""HANDS_SWAP_SIDES=0 forced and the hands disagree: the hands are the truth, whether
ft-hands published them (built after 2026-10-06) or kept the forced value (before)."""
d = tempfile.mkdtemp()
try:
path = os.path.join(d, "sides.json")
def live(**kw):
s = dict(pid=1, ring_ino=0, mode="0", names_swapped=False,
updated_ns=time.clock_gettime_ns(time.CLOCK_MONOTONIC), **kw)
with open(path, "w") as f:
json.dump(s, f)
return sides.read_live(path)
self.assertEqual(live(state="forced", swapped=False, decided_by="config")["swapped"], False)
self.assertEqual(live(state="forced, agrees", swapped=False, decided_by="config")["swapped"], False)
old = live(state="forced, disagrees", swapped=False, decided_by="config")
self.assertEqual((old["swapped"], old["decided_by"]), (True, "auto"))
new = live(state="forced, disagrees", swapped=True, decided_by="auto")
self.assertEqual((new["swapped"], new["decided_by"]), (True, "auto"))
finally:
shutil.rmtree(d)
class TakesTest(unittest.TestCase):
"""A swapped session: one take's parts recorded before the decision (ft-camd's names) and
@@ -226,6 +249,22 @@ class SessionSidesTest(unittest.TestCase):
self.assertTrue(s._session_json["sides"]["reversed_from"]["swapped"])
self.assertFalse(takes.read_json(os.path.join(self.tmp, "session.json"))["sides"]["swapped"])
def test_read_sides_forced(self):
"""PR #4 on the dataset: HANDS_SWAP_SIDES=0 from the old example config, and the hands
disagree. The session takes the hands' answer, not the forced one."""
s = session.Session(os.path.join(self.tmp, "base"), {}, {}, "room", self.script, dry_run=True,
hands_dir=self.tmp)
s.session_dir = self.tmp
s._session_json = {"sides": {"swapped": None}}
self.live(mode="0", state="forced", swapped=False, decided_by="config", names_swapped=False, evidence=None)
s._read_sides(force=True)
self.assertIs(s._sides_swapped(), False)
self.live(mode="0", state="forced, disagrees", swapped=False, decided_by="config", names_swapped=False)
s._read_sides(force=True)
self.assertIs(s._sides_swapped(), True)
self.assertEqual(s._session_json["sides"]["decided_by"], "auto")
self.assertEqual(s._session_json["sides"]["reversed_from"]["decided_by"], "config")
def test_recorder_parts(self):
calls = []
+48 -2
View File
@@ -66,6 +66,22 @@ RESUME_OK = [L("12:30:00", "[SystemdInhibitor] Received systemd resume notificat
for i, n in enumerate((9, 13, 6, 7))] + \
[L("12:30:02", "[DeckardCaptureSource] Streaming resumed (FPGA: VCINT, VC interleaving: enabled)")]
EXIT = [L("13:00:00", "XRService - main thread exiting"), L("13:00:00", "Exiting XRService")]
# The colour module unplugged while SteamVR runs (2026-10-05 12:42): XRService reopens the
# cameras with the side pair through the ISP on vfe0 and vfe1 (NV12); VCINT stays loaded, so the
# upper pair stays on vfe2.
UNPLUG = [L("12:42:25", "Received passthrough camera connection event (connected=0)"),
L("12:42:25", "[DeckardCaptureSource] Closing tracking camera interfaces camerasToUse: 1111"),
L("12:42:26", "FPGA state check: VCINT (register value: 0x00021211)"),
L("12:42:26", "Upper cameras FPGA interleaving support: 1 (Driver features available = 1 | VCINT loaded = 1)"),
L("12:42:26", "[buildMediaCtlSetupTasks] ISP enabled for tracking cameras (main VFE available)"),
L("12:42:26", "[buildMediaCtlSetupTasks] Created 4 tasks (4 tracking, 0 passthrough)")] + \
[L("12:42:26", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
for i, n in enumerate((0, 3, 6, 7))]
# Started without the module (FrameEyeCameraFeed's layout): the upper pair on vfe3 and vfe4.
NO_MODULE = [L("10:00:02", "[buildMediaCtlSetupTasks] ISP enabled for tracking cameras (main VFE available)"),
L("10:00:02", "[buildMediaCtlSetupTasks] Created 4 tasks (4 tracking, 0 passthrough)")] + \
[L("10:00:03", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
for i, n in enumerate((0, 3, 9, 13))]
def state(lines):
@@ -128,6 +144,23 @@ class SyntheticLogs(unittest.TestCase):
self.assertEqual(status, "degraded")
self.assertEqual(reason, "only 2 of 4 tracking cameras running")
def test_camera_map_with_module(self):
st = state(START + GOOD_OPEN)
self.assertEqual(st.camera_map(), {"slam_left": 9, "slam_right": 13, "upper_left": 6, "upper_right": 7})
def test_module_unplugged(self):
st = state(START + GOOD_OPEN + UNPLUG)
self.assertEqual(st.verdict()[0], "ok")
self.assertIs(st.episode["isp"], True)
self.assertEqual(st.camera_map(), {"slam_left": 0, "slam_right": 3, "upper_left": 6, "upper_right": 7})
self.assertEqual(st.tracking_nodes(), (0, 3, 6, 7))
def test_started_without_module(self):
st = state(START + NO_MODULE)
self.assertEqual(st.verdict()[0], "ok")
self.assertEqual(st.camera_map(), {"slam_left": 0, "slam_right": 3, "upper_left": 9, "upper_right": 13})
self.assertEqual(st.upper_nodes(), (9, 13))
def test_exited_and_empty(self):
self.assertEqual(state(START + GOOD_OPEN + EXIT).verdict()[0], "unknown")
self.assertEqual(state([]).verdict()[0], "unknown")
@@ -223,9 +256,9 @@ class RealLog(unittest.TestCase):
self.assertEqual(out.getvalue().split("\n")[0], "ok")
def make_ring(path, mono_names, alive=True):
def make_ring(path, mono_names, alive=True, nodes=None):
"""A ring header as ft-camd writes it (camd/fhring.h), no frames."""
cams = [(b"og01a1b", n.encode(), 9 + i) for i, n in enumerate(mono_names)]
cams = [(b"og01a1b", n.encode(), nodes[i] if nodes else 9 + i) for i, n in enumerate(mono_names)]
hb = time.clock_gettime_ns(time.CLOCK_MONOTONIC) if alive else 1
data = bytearray(camcheck.RING_HDR.pack(b"FHRING01", 1, 0, len(cams), 0, 0, 4242, 0))
struct.pack_into("<Q", data, 40, hb)
@@ -252,6 +285,19 @@ class Ring(unittest.TestCase):
self.assertEqual(r["status"], "degraded")
self.assertIn("ft-camd publishes only 2 of 4", r["reason"])
def test_ring_missing_the_side_cameras(self):
# an ft-camd from before NV12 support, without the colour module: only the upper pair
with open(self.log, "w") as f:
f.write("\n".join(START + NO_MODULE) + "\n")
make_ring(self.ring, ["og0ve10_5-003e_video9", "og0ve10_5-0060_video13"], nodes=[9, 13])
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
self.assertEqual(r["status"], "degraded")
self.assertEqual(r["ring_missing"], [0, 3])
self.assertIn("missing video0 video3", r["reason"])
self.assertTrue(camcheck.is_ring_short(r))
self.assertFalse(camcheck.is_vcint_failure(r))
self.assertEqual(r["map"]["slam_left"]["node"], 0)
def test_four_cameras_in_ring(self):
make_ring(self.ring, ["a_video9", "b_video13", "c_video6", "d_video7", "a_video9_dk"])
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
+23 -2
View File
@@ -34,8 +34,28 @@ from tools.show_set import index, read_set # noqa: E402
from tools import calib # noqa: E402
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # as ft-hands maps them
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # with the colour module
'msm_vfe2_video0': 'upper_left', 'msm_vfe2_video1': 'upper_right'}
def ring_names():
"""{/dev/videoN's N: calibration name} as ft-hands names them: by XRService's log
(camcheck.py), else by capture pipe (PIPES)."""
import camcheck
try:
by_log = {node: name for name, node in camcheck.read_log(camcheck.newest_log()).camera_map().items()}
except (OSError, ValueError):
by_log = {}
if by_log:
return by_log
out = {}
for path in os.listdir('/sys/class/video4linux'):
if path.startswith('video'):
with open('/sys/class/video4linux/%s/name' % path) as f:
name = PIPES.get(f.read().strip())
if name:
out[int(path[5:])] = name
return out
PAIRS = {'side': ('slam_left', 'slam_right'), 'upper': ('upper_left', 'upper_right')}
@@ -100,8 +120,9 @@ def live_pairs(count, names=PAIRS['side']):
if not ring.alive():
sys.exit('ft-camd isn\'t running (no heartbeat)')
cams = {}
names_of = ring_names()
for c in ring.cams:
name = PIPES.get(open('/sys/class/video4linux/video%d/name' % c.node).read().strip())
name = names_of.get(c.node)
if name and not c.name.endswith('-dark'):
cams[name] = c
for k in range(count):
+70 -14
View File
@@ -27,7 +27,8 @@
// default) tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// cameras if they're backwards (the tracked views move with their images), and checks once
// more. 0 and 1 force the naming (1: exchanged; --swap-sides is --sides 1); it still checks,
// and warns if the hands disagree. The decision is published in /run/user/UID/frametop-hands/sides.json (see
// and if the hands disagree it warns and publishes what the hands say as the truth ("swapped"),
// so recordings are labelled right while tracking keeps the forced names. The decision is published in /run/user/UID/frametop-hands/sides.json (see
// write_sides below) and, for recordings, in DIR/sides.json. --record-only can't tell (it tracks
// nothing): under auto it records the ring's names as they are.
//
@@ -63,7 +64,34 @@ namespace {
volatile std::sig_atomic_t g_stop = 0, g_record = 0;
// which calibrated camera each capture pipe carries (XRService's fixed routing)
// Which calibrated camera each video device carries. XRService numbers its tracking cameras
// (index 0 to 3: slam_left, slam_right, upper_left, upper_right) and logs the device each one
// opened ("TrackingCameraInit: index: 0. video device: /dev/video9"). The devices depend on
// the colour module: with it, the side cameras are on vfe3 and vfe4 and the upper pair on
// vfe2; without it, XRService runs the side cameras through the ISP on vfe0 and vfe1, and the
// upper pair on vfe3 and vfe4. So the running XRService's log decides; when it can't be read,
// the capture pipes as they are with the module. {} if the log has no cameras.
std::map<int, std::string> cameras_from_xrservice_log() {
static const char *const names[] = {"slam_left", "slam_right", "upper_left", "upper_right"};
const char *home = std::getenv("HOME");
std::ifstream in(std::string(home ? home : "") + "/.local/share/Steam/logs/xrservice.txt");
const std::string key = "TrackingCameraInit: index: ";
std::map<int, int> node_of; // index -> N of /dev/videoN, from the latest camera start
std::string line;
while (std::getline(in, line)) {
if (line.find("XRService logging to") != std::string::npos) node_of.clear();
const auto at = line.find(key);
int index = -1, node = -1;
if (at != std::string::npos &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d", &index, &node) == 2 &&
index >= 0 && index < 4)
node_of[index] = node;
}
std::map<int, std::string> out;
for (auto &[index, node] : node_of) out[node] = names[index];
return out;
}
const char *camera_for_pipe(int node) {
char path[64], name[64] = "";
std::snprintf(path, sizeof path, "/sys/class/video4linux/video%d/name", node);
@@ -302,6 +330,7 @@ int main(int argc, char **argv) {
double decided_after_s = -1;
if (sides_mode != "auto") truth = names_swapped, decided_by = sides_from, sides_state = "forced";
std::string rec_dir;
std::string cams_json; // which device each calibrated camera is (set below), for the sides files
std::vector<std::pair<size_t, bool>> rec_names; // from which recorded set on, names_swapped was what
// DIR/sides.json beside a recording's sets.bin: how its side cameras are named. A set's
// names are right when its names_swapped equals swapped (null: not known when recorded).
@@ -312,7 +341,8 @@ int main(int argc, char **argv) {
write_file(rec_dir + "/sides.json",
"{\"swapped\": " + json_bool(truth) + ", \"decided_by\": " + json_str(truth ? decided_by : "") +
", \"names_swapped\": [" + runs + "]" +
(decision_evidence.empty() ? "" : ", \"evidence\": " + decision_evidence) + "}\n");
(decision_evidence.empty() ? "" : ", \"evidence\": " + decision_evidence) +
(cams_json.empty() ? "" : ", \"cameras\": " + cams_json) + "}\n");
};
auto start_recording = [&](const std::string &dir, std::string &e) {
rec = std::make_unique<Recorder>();
@@ -340,19 +370,42 @@ int main(int argc, char **argv) {
// (--with-color). Recorded names hold 15 characters, so "upper_right_dark" wouldn't fit.
std::map<std::string, int> dark;
std::map<std::string, Camera> used;
// ft-camd's cameras by XRService's numbering, else by capture pipe (see camera_for_pipe);
// ft-ringplay's (no device) by the name it gives
const std::map<int, std::string> by_log = cameras_from_xrservice_log();
const char *named_by = by_log.empty() ? "capture pipe" : "XRService's log";
for (int i = 0; i < ring.cameras(); ++i) {
if (ring.camera(i).flags & FH_CAM_COLOR) {
const std::string name = "color_video" + std::to_string(ring.camera(i).node);
const fh_ring_cam_t &rc = ring.camera(i);
if (rc.flags & FH_CAM_COLOR) {
const std::string name = "color_video" + std::to_string(rc.node);
dark[name] = i, color[name] = i;
continue;
}
// ft-camd's cameras by capture pipe; ft-ringplay's (no device) by the name it gives
const char *name = camera_for_pipe(ring.camera(i).node);
if (!name && ring.camera(i).node < 0) name = ring.camera(i).name;
if (!name || !calib.count(name)) continue;
if (ring.camera(i).flags & FH_CAM_DARK) dark[std::string(name) + "_dk"] = i;
else index[name] = i, used[name] = calib[name];
const auto it = by_log.find(rc.node);
const char *name = rc.node < 0 ? rc.name
: !by_log.empty() ? (it == by_log.end() ? nullptr : it->second.c_str())
: camera_for_pipe(rc.node);
if (!name || !calib.count(name)) {
if (!(rc.flags & FH_CAM_DARK)) std::fprintf(stderr, "video%d (%s): not one of the calibrated cameras, left out\n", rc.node, rc.name);
continue;
}
if (int(rc.width) != calib[name].width || int(rc.height) != calib[name].height) {
std::fprintf(stderr, "video%d (%s) is %ux%u, but %s is calibrated at %dx%d: left out\n", rc.node, rc.name,
rc.width, rc.height, name, calib[name].width, calib[name].height);
continue;
}
if (rc.flags & FH_CAM_DARK) {
dark[std::string(name) + "_dk"] = i;
} else if (index.count(name)) {
std::fprintf(stderr, "video%d (%s) would be %s too (video%d is): left out\n", rc.node, rc.name, name,
ring.camera(index[name]).node);
} else {
index[name] = i, used[name] = calib[name];
cams_json += std::string(cams_json.empty() ? "" : ", ") + json_str(name) + ": {\"node\": " +
std::to_string(rc.node) + ", \"ring\": " + json_str(rc.name) + "}";
}
}
cams_json = "{\"named_by\": " + json_str(named_by) + ", \"cameras\": {" + cams_json + "}}";
// ft-camd tells the side cameras' buffers apart by XRService's allocation order, which
// some XRService restarts reverse (see the top).
const bool have_sides = index.count("slam_left") && index.count("slam_right");
@@ -397,7 +450,7 @@ int main(int argc, char **argv) {
}
const bool switching = automatic && !color.empty();
Cams mode = switching || color.empty() ? Cams::Mono : fixed;
std::printf("cameras:");
std::printf("cameras (by %s):", named_by);
for (auto &[name, i] : index) std::printf(" %s=video%d", name.c_str(), ring.camera(i).node);
for (auto &[name, i] : color) std::printf(" %s", name.c_str());
std::printf(" tracking with %s%s models: %s%s, %d threads on CPUs", cams_name(mode),
@@ -446,6 +499,7 @@ int main(int argc, char **argv) {
", \"decided_after_s\": " + std::to_string(decided_after_s) +
", \"evidence\": " + (decision_evidence.empty() ? "null" : decision_evidence) +
", \"checking\": " + (checking ? side_check.json() : "null") +
", \"cameras\": " + (cams_json.empty() ? "null" : cams_json) +
", \"updated_ns\": " + std::to_string(mono_ns()) + "}\n");
};
write_sides();
@@ -454,14 +508,16 @@ int main(int argc, char **argv) {
const bool backwards = v == SideCheck::Swapped; // relative to the names as they are now
const double after = (now - start) / 1e9;
const std::string ev = side_check.json(), text = side_check.summary();
if (sides_mode != "auto") { // forced: only say so
if (sides_mode != "auto") { // forced: the names stay; if the hands disagree, they're the truth
const std::string what = (sides_from == "option" ? "--sides " : "HANDS_SWAP_SIDES=") + sides_mode;
if (backwards)
std::printf("side cameras: %s looks WRONG: the hands say the side cameras are the other way round (%s). "
"Use auto.\n", what.c_str(), text.c_str());
"Tracking keeps the forced names; recordings are labelled by the hands. Use auto.\n",
what.c_str(), text.c_str());
else
std::printf("side cameras: %s agrees with the hands (%s)\n", what.c_str(), text.c_str());
sides_state = backwards ? "forced, disagrees" : "forced, agrees";
if (backwards) truth = !names_swapped, decided_by = "auto", decided_after_s = after;
decision_evidence = ev;
checking = false;
} else if (side_round == 0 || backwards) {
+139 -3
View File
@@ -15,6 +15,9 @@ to the input relay over its control socket (@frametop_relay):
- Pointer: speed, dot size, distance and the rest, applied live.
- Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or
one by one. The helper lists them (@ft_pointer_helper "overlays").
- Game optimization: pausing Frametop while a VR game runs, so it leaves the CPU and GPU to the game
(input/game_pause.py, in the relay: "pause on|off|?"): pause now, pause by itself during VR
games, the controller gesture that pauses and resumes, what happens to the desktop, a sound.
- Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
(gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker (SteamVR's or
our own) and eye bias (GAZE_TRACKER, GAZE_EYE in frametop.conf).
@@ -73,6 +76,9 @@ ACTION_LABELS = {
"sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout",
"screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
"float_toggle": "Float window in VR / put it back", "dock_all": "Put all floating windows back",
"pause_toggle": "Pause/resume Frametop (for VR games)",
"spin_next": "Spin the panels: next one on the right to the front",
"spin_prev": "Spin the panels: next one on the left to the front",
"key": "Pass through as key",
"none": "Do nothing",
}
@@ -95,6 +101,13 @@ CONTROLLER_BUTTONS = {
"right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
"right/thumbstick": "Right stick click",
}
# Pausing for VR games (input/game_pause.py): the gesture a rules file without "pause_gesture" gets,
# both thumbsticks clicked together twice, and what can happen to the desktop.
DEFAULT_PAUSE_GESTURE = {"buttons": ["left/thumbstick", "right/thumbstick"], "presses": 2}
PAUSE_DESKTOP_MODES = {"hide": "Hide it and slow it down (windows stay open)",
"close": "Close it (windows close; it starts again when you resume)"}
PAUSE_REASONS = {"game": "a VR game started", "gesture": "controller gesture", "command": "ft-pause",
"settings": "from here"}
# Gaze mode is a mouse and keyboard feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS]
@@ -152,9 +165,10 @@ GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, releas
MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window.
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window,
# Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
def key_bindings(rules):
@@ -293,6 +307,7 @@ class Backend(QObject):
gazeChanged = Signal()
driverChanged = Signal()
panelsChanged = Signal()
pauseChanged = Signal()
activity = Signal(str) # device id
captured = Signal(int, str) # code, name
capturedController = Signal(str, str) # button, label
@@ -320,6 +335,8 @@ class Backend(QObject):
self._check_asked = 0.0 # when quickcal or calibrate went to the gaze service (its errors)
self._driver_block = "" # set by _check_driver
self._panels = None # SteamVR's overlays, from the helper; None until it answers
self._pause = {} # the relay's pause state ("pause ?"), {} when it doesn't answer
self._pause_at = 0.0
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
self.sock.bind("") # autobind an abstract address the relay can reply to
self.sock.setblocking(False)
@@ -352,6 +369,7 @@ class Backend(QObject):
def _refresh(self):
self._send("devices")
self._send("pause ?")
self._send("vrstatus", HELPER)
self._send("gaze ?", HELPER)
if not self._send("status", GAZED) and self._gaze:
@@ -365,6 +383,9 @@ class Backend(QObject):
if self._gaze_mode is not None and now - self._gaze_at > 5:
self._gaze_mode = None
self.gazeChanged.emit()
if self._pause and now - self._pause_at > 5:
self._pause = {}
self.pauseChanged.emit()
def _check_driver(self):
block = driver_block()
@@ -403,13 +424,20 @@ class Backend(QObject):
if t == "devices":
self._nodes = msg.get("nodes", [])
self._pointer_mode = bool(msg.get("pointer_mode"))
self._relay_ok = True
was_ok, self._relay_ok = self._relay_ok, True
if not was_ok:
self.pauseChanged.emit() # (pauseStatus' "old")
self.devicesChanged.emit()
elif t == "overlays":
panels = [o for o in msg.get("list", []) if isinstance(o, dict) and isinstance(o.get("key"), str)]
if panels != self._panels:
self._panels = panels
self.panelsChanged.emit()
elif t == "pause":
self._pause_at = time.monotonic()
if msg != self._pause:
self._pause = msg
self.pauseChanged.emit()
elif t == "vrstatus":
self._vr = msg
self._vr_at = time.monotonic()
@@ -817,6 +845,108 @@ class Backend(QObject):
else:
self.message.emit("The pointer helper isn't running (frametop-pointer.service)", True)
# --- pausing for VR games ---
@Property("QVariantMap", notify=pauseChanged)
def pauseStatus(self):
"""The relay's pause state, and a line saying it ("relay": False when it doesn't answer)."""
st = self._pause
if not st:
# "old": the relay answers, but not about pausing (it predates it).
return {"relay": False, "old": self._relay_ok, "paused": False,
"summary": "Unknown: the input relay " + ("can't pause" if self._relay_ok else "isn't answering")}
if st.get("paused"):
reason = PAUSE_REASONS.get(st.get("reason"), st.get("reason") or "")
summary = f"Paused since {time.strftime('%H:%M', time.localtime(st.get('since') or time.time()))}"
summary += f" ({reason})" if reason else ""
if st.get("ends_with_game"):
summary += ", until the game ends"
else:
summary = "Running" + (", with a VR game" if st.get("game") else "")
if st.get("busy"):
summary += "; working on it…"
return {"relay": True, "paused": bool(st.get("paused")), "summary": summary,
"gestureReader": bool(st.get("gesture_reader"))}
@Slot(bool)
def setPaused(self, on):
if self._send(f"pause {'on' if on else 'off'} settings"):
QTimer.singleShot(300, self._refresh)
else:
self.message.emit("The input relay isn't running (frametop-input-relay.service)", True)
def _set_rule(self, key, value, text):
rules = read_json(RULES_PATH)
rules[key] = value
self._save_rules(rules)
self.message.emit(text, False)
@Property(bool, notify=mappingsChanged)
def pauseAuto(self):
return read_json(RULES_PATH).get("pause_auto", True) is not False
@Slot(bool)
def setPauseAuto(self, on):
self._set_rule("pause_auto", bool(on), "VR games pause Frametop" if on else "VR games no longer pause Frametop")
@Property("QVariantList", constant=True)
def pauseDesktopModes(self):
return [{"value": k, "text": v} for k, v in PAUSE_DESKTOP_MODES.items()]
@Property(str, notify=mappingsChanged)
def pauseDesktop(self):
mode = read_json(RULES_PATH).get("pause_desktop")
return mode if mode in PAUSE_DESKTOP_MODES else "hide"
@Slot(str)
def setPauseDesktop(self, mode):
if mode in PAUSE_DESKTOP_MODES:
self._set_rule("pause_desktop", mode, "Paused, the desktop: " + PAUSE_DESKTOP_MODES[mode].split(" (")[0].lower())
@Property(bool, notify=mappingsChanged)
def pauseSound(self):
return read_json(RULES_PATH).get("pause_sound", True) is not False
@Slot(bool)
def setPauseSound(self, on):
self._set_rule("pause_sound", bool(on), "A sound on pause and resume" if on else "No sound on pause and resume")
@Property("QVariantList", constant=True)
def gestureButtons(self):
return [{"value": "", "text": "None"}] + [{"value": b, "text": label} for b, label in CONTROLLER_BUTTONS.items()]
@Property("QVariantMap", notify=mappingsChanged)
def pauseGesture(self):
"""{"first", "second" ("" for none), "presses"} (the relay's game_pause.gesture_of)."""
spec = read_json(RULES_PATH).get("pause_gesture", DEFAULT_PAUSE_GESTURE)
buttons = [b for b in (spec or {}).get("buttons", []) if b in CONTROLLER_BUTTONS][:2] \
if isinstance(spec, dict) and isinstance(spec.get("buttons"), list) else []
presses = 1 if len(buttons) == 2 and spec.get("presses") == 1 else 2
return {"first": buttons[0] if buttons else "", "second": buttons[1] if len(buttons) > 1 else "",
"presses": presses}
@Slot(str, str, int)
def setPauseGesture(self, first, second, presses):
"""No first button: no gesture. One button always takes two presses."""
buttons = list(dict.fromkeys(b for b in (first, second) if b in CONTROLLER_BUTTONS))
if not buttons:
self._set_rule("pause_gesture", None, "No controller gesture pauses Frametop")
return
presses = 1 if len(buttons) == 2 and presses == 1 else 2
self._set_rule("pause_gesture", {"buttons": buttons, "presses": presses},
"Pause gesture: " + self.gestureText(buttons, presses))
@staticmethod
def gestureText(buttons, presses):
names = " and ".join(CONTROLLER_BUTTONS[b] for b in buttons)
together = " together" if len(buttons) == 2 else ""
return f"{names}{together}, {'twice' if presses == 2 else 'once'}"
@Property(str, notify=mappingsChanged)
def pauseGestureText(self):
g = self.pauseGesture
buttons = [b for b in (g["first"], g["second"]) if b]
return self.gestureText(buttons, g["presses"]) if buttons else "None"
# --- gaze ---
def _gaze_status(self, status):
now = time.monotonic()
@@ -1036,6 +1166,12 @@ class Backend(QObject):
else:
self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
@Slot()
def keepGazeAwake(self):
"""The Gaze page is open: the gaze service doesn't idle meanwhile (its status stays live,
and a check opens without waiting for the tracker). The page renews this every 10 s."""
self._send("wake 30", GAZED)
@Slot()
def gazeQuickCheck(self):
"""The gaze service's one-dot check, in the panel fixed to the headset."""
+147 -3
View File
@@ -19,6 +19,7 @@ Kirigami.ApplicationWindow {
Kirigami.Action { text: "Devices"; icon.name: "input-mouse"; onTriggered: root.show(devicesPage) },
Kirigami.Action { text: "Buttons"; icon.name: "input-keyboard"; onTriggered: root.show(buttonsPage) },
Kirigami.Action { text: "Controllers"; icon.name: "input-gamepad"; onTriggered: root.show(controllersPage) },
Kirigami.Action { text: "Game optimization"; icon.name: "applications-games"; onTriggered: root.show(gamesPage) },
Kirigami.Action { text: "Keyboard"; icon.name: "input-keyboard-virtual"; onTriggered: root.show(keyboardPage) },
Kirigami.Action { text: "Pointer"; icon.name: "transform-move"; onTriggered: root.show(pointerPage) },
Kirigami.Action { text: "Ignored panels"; icon.name: "view-hidden"; onTriggered: root.show(ignorePage) },
@@ -56,8 +57,8 @@ Kirigami.ApplicationWindow {
pageStack.push(page)
}
// FT_INPUT_PAGE=buttons|controllers|keyboard|pointer|ignore|gaze|bluetooth opens the app on that page.
pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, keyboard: keyboardPage,
// FT_INPUT_PAGE=buttons|controllers|games (Game optimization)|keyboard|pointer|ignore|gaze|bluetooth opens the app on that page.
pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, games: gamesPage, keyboard: keyboardPage,
pointer: pointerPage, ignore: ignorePage, gaze: gazePage,
bluetooth: bluetoothPage })[startPage] || devicesPage
@@ -826,6 +827,140 @@ Kirigami.ApplicationWindow {
}
}
// ---------------------------------------------------------------- Games
Component {
id: gamesPage
Kirigami.ScrollablePage {
id: gmpage
title: "Game optimization"
property var st: backend.pauseStatus
// The gesture's boxes follow the saved gesture (which may tidy what was picked).
function syncGesture() {
const g = backend.pauseGesture
firstBox.currentIndex = firstBox.indexOfValue(g.first)
secondBox.currentIndex = secondBox.indexOfValue(g.second)
pressBox.currentIndex = pressBox.indexOfValue(g.presses)
}
function saveGesture() {
backend.setPauseGesture(firstBox.currentValue, secondBox.currentValue, pressBox.currentValue)
}
Component.onCompleted: syncGesture()
Connections {
target: backend
function onMappingsChanged() { gmpage.syncGesture() }
}
ColumnLayout {
spacing: Kirigami.Units.largeSpacing
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: !gmpage.st.relay
type: Kirigami.MessageType.Error
text: gmpage.st.old
? "The input relay that runs is older and can't pause. Restart it: "
+ "systemctl --user restart frametop-input-relay"
: "The input relay isn't answering (frametop-input-relay.service). It does the pausing."
}
Kirigami.FormLayout {
Layout.fillWidth: true
RowLayout {
Kirigami.FormData.label: "Frametop:"
Controls.Label { text: gmpage.st.summary }
Controls.Button {
text: gmpage.st.paused ? "Resume" : "Pause now"
icon.name: gmpage.st.paused ? "media-playback-start" : "media-playback-pause"
enabled: gmpage.st.relay
onClicked: backend.setPaused(!gmpage.st.paused)
}
}
Controls.Switch {
Kirigami.FormData.label: "VR games:"
text: "Pause while a VR game runs, and resume when it ends"
checked: backend.pauseAuto
onToggled: backend.setPauseAuto(checked)
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Controller gesture" }
Controls.ComboBox {
id: firstBox
Kirigami.FormData.label: "Button:"
model: backend.gestureButtons
textRole: "text"
valueRole: "value"
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
onActivated: gmpage.saveGesture()
}
Controls.ComboBox {
id: secondBox
Kirigami.FormData.label: "Together with:"
model: backend.gestureButtons
textRole: "text"
valueRole: "value"
enabled: firstBox.currentValue !== ""
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
onActivated: gmpage.saveGesture()
}
Controls.ComboBox {
id: pressBox
Kirigami.FormData.label: "Pressed:"
model: [{ value: 2, text: "Twice" }, { value: 1, text: "Once" }]
textRole: "text"
valueRole: "value"
// One button pressed once would go off far too easily in games.
enabled: firstBox.currentValue !== "" && secondBox.currentValue !== ""
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
onActivated: gmpage.saveGesture()
}
Controls.Label {
Layout.fillWidth: true
wrapMode: Text.Wrap
opacity: 0.7
text: (backend.pauseGesture.first === "" ? "No gesture: a mapped button, a key combination, or ft-pause still work."
: "Pauses and resumes Frametop, in games too. The game still gets the presses, so pick buttons it "
+ "doesn't use this way. Two buttons count as together when both go down within a third of a "
+ "second, and twice means within 0.7 s.")
+ (gmpage.st.relay && backend.pauseGesture.first !== "" && !gmpage.st.gestureReader
? " Not reading the controllers right now (SteamVR isn't running?)." : "")
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "While paused" }
Controls.ComboBox {
Kirigami.FormData.label: "The desktop:"
model: backend.pauseDesktopModes
textRole: "text"
valueRole: "value"
Layout.preferredWidth: Kirigami.Units.gridUnit * 20
Component.onCompleted: currentIndex = indexOfValue(backend.pauseDesktop)
onActivated: backend.setPauseDesktop(currentValue)
}
Controls.Switch {
Kirigami.FormData.label: "Sound:"
text: "Play a sound on pause and resume"
checked: backend.pauseSound
onToggled: backend.setPauseSound(checked)
}
}
Controls.Label {
Layout.fillWidth: true
wrapMode: Text.Wrap
opacity: 0.7
text: "Paused, Frametop leaves the headset's CPU and GPU to the game. The gaze service and our eye "
+ "tracker stop, and nothing reads SteamVR's eye tracking. Hand tracking and remote desktop stop "
+ "if they run. The desktop hides and slows down, or closes, as chosen above. The 3D mouse lets go, "
+ "so the mouse is a plain one for SteamVR. Mapped buttons and key combinations do nothing but "
+ "pausing, the Steam menu, and your own commands. Resuming brings back what pausing stopped. "
+ "Map \"Pause/resume Frametop\" to a button or key combination on the other pages, or run "
+ "input/ft-pause."
}
}
}
}
// ---------------------------------------------------------------- Gaze
Component {
id: gazePage
@@ -834,6 +969,14 @@ Kirigami.ApplicationWindow {
title: "Gaze"
header: DriverWarning {}
property var status: backend.gazeStatus
// The gaze service idles while the gaze isn't used; open, this page keeps it going.
Timer {
running: true
repeat: true
interval: 10000
triggeredOnStart: true
onTriggered: backend.keepGazeAwake()
}
actions: [
Kirigami.Action {
text: "Quick check"
@@ -1067,7 +1210,8 @@ Kirigami.ApplicationWindow {
Controls.Label {
Kirigami.FormData.label: "Service:"
text: backend.gazeServiceRunning
? (gpage.status.ft_gaze ? "running" : "running, eye tracker reader restarting")
? (gpage.status.awake === false ? "idle: " + gpage.status.idle + " (it starts when the gaze is used)"
: gpage.status.ft_gaze ? "running" : "running, eye tracker reader starting")
: backend.gazeServiceInstalled ? "not running (frametop-gaze.service, starts with SteamVR)"
: "not installed: run gaze/run.sh install in the Frametop folder, in a terminal"
color: backend.gazeServiceRunning ? Kirigami.Theme.textColor : Kirigami.Theme.negativeTextColor
Executable
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/python3
"""ft-pause: pause Frametop for a VR game, so it leaves the CPU and GPU to the game, or resume it.
ft-pause on|off|toggle pause or resume (input/game_pause.py has what pausing stops)
ft-pause status the pause state, as JSON (the default)
The input relay does it (frametop-input-relay.service). So do the pause_toggle action, the
controller gesture (by default both thumbsticks clicked together twice), and, with auto pause on,
a VR game starting; Frametop Input Settings has the settings, on its Game optimization page.
"""
import json
import os
import socket
import sys
def ask(msg, timeout=2.0):
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
s.bind(f"\0ft_pause.{os.getpid()}")
s.settimeout(timeout)
try:
s.sendto(msg.encode(), "\0frametop_relay")
return json.loads(s.recv(65536))
except ConnectionRefusedError:
sys.exit("the input relay isn't running (frametop-input-relay.service)")
except (socket.timeout, ValueError):
sys.exit("no answer from the input relay (an older one, without pausing?)")
finally:
s.close()
def main():
cmd = sys.argv[1] if len(sys.argv) > 1 else "status"
if cmd in ("on", "off", "toggle"):
r = ask(f"pause {cmd} command")
print("Frametop paused" if r.get("paused") else "Frametop running")
elif cmd == "status":
print(json.dumps(ask("pause ?"), indent=1))
else:
sys.exit(__doc__)
if __name__ == "__main__":
main()
+515
View File
@@ -0,0 +1,515 @@
"""Pause Frametop for VR games, so it leaves the headset's CPU and GPU to the game. The input
relay (input-relay.py) runs this: it's the one part of Frametop that always runs.
Paused:
- The gaze service stops (frametop-gaze: ft-gazed, ft-gaze, our own eye tracker ft-eyes, the
gaze panel), and with it every read of SteamVR's eye tracking. Our frame grabber, the root
service ft-eyegrab, goes idle by itself 3 seconds after ft-eyes stops asking for frames.
- Hand tracking stops if it runs (frametop-camd, frametop-hands).
- The desktop, by "pause_desktop": "hide" (the default) hides every screen and floating window
and slows the desktop down ("pause on" to ft-screens: KWin gets one frame callback a second
instead of 90, so it and the apps on it hardly draw), and remote desktop stops if it runs
(session/remote-ctl.sh: krdp, Xvnc, FreeRDP). Windows stay open. "close" closes the desktop
(desktops.sh stop): its windows close, and resuming starts it again (about 12 seconds), in
its start profile if it has one.
- In the relay: the 3D mouse lets go and the mouse is a plain mouse for SteamVR, typing goes to
Steam, and mapped buttons and key combinations do nothing but PAUSED_ACTIONS and commands.
Resuming starts again only what pausing stopped. The pointer helper and ft-powerd keep running:
they cost little, and the helper is what says a VR game started.
Toggled by:
- The pause_toggle action: a mouse button, a key combination, or a controller button outside
games (the pointer helper only reads those outside games).
- A controller gesture, by default both thumbsticks clicked together twice ("pause_gesture").
It's read from vrserver's web socket (vrws.py), which works in games and takes nothing from
them, so the game sees the clicks too.
- input/ft-pause on|off|toggle|status, which uses the relay's control socket
("pause on|off|toggle|?").
- VR games, with "pause_auto" (on by default): the pointer helper says "vrgame 1|0" as a SteamVR
scene app starts and ends (and every 5 seconds). A game starting pauses; a pause that starts
while a game runs ends RESUME_DELAY after the game does, unless another game starts first.
Toggled back on during a game, Frametop stays on until that game ends. A pause that starts
outside a game lasts until it's toggled off.
Settings in ~/.config/frametop-input.json (Frametop Input Settings, Game optimization page): "pause_auto"
(bool), "pause_gesture" ({"buttons": one or two controller buttons, "presses": 1 or 2}, or null
for none; one button always takes two presses), "pause_desktop" ("hide" or "close"), and
"pause_sound" (a sound on pause and resume; bool). The state outlives a relay restart in
STATE_PATH.
"""
import json
import os
import queue
import socket
import subprocess
import threading
import time
import vrws
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DESKTOPS = os.path.join(REPO, "desktops.sh")
REMOTE_CTL = os.path.join(REPO, "session", "remote-ctl.sh")
RELAY = "\0frametop_relay"
SCREENS = "\0ft_screens"
STATE_PATH = f"/run/user/{os.getuid()}/frametop-pause.json"
# User services that pausing stops, if they run.
UNITS = ("frametop-gaze.service", "frametop-hands.service", "frametop-camd.service")
# What still works while paused, besides commands ("command:CMD"): the toggle itself, and Steam's menu.
PAUSED_ACTIONS = ("pause_toggle", "steam_menu")
DEFAULT_GESTURE = {"buttons": ["left/thumbstick", "right/thumbstick"], "presses": 2}
DESKTOP_MODES = ("hide", "close")
SOUNDS = {True: "/usr/share/sounds/ocean/stereo/device-removed.oga",
False: "/usr/share/sounds/ocean/stereo/device-added.oga"}
RESUME_DELAY = 5.0 # after a VR game ends: loading the next one can end one scene app and start another
GAME_STALE = 12.0 # the helper repeats "vrgame" every 5 s; silence this long means no game (or no SteamVR)
CHORD_WINDOW = 0.3 # a two-button gesture's buttons go down within this of each other
DOUBLE_WINDOW = 0.7 # a double press's second press comes within this of the first
LOOKUP_EVERY = 30.0 # the controllers are looked up again this often (and sooner when they may have changed)
LOOKUP_GAP = 3.0 # but not more often than this for a message from a device it doesn't know
def device_of(text):
"""A web socket message's "device", without parsing all of it (there are about 160 a second)."""
at = text.find('"device"')
if at < 0:
return None
start = text.find('"', text.find(":", at) + 1)
end = text.find('"', start + 1)
return text[start + 1:end] if 0 <= start < end else None
def gesture_of(rules, buttons):
"""The rules' pause gesture as (buttons, presses), or None for none. `buttons` are the
controller buttons there are (the relay's VR_BUTTONS)."""
spec = rules.get("pause_gesture", DEFAULT_GESTURE)
if not isinstance(spec, dict) or not isinstance(spec.get("buttons"), list):
return None
chosen = tuple(dict.fromkeys(b for b in spec["buttons"] if b in buttons))[:2]
if not chosen:
return None
return chosen, 1 if len(chosen) == 2 and spec.get("presses") == 1 else 2
def settings_of(rules, buttons):
return {"auto": rules.get("pause_auto", True) is not False,
"gesture": gesture_of(rules, buttons),
"desktop": rules.get("pause_desktop") if rules.get("pause_desktop") in DESKTOP_MODES else "hide",
"sound": rules.get("pause_sound", True) is not False}
class Gesture:
"""The pause gesture, from button presses and releases. A press of a two-button gesture is
both buttons going down within CHORD_WINDOW of each other, so a button held for a while (to
sprint, say) and the other clicked doesn't count. Two presses within DOUBLE_WINDOW make a
double press."""
def __init__(self, buttons, presses):
self.buttons, self.presses = tuple(buttons), presses
self.down = {} # button -> when it went down
self.first = None # when the first press of a double press was
self.fired = None
def reset(self):
self.down.clear()
self.first = None
def feed(self, button, pressed, now):
"""A button's state; True when it completes the gesture."""
if button not in self.buttons:
return False
if not pressed:
self.down.pop(button, None)
return False
if button in self.down:
return False # still down
self.down[button] = now
if len(self.down) < len(self.buttons) or now - min(self.down.values()) > CHORD_WINDOW:
return False
if self.presses == 2 and (self.first is None or now - self.first > DOUBLE_WINDOW):
self.first = now
return False
self.first = None
if self.fired is not None and now - self.fired < 1.0:
return False # a triple press is still one
self.fired = now
return True
class ControllerWatch(threading.Thread):
"""Follows the Frame controllers on vrserver's web socket for the pause gesture, and sends
the relay "pause toggle gesture" when it's made. Reconnects every 5 seconds while SteamVR
is away. With no gesture set, it doesn't connect."""
def __init__(self, log):
super().__init__(name="pause-gesture", daemon=True)
self.log = log
self.lock = threading.Lock()
self.gesture = None
self.changed = threading.Event()
self.connected = False
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
self.lookup_at = None # look the controllers up again then (see recheck)
def recheck(self, delay=1.5):
"""The controllers' root paths may change soon: the 3D mouse's virtual controller is taking
or giving back its hand role. SteamVR takes a moment to hand the role over."""
self.lookup_at = time.monotonic() + delay
def set_gesture(self, spec):
"""(buttons, presses), or None for no gesture."""
with self.lock:
g = self.gesture
if spec is None:
self.gesture = None
elif g is None or (g.buttons, g.presses) != spec:
self.gesture = Gesture(*spec)
self.changed.set()
def current(self):
with self.lock:
return self.gesture
def run(self):
while True:
if self.current() is None:
if self.connected:
self.log("pause gesture: none set, stopped reading the controllers")
self.connected = False
self.changed.wait()
self.changed.clear()
continue
try:
self.session()
except (OSError, ValueError) as e: # urllib's errors are OSErrors
if self.connected:
self.log(f"pause gesture: SteamVR's web socket went away ({e}); retrying")
self.connected = False
self.changed.wait(5)
self.changed.clear()
def session(self):
ws = vrws.VrSocket()
try:
ws.open(f"frametop_pause_{os.getpid()}")
sides = {} # root path -> side
next_poll = 0.0
last_poll = 0.0
while True:
g = self.current()
if g is None:
return
now = time.monotonic()
lookup_at = self.lookup_at
if lookup_at is not None and now >= lookup_at:
self.lookup_at = None
next_poll = now
if now >= next_poll:
# A controller connects later, or changes its root path when the 3D mouse's
# virtual controller takes or gives back its hand role. Each lookup is an HTTP
# request to vrserver, so it's at connect, every LOOKUP_EVERY, when a message
# comes from a device not on the list, and when the relay says (recheck).
next_poll, last_poll = now + LOOKUP_EVERY, now
found = vrws.controllers()
if found != sides:
for path in sides.keys() - found.keys():
ws.subscribe(path, on=False)
for path in found.keys() - sides.keys():
ws.subscribe(path)
sides = found
g.reset()
if not self.connected:
self.connected = True
self.log(f"pause gesture: following {' and '.join(sorted(sides.values())) or 'no'} "
f"controller{'s' if len(sides) != 1 else ''} on SteamVR's web socket")
wanted = {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons}
for text in ws.messages(timeout=1.0):
device = device_of(text)
if device and device not in sides:
next_poll = min(next_poll, last_poll + LOOKUP_GAP) # its root path changed, maybe
# About 160 messages a second, nearly all capacitive sensing: parse only the
# few that carry one of the gesture's buttons.
if not any(c in text for _, c in wanted.values()):
continue
try:
msg = json.loads(text)
except ValueError:
continue
side = sides.get(msg.get("device"))
components = msg.get("components") or {}
for button, (s, component) in wanted.items():
if s == side and component in components:
if g.feed(button, bool(components[component]), time.monotonic()):
self.log("pause gesture")
try:
self.sock.sendto(b"pause toggle gesture", RELAY)
except OSError:
pass
finally:
ws.close()
def run(cmd, timeout=30):
"""A command's exit status (None if it couldn't run), its output going nowhere."""
try:
return subprocess.run(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=timeout).returncode
except (OSError, subprocess.TimeoutExpired):
return None
def output(cmd, timeout=10):
try:
return subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, text=True,
timeout=timeout).stdout
except (OSError, subprocess.TimeoutExpired):
return ""
def active(unit):
return run(["systemctl", "--user", "-q", "is-active", unit], timeout=10) == 0
def desktop_running():
return run(["pgrep", "-x", "ft-screens"], timeout=5) == 0
class Worker(threading.Thread):
"""Does the slow part of pausing and resuming, in order, one job at a time: stopping and
starting services, remote desktop, and closing or starting the desktop. `stopped` is what
the last pause stopped, for the resume to start again."""
def __init__(self, log, save):
super().__init__(name="pause-worker", daemon=True)
self.log, self.save = log, save
self.jobs = queue.Queue()
self.lock = threading.Lock()
self.stopped = {"units": [], "remote": False, "desktop": None}
self.busy = False
def submit(self, job, *args):
self.busy = True
self.jobs.put((job, args))
def run(self):
while True:
job, args = self.jobs.get()
try:
job(*args)
except Exception as e: # one failure mustn't stop the next pause or resume
self.log(f"pause: {job.__name__} failed: {e!r}")
self.busy = not self.jobs.empty()
self.save()
def snapshot(self):
with self.lock:
return {"units": list(self.stopped["units"]), "remote": self.stopped["remote"],
"desktop": self.stopped["desktop"]}
def restore(self, stopped):
with self.lock:
self.stopped = {"units": [u for u in stopped.get("units", []) if u in UNITS],
"remote": bool(stopped.get("remote")),
"desktop": stopped.get("desktop") if stopped.get("desktop") in ("hidden", "closed") else None}
def stop_units(self):
"""Stop the UNITS that run, adding them to what the resume starts again."""
units = [u for u in UNITS if active(u)]
if units:
run(["systemctl", "--user", "stop", *units], timeout=30)
with self.lock:
self.stopped["units"] += [u for u in units if u not in self.stopped["units"]]
self.log(f"pause: stopped {', '.join(units)}")
def pause(self, desktop):
if desktop == "close":
if desktop_running():
run([DESKTOPS, "stop"], timeout=40) # remote desktop goes with it
with self.lock:
self.stopped["desktop"] = "closed"
self.log("pause: desktop closed")
else:
if "running=1" in output([REMOTE_CTL, "status"]):
run([REMOTE_CTL, "stop"])
with self.lock:
self.stopped["remote"] = True
self.log("pause: remote desktop stopped")
with self.lock:
self.stopped["desktop"] = "hidden" # ft-screens was told already (or isn't running)
self.stop_units()
def enforce(self):
"""While paused: SteamVR restarted, and started the gaze service with it."""
self.stop_units()
def resume(self):
stopped = self.snapshot()
units = stopped["units"]
if units:
# They need SteamVR (Requisite=steamvr.service): without it they start with it next time.
if run(["systemctl", "--user", "start", *units], timeout=30) == 0:
self.log(f"resume: started {', '.join(units)}")
else:
self.log(f"resume: couldn't start {', '.join(units)} (SteamVR not running?)")
if stopped["remote"]:
# In a scope of its own, not this service's, so a relay restart doesn't end it.
run(["systemd-run", "--user", "--scope", "--collect", "--quiet", REMOTE_CTL, "start"])
self.log("resume: remote desktop started")
if stopped["desktop"] == "closed":
if run([DESKTOPS, "start"], timeout=60) == 0:
self.log("resume: desktop started")
else:
self.log("resume: the desktop didn't start (see /tmp/frametop-session.log)")
with self.lock:
self.stopped = {"units": [], "remote": False, "desktop": None}
class GamePause:
"""The pause state, its automatic side, and the relay's way in. `on_change(paused)` is
called (from the relay's thread) when the state flips, for the relay's own part."""
def __init__(self, log, on_change, buttons):
self.log, self.on_change, self.buttons = log, on_change, buttons
self.lock = threading.Lock() # the state file, written from both threads
self.paused = False
self.reason = "" # what paused it: gesture, button, key, controller, command, game
self.since = 0.0 # wall clock
self.ends_with_game = False # resume when the VR game ends
self.game = False
self.game_seen = 0.0 # last "vrgame" from the helper
self.resume_at = None
self.settings = settings_of({}, buttons)
self.screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
self.worker = Worker(log, self.save)
self.watch = ControllerWatch(log)
self.worker.start()
self.watch.start()
self.load()
# --- state file ---
def save(self):
with self.lock:
state = {"paused": self.paused, "reason": self.reason, "since": self.since,
"ends_with_game": self.ends_with_game, "stopped": self.worker.snapshot()}
try:
tmp = STATE_PATH + ".tmp"
with open(tmp, "w") as f:
json.dump(state, f)
os.replace(tmp, STATE_PATH)
except OSError as e:
self.log(f"pause: can't save {STATE_PATH}: {e}")
def load(self):
"""A relay restarted while paused stays paused (the relay's part is applied by its first
configure)."""
try:
with open(STATE_PATH) as f:
state = json.load(f)
except (OSError, ValueError):
return
if not isinstance(state, dict) or not state.get("paused"):
return
self.paused = True
self.reason = str(state.get("reason", ""))
self.since = float(state.get("since", 0) or 0)
self.ends_with_game = bool(state.get("ends_with_game"))
self.worker.restore(state.get("stopped") or {})
self.log(f"paused since before this relay started ({self.reason})")
self.worker.submit(self.worker.enforce)
# --- settings ---
def configure(self, rules):
self.settings = settings_of(rules, self.buttons)
self.watch.set_gesture(self.settings["gesture"])
if not self.settings["auto"]:
self.ends_with_game, self.resume_at = False, None
# --- pausing and resuming ---
def sound(self, paused):
if self.settings["sound"] and os.path.exists(SOUNDS[paused]):
try:
subprocess.Popen(["pw-play", SOUNDS[paused]], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)
except OSError:
pass
def tell_screens(self, paused):
try:
self.screens.sendto(b"pause on" if paused else b"pause off", SCREENS)
except OSError:
pass # the desktop isn't running
def pause(self, reason, now):
if self.paused:
return
self.paused, self.reason, self.since = True, reason, time.time()
self.ends_with_game = self.game and self.settings["auto"]
self.resume_at = None
self.log(f"Frametop paused ({reason}{', until the game ends' if self.ends_with_game else ''})")
self.sound(True)
self.tell_screens(True) # at once, even when the desktop is closing: that takes a while
self.on_change(True)
self.worker.submit(self.worker.pause, self.settings["desktop"])
self.save()
def resume(self, reason, now):
if not self.paused:
return
self.paused, self.resume_at, self.ends_with_game = False, None, False
self.log(f"Frametop resumed ({reason})")
self.sound(False)
self.tell_screens(False)
self.on_change(False)
self.worker.submit(self.worker.resume)
self.save()
def toggle(self, reason, now):
"""Pausing only happens by itself as a game starts, so a game resumed during stays on."""
(self.resume if self.paused else self.pause)(reason, now)
def set(self, on, reason, now):
if on != self.paused:
self.toggle(reason, now)
# --- VR games ---
def game_state(self, running, now):
"""From the pointer helper: a VR game runs or not (on a change and every 5 s)."""
self.game_seen = now
if running == self.game:
return
self.game = running
self.log(f"a VR game {'started' if running else 'ended'}")
if running:
self.resume_at = None # the next game came before the resume
if self.settings["auto"]:
self.pause("game", now)
elif self.paused and self.ends_with_game:
self.resume_at = now + RESUME_DELAY
def tick(self, now):
if self.game and now - self.game_seen > GAME_STALE:
self.game_state(False, now) # the helper went quiet: SteamVR (or the helper) stopped
if self.resume_at is not None and now >= self.resume_at:
self.resume("the game ended", now)
def timeout(self, now):
"""How long the relay may wait before tick() has something to do."""
if self.resume_at is not None:
return max(0.05, self.resume_at - now)
return 1.0 if self.game else 3600.0
def controllers_changed(self):
"""The 3D mouse took or gave back its hand role, which changes a controller's root path."""
self.watch.recheck()
def helper_started(self):
"""The pointer helper (re)started, so SteamVR did too, maybe with the gaze service."""
if self.paused:
self.worker.submit(self.worker.enforce)
def status(self):
return {"t": "pause", "paused": self.paused, "reason": self.reason, "since": self.since,
"ends_with_game": self.ends_with_game, "game": self.game,
"busy": self.worker.busy, "stopped": self.worker.snapshot(),
"gesture_reader": self.watch.connected, "auto": self.settings["auto"],
"desktop": self.settings["desktop"]}
+221 -42
View File
@@ -30,9 +30,12 @@ gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
window back (both to ft-floatd, @frametop_float), profile:NAME = switch to that profile (ft-layout
window back (both to ft-floatd, @frametop_float), spin_next and spin_prev = turn every panel in the
room about your head so the next one to the right or left comes to the front (ft-screens' "spin",
Meta+Alt+Tab and Meta+Alt+Shift+Tab by default), profile:NAME = switch to that profile (ft-layout
use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), command:CMD = run
Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), pause_toggle =
pause Frametop for a VR game, or resume it (game_pause.py), command:CMD = run
CMD with sh -c (on the host, as this service: its environment, output to its log, and
COMMAND_PATH, so ft-layout, ft-float and ft-steam need no path), key = pass through as a key, none).
@@ -45,8 +48,8 @@ A modifier on its own ("125": Meta) is a tap: pressed and released with no other
button, or scroll in between; the desktop gets an F24 press before its release, so Plasma's
launcher doesn't open on a Meta tap that's bound. A rules file without "key_bindings" gets
DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left, Meta+K: gaze_right, Meta+Shift+F:
float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu and
command actions work without pointer mode too. A combination with Meta also sends the desktop an
float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu,
pause and command actions work without pointer mode too. A combination with Meta also sends the desktop an
F24 press and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and
a gaze click isn't Meta+click (KWin's window move and resize). Another key while Meta is still
held gives the desktop Meta back. While typing goes to Steam, keyboards aren't grabbed, so Steam
@@ -92,6 +95,12 @@ keyboards) get their volume entries remapped to unused stand-in codes, so their
other keys keep working for SteamVR; a device without a keymap that has only volume
keys (the headset's pmic_resin) is grabbed. The keymaps go back when the relay exits.
Frametop can pause for VR games (game_pause.py: by hand, with a controller gesture, or by itself
while a game runs). Paused, it stops the services that cost the game CPU and GPU, and the relay
plays a plain one: the pointer devices feed the virtual mouse and keyboard as without pointer
mode, typing goes to Steam, and mapped buttons and key combinations do only pause_toggle,
steam_menu and command:CMD (game_pause.PAUSED_ACTIONS).
Pointer mode (POINTER=1 in ~/.config/frametop.conf) sends pointer devices to
the ft-pointer helper (pointer/helper), which drives the ft_pointer
SteamVR driver. With POINTER=0, pointer devices go to the virtual mouse and
@@ -104,7 +113,10 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
vrcapture <s> take every controller button for s seconds (0: stop), so the settings
app can capture one; watchers see them as events with id frame_controller
vrbtn, vrhello, gazeawake from the pointer helper (above)
vrgame 1|0 from the pointer helper: a VR game runs (on a change and every 5 s), for pausing
textfield 1|0 from the desktop's input method (above)
pause on|off|toggle [reason] pause Frametop or resume it (input/ft-pause; the gesture reader)
pause ? the pause state, as {"t": "pause", ...}
Runs on the Frame host as a user service (frametop-input-relay.service). The
virtual devices are parked in systemd's file descriptor store, so a relay
@@ -118,6 +130,7 @@ kernel's evdev and uinput interfaces.
"""
import array
import atexit
import collections
import errno
import fcntl
import json
@@ -130,6 +143,8 @@ import subprocess
import sys
import time
import game_pause
# Linux input constants (include/uapi/linux/input-event-codes.h, input.h, uinput.h).
EV_SYN, EV_KEY, EV_REL, EV_MSC = 0x00, 0x01, 0x02, 0x04
SYN_REPORT = 0
@@ -139,10 +154,10 @@ REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C} # REL_HWHEEL, REL_WHEEL and their _HI_RES
BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
# Volume keys are remapped to KEY_MACRO29 and KEY_MACRO30: above 255, so X11 can't
# carry them, and bound to nothing in the default keymap.
VOLUME_STANDIN = {KEY_VOLUMEUP: 0x2AC, KEY_VOLUMEDOWN: 0x2AD}
KEY_MUTE, KEY_VOLUMEDOWN, KEY_VOLUMEUP = 113, 114, 115
# Volume keys are remapped to KEY_MACRO28, KEY_MACRO29 and KEY_MACRO30: above 255, so X11
# can't carry them, and bound to nothing in the default keymap.
VOLUME_STANDIN = {KEY_MUTE: 0x2AB, KEY_VOLUMEUP: 0x2AC, KEY_VOLUMEDOWN: 0x2AD}
VOLUME_ORIGINAL = {v: k for k, v in VOLUME_STANDIN.items()}
VOLUME_CODES = set(VOLUME_STANDIN) | set(VOLUME_ORIGINAL)
BUS_USB, BUS_BLUETOOTH, BUS_VIRTUAL = 0x03, 0x05, 0x06
@@ -196,14 +211,16 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
"gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
"dock_all", "steam_menu", "key", "none")
"dock_all", "spin_next", "spin_prev", "steam_menu", "pause_toggle", "key", "none")
# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
# Meta+Shift+F floats a window.
# Meta+Shift+F floats a window, and Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels around
# you (ft-screens' lazy susan); not Meta+Tab, which is Cmd+Tab on a Mac reached through a remote
# desktop like RustDesk.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
KEY_F24 = 194 # sent to the desktop with a Meta combination (see the top)
# Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
@@ -219,6 +236,9 @@ GAZED = "\0ft_gazed"
FLOAT = "\0frametop_float" # ft-floatd, floating windows in the Frametop desktop
# Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
# Actions for ft-screens: spin_next and spin_prev turn every panel in the room about your head,
# so the next one to the right or left comes to the front. They don't need pointer mode either.
SCREENS_ACTIONS = {"spin_next": b"spin next", "spin_prev": b"spin prev"}
PROFILE = "profile:" # "profile:NAME": switch to that profile (doesn't need pointer mode either)
COMMAND = "command:" # "command:CMD": run CMD (nor does this)
@@ -229,7 +249,17 @@ def known_action(a):
def needs_pointer(a):
return a not in FLOAT_ACTIONS and a != "steam_menu" and not a.startswith((PROFILE, COMMAND))
return a not in FLOAT_ACTIONS and a not in SCREENS_ACTIONS and a not in ("steam_menu", "pause_toggle") and not a.startswith((PROFILE, COMMAND))
def works_paused(a):
"""An action that still does something while Frametop is paused (game_pause.py)."""
return a in game_pause.PAUSED_ACTIONS or a.startswith(COMMAND)
# What pressed pause_toggle, for the log (do_action's source).
PAUSE_SOURCES = {"mouse": "mouse button", "keyboard": "key combination", "left": "controller button",
"right": "controller button"}
KEYS = "\0frametop_keys" # keys of keyboards grabbed for the desktop, for other readers
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FT_LAYOUT = os.path.join(REPO, "layout", "ft-layout")
@@ -399,9 +429,13 @@ class Volume:
def key(self, fd, code, value, now):
if value == 1:
self.held = (fd, code)
self.step(code)
self.next_at = now + self.DELAY
if VOLUME_ORIGINAL.get(code, code) == KEY_MUTE:
subprocess.Popen(["wpctl", "set-mute", "@DEFAULT_AUDIO_SINK@", "toggle"],
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
else:
self.held = (fd, code)
self.step(code)
self.next_at = now + self.DELAY
elif value == 0 and self.held == (fd, code):
self.release()
@@ -437,14 +471,22 @@ class Pointer:
CLAIM_PULSE = 0.06 # seconds the claim button (switchlaserhand, no click) is held
RESUME_PAUSE = 1.5 # mouse idle this long, then moving again, re-claims the laser
WAKE_WINDOW = 1.0 # seconds in which WAKE_COUNTS of motion must add up
QUEUE_MAX = 512 # commands kept while the helper is behind (see send)
MOVE_EVERY = 0.004 # mouse motion goes to the helper at most this often (see flush)
def __init__(self, sensitivity, idle, wake_counts=40):
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
# Never blocks (see send): a stalled helper must not stall the keyboard, volume keys and pausing.
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
# Commands the helper's full socket didn't take yet, in order: text, or [dyaw, dpitch] for
# mouse moves, which add up into one while they wait.
self.queue = collections.deque()
self.behind_logged = -60.0
self.sensitivity = sensitivity # degrees per mouse count
self.idle = idle
self.active = False
self.last_used = 0.0
self.dx = self.dy = 0
self.move_at = 0.0 # motion last went to the helper then
self.scroll_until = None
self.claim_at = None # when to press the claim button
self.claim_release = None
@@ -457,11 +499,59 @@ class Pointer:
self.pending_since = 0.0
self.gaze_awake_until = 0.0 # the helper's gaze mode keeps the pointer until then
def send(self, command):
try:
self.sock.sendto(command.encode(), HELPER)
except OSError:
pass # helper not running (SteamVR not running)
def send(self, command, droppable=False):
"""To the helper, in order, without blocking. While the helper doesn't keep up (place and
grabprobe hold it for seconds, and ft-gazed's 90 Hz gaze fills its socket meanwhile), commands
wait in the queue and go out from tick(). A droppable one (a scroll notch: scrolling seconds
late is no use) is dropped instead; presses, releases and the rest are kept, so no button
stays down. Moves add up (_move)."""
if not self.queue:
try:
self.sock.sendto(command.encode(), HELPER)
return
except BlockingIOError:
pass
except OSError:
return # helper not running (SteamVR not running)
if not droppable:
self._queue(command)
def _move(self, dyaw, dpitch):
if self.queue and isinstance(self.queue[-1], list):
self.queue[-1][0] += dyaw
self.queue[-1][1] += dpitch
return
if not self.queue:
try:
self.sock.sendto(f"move {dyaw:.4f} {dpitch:.4f}".encode(), HELPER)
return
except BlockingIOError:
pass
except OSError:
return
self._queue([dyaw, dpitch])
def _queue(self, item):
if not self.queue and time.monotonic() - self.behind_logged > 60:
self.behind_logged = time.monotonic()
log("pointer helper is behind: holding its commands (logged once a minute)")
self.queue.append(item)
if len(self.queue) > self.QUEUE_MAX:
self.queue.popleft() # stalled for long: the oldest goes
def drain(self):
"""Send what waits, in order, as far as the helper takes it."""
while self.queue:
item = self.queue[0]
text = f"move {item[0]:.4f} {item[1]:.4f}" if isinstance(item, list) else item
try:
self.sock.sendto(text.encode(), HELPER)
except BlockingIOError:
return
except OSError:
self.queue.clear() # the helper went away: nothing to deliver to
return
self.queue.popleft()
def wake(self, now):
if not self.active:
@@ -489,7 +579,7 @@ class Pointer:
elif code == REL_Y:
self.dy += value
elif code == REL_WHEEL and value:
self.send(f"scroll 0 {1 if value > 0 else -1}")
self.send(f"scroll 0 {1 if value > 0 else -1}", droppable=True)
self.scroll_until = now + self.SCROLL_PULSE
def action(self, name, value, now, source="mouse"):
@@ -521,7 +611,7 @@ class Pointer:
return # the rest act on press
elif name in ("scroll_up", "scroll_down"):
self.wake(now)
self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}")
self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}", droppable=True)
self.scroll_until = now + self.SCROLL_PULSE
elif name == "dashboard":
self.dashboard(now)
@@ -560,11 +650,16 @@ class Pointer:
self.sensitivity *= 1.25 if name == "sens_up" else 0.8
log(f"sensitivity {self.sensitivity:.4f} deg/count")
def flush(self):
if self.dx or self.dy:
# Mouse right turns the ray right (negative yaw); mouse down tilts it down.
self.send(f"move {-self.dx * self.sensitivity:.4f} {-self.dy * self.sensitivity:.4f}")
self.dx = self.dy = 0
def flush(self, now=None):
"""Send the motion so far. With now (a mouse's SYN_REPORT), only once MOVE_EVERY has passed
since the last: a 1000 Hz mouse sent the helper, which runs every 8 ms, 1000 datagrams a
second. tick() sends the rest when it's due; a button sends it first, so it lands there."""
if not (self.dx or self.dy) or (now is not None and now - self.move_at < self.MOVE_EVERY):
return
self.move_at = time.monotonic() if now is None else now
# Mouse right turns the ray right (negative yaw); mouse down tilts it down.
self._move(-self.dx * self.sensitivity, -self.dy * self.sensitivity)
self.dx = self.dy = 0
def dashboard(self, now=None):
"""Toggle the SteamVR dashboard with the virtual controller's system button.
@@ -579,6 +674,8 @@ class Pointer:
self.system_at = now + (0.4 if woke else 0.0)
def tick(self, now):
self.drain()
self.flush(now)
if self.system_at is not None and now >= self.system_at:
self.send("btn system 1")
self.system_at = None
@@ -603,7 +700,26 @@ class Pointer:
def timeout(self):
pending = (self.scroll_until, self.claim_at, self.claim_release, self.system_at, self.system_release)
return 0.02 if any(t is not None for t in pending) else 0.5
wait = 0.02 if self.queue or any(t is not None for t in pending) else 0.5
if self.dx or self.dy: # motion held back by flush
wait = max(0.0, min(wait, self.move_at + self.MOVE_EVERY - time.monotonic()))
return wait
def stand_down(self):
"""Frametop is pausing: a pulse under way ends now, and the pointer lets go."""
if self.system_release is not None:
self.send("btn system 0")
if self.claim_release is not None:
self.send("btn a 0")
if self.scroll_until is not None:
self.send("scroll 0 0")
self.system_at = self.system_release = self.claim_at = self.claim_release = self.scroll_until = None
self.dx = self.dy = self.pending = 0
self.gaze_awake_until = 0.0
if self.active:
self.send("hide")
self.active = False
log("pointer off (paused)")
class Node:
@@ -688,23 +804,43 @@ def main():
# desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
# every second); typing_applied: the grabs match that as of the last apply_roles().
# vr_capture_until: every controller button is taken until then (the settings app capturing one).
state = {"pointer": None, "rules": {}, "share_keys": False,
"desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0}
# pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they
# differ while Frametop is paused).
state = {"pointer": None, "pointer_conf": None, "rules": {}, "share_keys": False,
"desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False,
"pointer_holding": False}
def pause_changed(paused):
"""Frametop paused or resumed (game_pause.py): the relay's own part."""
p = state["pointer_conf"]
if paused:
if p:
p.stand_down()
state["pointer"] = None
state["desktop_until"] = 0.0 # typing goes to Steam
else:
state["pointer"] = p
apply_roles()
vr_bind(time.monotonic())
pause = game_pause.GamePause(log, pause_changed, VR_BUTTONS)
def load_config():
conf = read_config()
state["rules"] = read_rules()
state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
if conf.get("POINTER", "0") == "1":
p = state["pointer"] or Pointer(0.03, 30)
p = state["pointer_conf"] or Pointer(0.03, 30)
p.sensitivity = float(conf.get("POINTER_SENSITIVITY", "0.03"))
p.idle = float(conf.get("POINTER_IDLE", "30"))
p.wake_counts = int(conf.get("POINTER_WAKE_COUNTS", "40"))
state["pointer"] = p
state["pointer_conf"] = p
log(f"pointer mode: {p.sensitivity} deg/count, idle {p.idle} s, wake {p.wake_counts} counts")
else:
state["pointer"] = None
state["pointer_conf"] = None
log("pointer mode off: pointer devices feed the virtual mouse and keyboard")
state["pointer"] = None if pause.paused else state["pointer_conf"]
pause.configure(state["rules"])
load_config()
tap = None # the modifier (folded, MODIFIERS) pressed alone, with nothing since: its release is a tap
@@ -719,11 +855,14 @@ def main():
state["vr_capture_until"] = 0.0
buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items()
if b in VR_BUTTONS and known_action(a) and a not in ("key", "none")
and a not in GAZE_ACTIONS) or "-"
and a not in GAZE_ACTIONS and (not pause.paused or works_paused(a))) or "-"
if state["rules"].get("controller_in_games"):
buttons = "+games " + buttons
state["vr_bind_retry"] = False
try:
screens_sock.sendto(f"vrbind {buttons}".encode(), HELPER)
except BlockingIOError:
state["vr_bind_retry"] = True # the helper is behind: again on the next loop
except OSError:
pass # helper not running; it says vrhello when it starts
@@ -764,6 +903,12 @@ def main():
def do_action(action, value, now, source="mouse"):
"""A mapped mouse or controller button, or key combination (pointer mode only, but
for the actions needs_pointer() says don't)."""
if action == "pause_toggle":
if value == 1:
pause.toggle(PAUSE_SOURCES.get(source, source), now)
return
if pause.paused and not works_paused(action):
return
if action == "keyboard_toggle":
if value == 1 and vr_keyboard_mode() != "never":
vr_keyboard("toggle")
@@ -791,6 +936,13 @@ def main():
except OSError:
pass # the Frametop desktop isn't running
log(action)
elif action in SCREENS_ACTIONS:
if value == 1:
try:
screens_sock.sendto(SCREENS_ACTIONS[action], SCREENS)
except OSError:
pass # the Frametop desktop isn't running
log(action)
elif state["pointer"]:
state["pointer"].action(action, value, now, source)
@@ -832,7 +984,7 @@ def main():
return value == 2 and code in combos_down
combo = "+".join(str(c) for c in sorted(held_modifiers) + [code])
action = state["rules"]["key_bindings"].get(combo)
if not known_action(action) or action in ("key", "none"):
if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
return False
combos_down[code] = action
if held_meta - meta_hidden:
@@ -853,7 +1005,7 @@ def main():
def modifier_tap(mod, now):
"""A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
action = state["rules"]["key_bindings"].get(str(mod))
if not known_action(action) or action in ("key", "none"):
if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
return
# The desktop gets a key in between before the release goes there, so the tap isn't one
# there too: a Meta tap would open Plasma's launcher.
@@ -1054,7 +1206,7 @@ def main():
try:
if cmd == "keyboard":
# From ft-screens (unbound, no reply): where typing goes, repeated every second.
desktop = len(words) > 1 and words[1] == "desktop"
desktop = len(words) > 1 and words[1] == "desktop" and not pause.paused
state["desktop_until"] = now + 3.0 if desktop else 0.0
continue
if cmd == "vrbtn" and len(words) == 3 and words[1] in VR_BUTTONS and words[2] in ("0", "1"):
@@ -1062,6 +1214,20 @@ def main():
continue
if cmd == "vrhello":
vr_bind(now)
pause.helper_started()
continue
if cmd == "vrgame" and len(words) == 2:
pause.game_state(words[1] == "1", now)
continue
if cmd == "pause" and len(words) >= 2 and words[1] in ("on", "off", "toggle", "?"):
# From input/ft-pause, the settings app, or the gesture reader (unbound, no reply).
reason = words[2] if len(words) > 2 else "command"
if words[1] == "toggle":
pause.toggle(reason, now)
elif words[1] != "?":
pause.set(words[1] == "on", reason, now)
if addr:
reply(addr, pause.status())
continue
if cmd == "textfield" and len(words) == 2:
text_field(words[1] == "1")
@@ -1073,7 +1239,8 @@ def main():
if not addr:
continue # unbound sender, nowhere to reply
if cmd == "devices":
reply(addr, {"t": "devices", "pointer_mode": state["pointer"] is not None,
reply(addr, {"t": "devices", "pointer_mode": state["pointer_conf"] is not None,
"paused": pause.paused,
"actions": ACTIONS,
"nodes": [n.describe() for n in nodes.values() if n.candidate]})
elif cmd == "watch":
@@ -1155,17 +1322,29 @@ def main():
if added:
apply_roles()
# The 3D mouse connecting or letting go moves a hand role, so the pause gesture's reader
# looks the controllers up again (game_pause.py).
holding = bool(state["pointer_conf"] and state["pointer_conf"].active)
if holding != state["pointer_holding"]:
state["pointer_holding"] = holding
pause.controllers_changed()
ready, _, _ = select.select(list(nodes) + [control], [], [],
volume.timeout(now, pointer.timeout() if pointer else 0.5))
min(volume.timeout(now, pointer.timeout() if pointer else 0.5), pause.timeout(now)))
now = time.monotonic()
if pointer:
pointer.tick(now)
elif state["pointer_conf"]:
state["pointer_conf"].drain() # paused: what waited still goes, in order
if state["vr_bind_retry"]:
vr_bind(now)
volume.tick(now)
pause.tick(now)
if state["vr_capture_until"] and now >= state["vr_capture_until"]:
vr_bind(now) # capture over: back to the mapped buttons
if (now < state["desktop_until"]) != state["typing_applied"] or waiting:
waiting = apply_roles()
for fd in ready:
pointer = state["pointer"] # (a pause or resume in this batch changes it)
if fd is control:
handle_control(now)
continue
@@ -1220,7 +1399,7 @@ def main():
continue
if etype == EV_KEY:
action = buttons.get(str(code), DEFAULT_BUTTONS.get(code, "key"))
if pointer and action not in ("key", "none"):
if (pointer or action == "pause_toggle") and action not in ("key", "none"):
do_action(action, value, now)
continue
if action == "none":
@@ -1239,7 +1418,7 @@ def main():
mouse.emit(etype, code, value)
elif etype == EV_SYN and code == SYN_REPORT:
if pointer:
pointer.flush()
pointer.flush(now)
mouse.sync()
keyboard.sync()
+79 -5
View File
@@ -3,8 +3,9 @@
Runs the relay's main() against a fake pass-through keyboard and a fake mouse (pipes), with
every socket it sends to renamed, no uinput devices, no grabs, and ft-steam swapped for a
logger. Nothing reaches the live desktop, SteamVR, or the running relay, so it's safe next
to them. Rules come from the test, not ~/.config/frametop-input.json.
logger. Pausing (game_pause.py) is a stub: no threads, state file, or services. Nothing reaches
the live desktop, SteamVR, or the running relay, so it's safe next to them. Rules come from the
test, not ~/.config/frametop-input.json.
input/test/keys-test.py [RELAY] (default: input/input-relay.py next to this folder)
"""
@@ -26,8 +27,41 @@ with open(relay_path) as f:
src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
relay.__file__ = os.path.abspath(relay_path)
sys.path.insert(0, os.path.dirname(os.path.abspath(relay_path))) # its game_pause and vrws
exec(compile(src, relay_path, "exec"), relay.__dict__)
class StubPause:
"""game_pause.GamePause without its threads, state file, or services."""
paused = False
def __init__(self, log, on_change, buttons):
self.on_change = on_change
stub["pause"] = self
def toggle(self, reason, now):
self.paused = not self.paused
self.on_change(self.paused)
def set(self, on, reason, now):
if on != self.paused:
self.toggle(reason, now)
def status(self):
return {"t": "pause", "paused": self.paused}
def timeout(self, now):
return 3600.0
def configure(self, *args):
pass
game_state = tick = helper_started = controllers_changed = configure
stub = {}
relay.game_pause.GamePause = StubPause
for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "KEYS"):
setattr(relay, name, f"\0{tag}_{name.lower()}")
OUT = tempfile.mkdtemp(prefix="ft-keys-test-")
@@ -80,6 +114,7 @@ fake_os = type(os)("os")
fake_os.__dict__.update(os.__dict__)
fake_os.listdir = lambda path: ["event900", "event901"] if path == "/dev/input" else os.listdir(path)
fake_os.stat = lambda path, *a, **k: Inode() if path in FAKE else os.stat(path, *a, **k)
fake_os.access = lambda path, mode, *a, **k: path in FAKE or os.access(path, mode, *a, **k)
relay.os = fake_os
@@ -143,6 +178,15 @@ def use(key_bindings):
typed()
def pause(cmd):
c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
c.bind("")
c.settimeout(2)
c.sendto(f"pause {cmd} test".encode(), f"\0{tag}_relay")
c.recv(4096)
time.sleep(0.05)
def lines(path, wait=0.3):
time.sleep(wait)
try:
@@ -162,7 +206,7 @@ def check(label, got, want):
failures.append(label)
META, RMETA, SHIFT, CTRL, RCTRL, A, F, J = 125, 126, 42, 29, 97, 30, 33, 36
META, RMETA, SHIFT, CTRL, RCTRL, ALT, A, F, J, P, TAB = 125, 126, 42, 29, 97, 56, 30, 33, 36, 25, 15
F24 = ["key 194 1", "key 194 0"]
@@ -206,9 +250,39 @@ def tests():
check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "3"])
check("Ctrl tap: F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
use({"125+36": "gaze_left", "29": f"command:echo paused >> {cmd_log}", "42+125+25": "pause_toggle"})
pause("on")
check("pause on: paused", stub["pause"].paused, True)
key(META, 1); key(J, 1); key(J, 0); key(META, 0)
check("paused: Meta+J (gaze click) does nothing and is typed as is", typed(),
["key 125 1", "key 36 1", "key 36 0", "key 125 0"])
key(RCTRL, 1); key(RCTRL, 0)
check("paused: a command still runs", lines(cmd_log), ["combo", "3", "paused"])
typed()
key(SHIFT, 1); key(META, 1); key(P, 1); key(P, 0); key(META, 0); key(SHIFT, 0)
check("paused: Meta+Shift+P (pause_toggle) resumes", stub["pause"].paused, False)
check("Meta+Shift+P isn't typed", [k for k in typed() if k.startswith("key 25 ")], [])
key(META, 1); key(J, 1); key(J, 0); key(META, 0)
check("resumed: Meta+J is a combination again", typed(), ["key 125 1"] + F24 + ["key 125 0"])
use(None) # the defaults: Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels (ft-screens)
key(META, 1); key(ALT, 1); key(TAB, 1); key(TAB, 0); key(ALT, 0); key(META, 0)
got = typed()
check("Meta+Alt+Tab (default): spin next", [m for m in got if m.startswith("spin")], ["spin next"])
check("Meta+Alt+Tab: Tab isn't typed", [m for m in got if m.startswith("key 15 ")], [])
key(META, 1); key(ALT, 1); key(SHIFT, 1); key(TAB, 1); key(TAB, 0); key(SHIFT, 0); key(ALT, 0); key(META, 0)
check("Meta+Alt+Shift+Tab (default): spin prev", [m for m in typed() if m.startswith("spin")], ["spin prev"])
pause("on")
key(META, 1); key(ALT, 1); key(TAB, 1); key(TAB, 0); key(ALT, 0); key(META, 0)
check("paused: Meta+Alt+Tab doesn't spin", [m for m in typed() if m.startswith("spin")], [])
pause("off")
check("spinning works without pointer mode", (relay.needs_pointer("spin_next"), relay.needs_pointer("spin_prev")),
(False, False))
check("an empty command isn't an action", relay.known_action("command: "), False)
check("steam_menu and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("command:ls")), (False, False))
check("steam_menu, pause_toggle and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
(False, False, False))
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
shutil.rmtree(OUT, ignore_errors=True)
os._exit(1 if failures else 0)
+214
View File
@@ -0,0 +1,214 @@
#!/usr/bin/env python3
"""Offline test of pausing for VR games (input/game_pause.py): the controller gesture, and when
Frametop pauses and resumes by itself. The worker (services, remote desktop, the desktop) and
the controller reader are stubs, the state file is a temporary one, and ft-screens' socket is
renamed, so nothing reaches the live Frametop, SteamVR, or the running relay.
input/test/pause-test.py
"""
import os
import sys
import tempfile
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
import game_pause as gp # noqa: E402
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
# ---------------------------------------------------------------- the gesture
L, R = "left/thumbstick", "right/thumbstick"
def presses(g, events):
"""events: (time, button, down); how many times the gesture went off."""
return sum(g.feed(b, d, t) for t, b, d in events)
def both(t, hold=0.1, gap=0.02):
"""Both sticks clicked at t (the right one gap later), held for hold."""
return [(t, L, True), (t + gap, R, True), (t + hold, L, False), (t + hold + gap, R, False)]
g = gp.Gesture((L, R), 2)
check("both sticks twice: goes off", presses(g, both(0) + both(0.3)), 1)
g = gp.Gesture((L, R), 2)
check("both sticks once: nothing", presses(g, both(0)), 0)
g = gp.Gesture((L, R), 2)
check("both sticks twice, 0.9 s apart: nothing", presses(g, both(0) + both(0.9)), 0)
g = gp.Gesture((L, R), 2)
check("left held (sprint), right clicked twice: nothing",
presses(g, [(0, L, True), (1.0, R, True), (1.1, R, False), (1.3, R, True), (1.4, R, False)]), 0)
g = gp.Gesture((L, R), 2)
check("sticks 0.5 s apart, twice: nothing", presses(g, both(0, gap=0.5, hold=0.6) + both(1.2, gap=0.5, hold=0.6)), 0)
g = gp.Gesture((L, R), 2)
check("three double presses in a row: once", presses(g, both(0) + both(0.3) + both(0.6)), 1)
g = gp.Gesture((L, R), 2)
check("again after a second: twice in all", presses(g, both(0) + both(0.3) + both(2.0) + both(2.3)), 2)
g = gp.Gesture((L, R), 2)
check("repeated down states count once",
presses(g, [(0, L, True), (0.01, L, True), (0.02, R, True), (0.03, R, True), (0.1, L, False), (0.1, R, False)]
+ both(0.3)), 1)
g = gp.Gesture((L, R), 1)
check("both sticks once (presses 1): goes off", presses(g, both(0)), 1)
g = gp.Gesture(("right/b",), 2)
check("one button double-clicked: goes off",
presses(g, [(0, "right/b", True), (0.1, "right/b", False), (0.3, "right/b", True)]), 1)
g = gp.Gesture(("right/b",), 2)
check("other buttons don't count", presses(g, [(0, "right/a", True), (0.2, "right/a", True)]), 0)
VR = ("left/thumbstick", "right/thumbstick", "right/b", "right/a")
check("no pause_gesture: both sticks twice", gp.gesture_of({}, VR), ((L, R), 2))
check("pause_gesture null: none", gp.gesture_of({"pause_gesture": None}, VR), None)
check("one button, presses 1: still twice", gp.gesture_of({"pause_gesture": {"buttons": ["right/b"], "presses": 1}}, VR),
(("right/b",), 2))
check("unknown buttons dropped", gp.gesture_of({"pause_gesture": {"buttons": ["right/zz", "right/a"]}}, VR),
(("right/a",), 2))
check("settings: defaults", gp.settings_of({}, VR)["auto"], True)
check("settings: unknown desktop mode is hide", gp.settings_of({"pause_desktop": "x"}, VR)["desktop"], "hide")
# ---------------------------------------------------------------- pausing by itself
class StubWorker:
def __init__(self, log, save):
self.jobs = []
self.busy = False
def start(self):
pass
def submit(self, job, *args):
self.jobs.append(job.__name__)
def snapshot(self):
return {"units": [], "remote": False, "desktop": None}
def restore(self, stopped):
self.restored = stopped
def pause(self, desktop):
pass
def resume(self):
pass
def enforce(self):
pass
class StubWatch:
connected = False
def __init__(self, log):
pass
def start(self):
pass
def set_gesture(self, spec):
self.spec = spec
tmp = tempfile.mkdtemp(prefix="ft-pause-test-")
gp.STATE_PATH = os.path.join(tmp, "pause.json")
gp.SCREENS = f"\0ft_pause_test_{os.getpid()}_screens"
gp.Worker, gp.ControllerWatch = StubWorker, StubWatch
changes = []
def new(rules=None):
"""A relay starting afresh: not paused before."""
if os.path.exists(gp.STATE_PATH):
os.remove(gp.STATE_PATH)
changes.clear()
p = gp.GamePause(lambda *a: None, changes.append, VR)
p.configure(dict({"pause_sound": False}, **(rules or {})))
return p
p = new()
p.game_state(True, 0)
check("a game starts: paused", (p.paused, p.reason, p.ends_with_game), (True, "game", True))
check("the relay heard", changes, [True])
p.game_state(True, 5)
p.tick(5)
check("the helper repeats it: still paused, nothing new", (p.paused, changes), (True, [True]))
p.game_state(False, 10)
p.tick(12)
check("the game ends: not yet resumed", p.paused, True)
p.tick(15.1)
check("5 s later: resumed", (p.paused, changes), (False, [True, False]))
check("the worker paused and resumed", p.worker.jobs, ["pause", "resume"])
p = new()
p.game_state(True, 0)
p.game_state(False, 10)
p.game_state(True, 12)
p.tick(16)
check("the next game starts within 5 s: stays paused", p.paused, True)
p.game_state(False, 20)
p.tick(25.1)
check("and resumes 5 s after that one ends", p.paused, False)
p = new()
p.game_state(True, 0)
p.toggle("gesture", 1)
check("resumed during a game: running", p.paused, False)
p.game_state(True, 6)
p.tick(30)
check("and stays running while that game runs", p.paused, False)
p.game_state(False, 40)
p.game_state(True, 50)
check("the next game pauses again", p.paused, True)
p = new()
p.toggle("command", 0)
check("paused outside a game: not tied to one", (p.paused, p.ends_with_game), (True, False))
p.game_state(True, 1)
p.game_state(False, 10)
p.tick(20)
check("a game comes and goes: still paused", p.paused, True)
p.toggle("command", 21)
check("toggled: running", p.paused, False)
p = new({"pause_auto": False})
p.game_state(True, 0)
check("auto off: a game doesn't pause", p.paused, False)
p.toggle("gesture", 1)
p.game_state(False, 10)
p.tick(20)
check("auto off: a pause during a game outlasts it", p.paused, True)
p = new()
p.game_state(True, 0)
p.tick(0 + gp.GAME_STALE + 0.1)
check("the helper goes quiet: the game counts as ended", p.game, False)
p.tick(0 + gp.GAME_STALE + gp.RESUME_DELAY + 0.2)
check("and the pause ends", p.paused, False)
p = new()
p.toggle("gesture", 0)
q = gp.GamePause(lambda *a: None, changes.append, VR)
check("a relay restarted while paused stays paused", (q.paused, q.reason), (True, "gesture"))
check("and stops the services again", q.worker.jobs, ["enforce"])
q.toggle("gesture", 1)
r = gp.GamePause(lambda *a: None, changes.append, VR)
check("resumed: the next relay starts running", r.paused, False)
p = new({"pause_gesture": None})
check("pause_gesture null: the reader gets none", p.watch.spec, None)
check("status", sorted(p.status()), sorted(["t", "paused", "reason", "since", "ends_with_game", "game", "busy",
"stopped", "gesture_reader", "auto", "desktop"]))
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
for f in os.listdir(tmp):
os.remove(os.path.join(tmp, f))
os.rmdir(tmp)
sys.exit(1 if failures else 0)
Executable
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/python3
"""vrws: vrserver's local web socket, with nothing but the standard library.
vrserver (SteamVR) serves its controller binding page on 127.0.0.1:27062, and that page
follows the controllers' raw input through a web socket there (mailbox input_server,
request_input_state_updates). Reading it takes nothing from anyone: whatever the buttons are
bound to still happens, and it works whatever app has input focus, VR games included. SteamVR
input only reaches the focused app, or an overlay with global input, which then takes the button
from the game (pointer/helper/vrbuttons.h). frame-voice reads its push-to-talk button the same
way. It's undocumented, so a SteamVR update could change it (scripts/update-check.py checks it).
The host's Python has no `websockets` module, so this is a small RFC 6455 client of its own.
controllers() the Frame controllers SteamVR has now: {root path: "left" | "right"}. A
controller's root path is /user/hand/<side>, or /devices/cv/<serial> while the
3D mouse's virtual controller holds that hand role.
VrSocket() a connection: open(mailbox), subscribe(path), messages(timeout) -> raw texts,
each a JSON object like {"type": "update_component_states", "device": path,
"components": {"/input/thumbstick/click": true, ...}} with the components
that changed (mostly capacitive sensing, about 160 a second)
Run as a program, it prints the controllers' button changes for a while (default 10 s):
input/vrws.py [seconds]
"""
import base64
import json
import os
import select
import socket
import struct
import sys
import time
import urllib.request
HOST, PORT = "127.0.0.1", 27062
ORIGIN = f"http://{HOST}:{PORT}"
# vrserver answers the binding page's own requests; these headers make ours look like them.
HEADERS = {"Referer": f"{ORIGIN}/dashboard/controllerbinding.html"}
def getstate(timeout=3):
"""vrserver's /input/getstate.json: the devices and their input components."""
req = urllib.request.Request(f"{ORIGIN}/input/getstate.json", headers=HEADERS)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.load(resp)
def controllers(timeout=3):
"""{root path: side} for the Frame controllers SteamVR has now."""
return {d["root_path"]: d["side"] for d in getstate(timeout).get("devices", [])
if d.get("controller_type") == "frame_controller" and d.get("root_path")
and d.get("side") in ("left", "right")}
class VrSocket:
def __init__(self, timeout=3):
self.sock = socket.create_connection((HOST, PORT), timeout=timeout)
try:
key = base64.b64encode(os.urandom(16)).decode()
self.sock.sendall((f"GET / HTTP/1.1\r\nHost: {HOST}:{PORT}\r\nUpgrade: websocket\r\n"
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n"
f"Origin: {ORIGIN}\r\nReferer: {HEADERS['Referer']}\r\n\r\n").encode())
head = b""
while b"\r\n\r\n" not in head:
chunk = self.sock.recv(4096)
if not chunk:
raise OSError("vrserver closed the connection during the handshake")
head += chunk
head, rest = head.split(b"\r\n\r\n", 1)
status = head.split(b"\r\n", 1)[0]
if b" 101 " not in status + b" ":
raise OSError(f"vrserver refused the web socket: {status.decode(errors='replace')}")
except BaseException:
self.sock.close()
raise
self.sock.settimeout(None)
self.buf = bytearray(rest)
self.parts = bytearray() # a fragmented message so far
self.mailbox = ""
def fileno(self):
return self.sock.fileno()
def close(self):
try:
self._frame(0x8, b"")
except OSError:
pass
self.sock.close()
def _frame(self, opcode, payload):
"""One frame to vrserver: final, masked (as a client's must be)."""
n = len(payload)
head = bytes([0x80 | opcode])
if n < 126:
head += bytes([0x80 | n])
elif n < 65536:
head += bytes([0x80 | 126]) + struct.pack(">H", n)
else:
head += bytes([0x80 | 127]) + struct.pack(">Q", n)
mask = os.urandom(4)
self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
def send(self, text):
self._frame(0x1, text.encode())
def open(self, mailbox):
self.send(f"mailbox_open {mailbox}")
self.mailbox = mailbox
def subscribe(self, path, on=True):
kind = "request_input_state_updates" if on else "cancel_input_state_updates"
self.send("mailbox_send input_server " +
json.dumps({"type": kind, "device_path": path, "returnAddress": self.mailbox}))
def messages(self, timeout=None):
"""The text messages that came in, as strings: waits up to `timeout` seconds (None:
forever) for some, [] at the timeout. Raises OSError when the connection ends."""
out = self._parse()
if out:
return out
if not select.select([self.sock], [], [], timeout)[0]:
return []
chunk = self.sock.recv(65536)
if not chunk:
raise OSError("vrserver closed the web socket")
self.buf += chunk
return self._parse()
def _parse(self):
"""Every whole frame in the buffer; a partial one waits for the rest."""
out, buf, pos = [], self.buf, 0
while len(buf) - pos >= 2:
b0, b1 = buf[pos], buf[pos + 1]
n, head = b1 & 0x7F, 2
if n == 126:
if len(buf) - pos < 4:
break
n, head = struct.unpack_from(">H", buf, pos + 2)[0], 4
elif n == 127:
if len(buf) - pos < 10:
break
n, head = struct.unpack_from(">Q", buf, pos + 2)[0], 10
masked = b1 & 0x80
if masked:
head += 4
if len(buf) - pos < head + n:
break
data = bytes(buf[pos + head:pos + head + n])
if masked: # servers don't mask, but nothing says they can't
mask = buf[pos + head - 4:pos + head]
data = bytes(b ^ mask[i % 4] for i, b in enumerate(data))
pos += head + n
opcode = b0 & 0x0F
if opcode == 0x8:
raise OSError("vrserver closed the web socket")
if opcode == 0x9:
self._frame(0xA, data) # ping: pong
elif opcode in (0x0, 0x1, 0x2):
self.parts += data
if b0 & 0x80:
out.append(self.parts.decode(errors="replace"))
self.parts = bytearray()
del buf[:pos]
return out
def main():
seconds = float(sys.argv[1]) if len(sys.argv) > 1 else 10.0
ws = VrSocket()
ws.open(f"frametop_vrws_{os.getpid()}")
sides = controllers()
for path, side in sides.items():
print(f"{side} controller: {path}", flush=True)
ws.subscribe(path)
if not sides:
print("no Frame controllers", flush=True)
start = time.monotonic()
count = 0
try:
while time.monotonic() - start < seconds:
for text in ws.messages(timeout=0.5):
count += 1
if "/click" not in text:
continue
msg = json.loads(text)
side = sides.get(msg.get("device"), msg.get("device"))
for name, value in sorted(msg.get("components", {}).items()):
if name.endswith("/click"):
print(f"{time.monotonic() - start:7.3f} {side} {name[len('/input/'):-len('/click')]} "
f"{'down' if value else 'up'}", flush=True)
finally:
ws.close()
print(f"{count} messages in {seconds:.0f} s")
if __name__ == "__main__":
main()
+2 -1
View File
@@ -75,7 +75,7 @@ step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, b
step "4/10 3D mouse: SteamVR driver"
"$root/pointer/driver/build.sh"
"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
"$root/pointer/driver/install.sh" install
step "5/10 3D mouse: pointer helper service"
"$root/pointer/helper/build.sh"
@@ -93,6 +93,7 @@ step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frame
"$root/remote/install.sh"
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
"$root/scripts/conf-migrate.sh"
step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
gaze=0
+3
View File
@@ -0,0 +1,3 @@
#!/bin/bash
# Hide/Show Screens' own program (see ft-layout-reset).
exec "$(dirname "$(readlink -f "$0")")/ft-layout" toggle "$@"
+4
View File
@@ -0,0 +1,4 @@
#!/bin/bash
# Reset Screen Layout's own program. Steam lists menu entries by program, so entries that
# shared ft-layout (with different arguments) launched as one.
exec "$(dirname "$(readlink -f "$0")")/ft-layout" apply "$@"
+2 -1
View File
@@ -534,7 +534,8 @@ def capture_screens():
# ---------------------------------------------------------------- gamescope (dashboard panels)
def vrcmd(*args, timeout=10):
env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD))
# SteamVR's config folder: in the desktop, XDG_CONFIG_HOME is its own (docs/design.md).
env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD), XDG_CONFIG_HOME=os.path.expanduser("~/.config"))
try:
return subprocess.run([VRCMD, *args], capture_output=True, text=True, timeout=timeout, env=env).stdout
except (OSError, subprocess.TimeoutExpired):
+98 -46
View File
@@ -12,7 +12,7 @@
// aim <yaw_deg> <pitch_deg> absolute direction (yaw 0 = -Z, the SteamVR forward)
// gaze follow the head (origin and direction) again
// distance <metres> cursor distance from the anchor (default 1.5)
// pose <x> <y> <z> <yaw> <pitch> exact pose, sent every frame by the ft-pointer helper
// pose <x> <y> <z> <yaw> <pitch> exact pose, from the ft-pointer helper when it changes (it holds)
// posq <x> <y> <z> <qw> <qx> <qy> <qz> exact pose with a full rotation (tilting a panel while moving it)
// btn <name> <0|1> name: trigger, b, x, system, joystick, a (a = claim the laser, no click)
// scroll <x> <y> joystick deflection -1..1
@@ -224,12 +224,23 @@ public:
pose.result = ok ? TrackingResult_Running_OK : TrackingResult_Uninitialized;
pose.deviceIsConnected = snapshot.visible;
pose_ = pose;
VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t));
// Connected, the pose goes out every frame, as SteamVR expects of a tracked device.
// Disconnected, once: it says the same thing every frame after.
if (snapshot.visible || !reportedOff_) {
VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t));
reportedOff_ = !snapshot.visible;
}
// Inputs only when they change (and all of them the first time).
auto input = VRDriverInput();
for (int i = 0; i < kButtons; ++i) input->UpdateBooleanComponent(buttons_[i], snapshot.buttons[i], 0);
input->UpdateScalarComponent(scrollX_, snapshot.scrollX, 0);
input->UpdateScalarComponent(scrollY_, snapshot.scrollY, 0);
for (int i = 0; i < kButtons; ++i)
if (!inputsSent_ || snapshot.buttons[i] != sentButtons_[i])
input->UpdateBooleanComponent(buttons_[i], sentButtons_[i] = snapshot.buttons[i], 0);
if (!inputsSent_ || snapshot.scrollX != sentScrollX_)
input->UpdateScalarComponent(scrollX_, sentScrollX_ = snapshot.scrollX, 0);
if (!inputsSent_ || snapshot.scrollY != sentScrollY_)
input->UpdateScalarComponent(scrollY_, sentScrollY_ = snapshot.scrollY, 0);
inputsSent_ = true;
}
private:
@@ -239,6 +250,10 @@ private:
int32_t role_ = TrackedControllerRole_RightHand;
bool hinted_ = false; // whether the role hint currently claims role_
DriverPose_t pose_{};
bool reportedOff_ = false; // the disconnected pose has gone out (it isn't repeated)
// What the input components were last set to.
bool inputsSent_ = false, sentButtons_[kButtons] = {};
float sentScrollX_ = 0.f, sentScrollY_ = 0.f;
VRInputComponentHandle_t buttons_[kButtons] = {};
VRInputComponentHandle_t scrollX_ = 0, scrollY_ = 0;
};
@@ -293,65 +308,102 @@ private:
}
}
// Parsed first, then applied under the lock, which RunFrame (vrserver's frame) takes too.
void Handle(const char *cmd) {
std::lock_guard<std::mutex> guard(state_.lock);
enum { kNone, kPosq, kPose, kMove, kRecenter, kAim, kGaze, kHide, kShow, kRole, kBtn, kDistance, kScroll };
int kind = kNone;
char name[32];
float a, b;
int v;
float x, y, z, qw, qx, qy, qz;
if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &x, &y, &z, &qw, &qx, &qy, &qz) == 7) {
float f[7];
int v = 0, role = 0, button = -1;
if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4], &f[5], &f[6]) == 7) {
kind = kPosq;
} else if (std::sscanf(cmd, "pose %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4]) == 5) {
kind = kPose;
} else if (std::sscanf(cmd, "move %f %f", &f[0], &f[1]) == 2) {
kind = kMove;
} else if (std::strncmp(cmd, "recenter", 8) == 0) {
kind = kRecenter;
} else if (std::sscanf(cmd, "aim %f %f", &f[0], &f[1]) == 2) {
kind = kAim;
} else if (std::strncmp(cmd, "gaze", 4) == 0) {
kind = kGaze;
} else if (std::strncmp(cmd, "hide", 4) == 0) {
kind = kHide;
} else if (std::strncmp(cmd, "show", 4) == 0) {
kind = kShow;
} else if (std::sscanf(cmd, "role %31s", name) == 1) {
role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand
: !std::strcmp(name, "right") ? TrackedControllerRole_RightHand
: !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
: 0;
if (role) kind = kRole;
} else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
for (int i = 0; i < kButtons; ++i)
if (std::strcmp(name, kButtonNames[i]) == 0) button = i;
if (button >= 0) kind = kBtn;
} else if (std::sscanf(cmd, "distance %f", &f[0]) == 1) {
kind = kDistance;
} else if (std::sscanf(cmd, "scroll %f %f", &f[0], &f[1]) == 2) {
kind = kScroll;
}
if (kind == kNone) return;
std::lock_guard<std::mutex> guard(state_.lock);
switch (kind) {
case kPosq:
state_.explicitPose = true;
state_.hasQuat = true;
state_.gaze = false;
state_.pos[0] = x;
state_.pos[1] = y;
state_.pos[2] = z;
state_.quat[0] = qw;
state_.quat[1] = qx;
state_.quat[2] = qy;
state_.quat[3] = qz;
} else if (std::sscanf(cmd, "pose %f %f %f %f %f", &x, &y, &z, &a, &b) == 5) {
std::memcpy(state_.pos, f, sizeof state_.pos);
std::memcpy(state_.quat, f + 3, sizeof state_.quat);
break;
case kPose:
state_.explicitPose = true;
state_.hasQuat = false;
state_.gaze = false;
state_.pos[0] = x;
state_.pos[1] = y;
state_.pos[2] = z;
state_.yaw = a;
state_.pitch = b;
} else if (std::sscanf(cmd, "move %f %f", &a, &b) == 2) {
std::memcpy(state_.pos, f, sizeof state_.pos);
state_.yaw = f[3];
state_.pitch = f[4];
break;
case kMove:
state_.explicitPose = false;
if (state_.gaze) state_.recenter = true; // first move starts from the gaze
state_.yaw += a; // wrap by hand: libm remainder() is GLIBC_2.43 in the build container
state_.yaw += f[0]; // wrap by hand: libm remainder() is GLIBC_2.43 in the build container
while (state_.yaw > 180.f) state_.yaw -= 360.f;
while (state_.yaw < -180.f) state_.yaw += 360.f;
state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + b));
} else if (std::strncmp(cmd, "recenter", 8) == 0) {
state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + f[1]));
break;
case kRecenter:
state_.explicitPose = false;
state_.recenter = true;
} else if (std::sscanf(cmd, "aim %f %f", &a, &b) == 2) {
break;
case kAim:
state_.gaze = false;
state_.yaw = a;
state_.pitch = b;
} else if (std::strncmp(cmd, "gaze", 4) == 0) {
state_.yaw = f[0];
state_.pitch = f[1];
break;
case kGaze:
state_.gaze = true;
} else if (std::strncmp(cmd, "hide", 4) == 0) {
break;
case kHide:
state_.visible = false;
} else if (std::strncmp(cmd, "show", 4) == 0) {
break;
case kShow:
state_.visible = true;
} else if (std::sscanf(cmd, "role %31s", name) == 1) {
state_.role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand
: !std::strcmp(name, "right") ? TrackedControllerRole_RightHand
: !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
: state_.role;
} else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
for (int i = 0; i < kButtons; ++i)
if (std::strcmp(name, kButtonNames[i]) == 0) state_.buttons[i] = v != 0;
} else if (std::sscanf(cmd, "distance %f", &a) == 1) {
state_.distance = std::fmax(0.3f, std::fmin(10.f, a));
} else if (std::sscanf(cmd, "scroll %f %f", &a, &b) == 2) {
state_.scrollX = a;
state_.scrollY = b;
break;
case kRole:
state_.role = role;
break;
case kBtn:
state_.buttons[button] = v != 0;
break;
case kDistance:
state_.distance = std::fmax(0.3f, std::fmin(10.f, f[0]));
break;
case kScroll:
state_.scrollX = f[0];
state_.scrollY = f[1];
break;
}
}
+21 -5
View File
@@ -14,6 +14,10 @@ frame="$root/scripts/frame.sh"
src=$FRAME_REPO/pointer/driver
dest='$HOME/.local/share/frametop/ft_pointer' # expanded on the Frame, in the commands below
reg='/opt/steamvr/bin/linuxarm64/vrpathreg'
# SteamVR's tools with SteamVR's config folder and libraries. A terminal in the Frametop desktop
# has XDG_CONFIG_HOME=~/.config/frametop (session/frametop-session.sh), where vrpathreg finds no
# registry, so it made one there that SteamVR never reads.
steamvr='env XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64' # expanded on the Frame
case ${1:-install} in
install)
@@ -21,6 +25,11 @@ case ${1:-install} in
# Replacing the files of a driver SteamVR has loaded leaves its input bindings in a bad
# state (the 3D mouse no longer gets the laser) until SteamVR restarts, so an unchanged
# driver is left alone.
# A registry an earlier install made in the desktop's config folder (see steamvr above) has
# no SteamVR in it, and hid SteamVR from OpenVR programs started in the desktop
# (VRInitError_Init_InstallationNotFound): it goes.
# vrpathreg runs 'xdg-open vrmonitor://driverinstalled' for SteamVR's desktop monitor, which
# the Frame doesn't have, so the desktop said "could not read file": a stand-in takes it.
"$frame" --host "set -e; test -f $src/build/driver_ft_pointer.so
rm -rf $dest.new; mkdir -p $dest.new/bin/linuxarm64
cp -r $src/ft_pointer/. $dest.new/
@@ -31,15 +40,22 @@ else
rm -rf $dest; mv $dest.new $dest
echo 'installed; restart SteamVR to load it'
fi
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg adddriver $dest
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg show | grep -A3 -i 'external'" ;;
stray=\$HOME/.config/frametop/openvr/openvrpaths.vrpath
if grep -qs '\"jsonid\" : \"vrpathreg\"' \$stray && grep -qsE '\"runtime\"[[:space:]]*:[[:space:]]*null' \$stray; then
rm -f \$stray; rmdir \$HOME/.config/frametop/openvr 2>/dev/null || true
echo 'removed a SteamVR path registry an earlier install left in ~/.config/frametop/openvr'
fi
noopen=\$(mktemp -d); trap 'rm -rf \$noopen' EXIT
printf '#!/bin/sh\nexit 0\n' > \$noopen/xdg-open; chmod +x \$noopen/xdg-open
PATH=\$noopen:\$PATH $steamvr $reg adddriver $dest
$steamvr $reg show | grep -A3 -i 'external'" ;;
uninstall)
"$frame" --host "LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;;
"$frame" --host "$steamvr $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;;
send)
"$frame" --host "python3 -c 'import socket,sys; s=socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM); s.sendto(sys.argv[1].encode(), \"\\0ft_pointer\")' $(printf %q "${2:?command}")" ;;
probe) "$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' ;;
probe) "$frame" -C pointer/probe "$steamvr ./build/vrprobe" ;;
aimhere)
read -r yaw pitch < <("$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p')
read -r yaw pitch < <("$frame" -C pointer/probe "$steamvr ./build/vrprobe" | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p')
[ -n "${yaw:-}" ] || { echo "head pose not valid (headset off?)" >&2; exit 1; }
"$0" send "aim $yaw $pitch" && echo "aimed at yaw $yaw pitch $pitch" ;;
log) "$frame" --host "grep -iE 'ft_pointer' ~/.local/share/Steam/logs/vrserver.txt | tail -n ${2:-20}" ;;
+348 -158
View File
@@ -36,6 +36,22 @@
// pointer is off the helper also stops listing overlays with vrcmd, whose connection every
// second kept SteamVR from going to standby.
//
// Idle: while the pointer is off (and hand gestures are off), the main loop waits for a command
// on its socket for up to 250 ms instead of running every 8 ms, or 20 ms while it reads mapped
// Frame controller buttons (vrbuttons.h), which have no event to wait for. What has to go on
// meanwhile still does: the headset's activity level, the game check and the relay's "vrgame"
// and "gazeawake" repeats, and a mouse command wakes it at once. The overlay lookups (the 50 ms
// visibility poll, the once-a-second handles) wait for the pointer to wake, and run right then.
//
// Unchanged frames: while the pointer is awake, a frame where nothing moved reuses the last one's
// result. The collision passes (ComputeOverlayIntersection on every visible overlay, twice) run
// again when the mouse or the anchor moved, the eye moved 5 mm, an overlay showed or hid, or
// 100 ms have passed (overlays also move by themselves), and always while dragging. The dots'
// setters go to SteamVR only when their value changes, and a dot's placement only when it moved
// 0.2 mm or the eye 5 mm (its facing and size then change by well under 1%). The pose goes to the
// driver, which keeps the last one, when the laser's origin moved 0.2 mm or its direction 0.04
// deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms.
//
// Last used wins: when a real controller moves (picked up), the pointer is released
// (driver "hide", which also drops its hand role hint), so the controller gets
// its role and laser back. The next mouse input reconnects and claims the laser again.
@@ -267,7 +283,8 @@
//
// Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
// follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks), gz ... (the gaze, from ft-gazed),
// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks, and "? headset" adds
// worn|away to the reply), gz ... (the gaze, from ft-gazed),
// reload (re-read the settings below), debug (toggle a twice-a-second state log),
// overlays (replies with the overlay list as JSON, see OverlayList),
// vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
@@ -310,6 +327,7 @@ extern "C" {
#include <atomic>
#include <chrono>
#include <cmath>
#include <condition_variable>
#include <cstdio>
#include <cstdlib>
#include <cstring>
@@ -327,6 +345,7 @@ extern "C" {
#include <fcntl.h>
#include <fnmatch.h>
#include <poll.h>
#include <sys/mman.h>
#include <sys/socket.h>
#include <sys/stat.h>
@@ -499,29 +518,62 @@ std::string JsonQuote(const std::string &s) {
// public call to enumerate other apps' overlays). Hidden ones are listed too: the
// window controls under a floating panel only appear while something hovers the
// panel, and the cursor has to find them the moment they do, not a second later.
// Each vrcmd run is a shell and a new SteamVR client, about 30 ms of CPU, so the list is read
// every 20 seconds (kEvery) while the pointer is awake, and at once when something says it may
// have changed (Kick: the pointer waking, the dashboard opening or closing, a click that hit
// nothing), at most once a second. Showing and hiding the overlays it knows doesn't need a new
// list: the main loop polls their visibility (IsOverlayVisible) every 50 ms.
// Paused while the pointer is off: each vrcmd run connects to SteamVR as a new app, and a new
// app every second kept SteamVR (and the headset's displays) from going to standby.
// "overlays" requests (Frametop Input Settings' Ignored panels page) refresh the list even
// while paused, and are answered from this thread once it's fresh.
class OverlayList {
public:
static constexpr auto kEvery = std::chrono::seconds(20), kGap = std::chrono::seconds(1);
void Start() {
out_ = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
thread_ = std::thread([this] {
std::unique_lock<std::mutex> lk(lock_);
while (running_) {
if (!paused_ || requested_) Refresh();
// Wait a second, or less when the pointer wakes (refresh right away then).
for (int i = 0; i < 10 && running_; ++i) {
const bool wasPaused = paused_;
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if ((wasPaused && !paused_) || requested_) break;
// Sleeps while paused (until a request or the wake); otherwise until it's time, or
// a kick once kGap has passed since the last read.
const auto due = std::max(last_ + kGap, kicked_ ? last_ : last_ + kEvery);
if (!waiting_.empty()) {
} else if (paused_) {
cv_.wait(lk);
continue;
} else if (std::chrono::steady_clock::now() < due) {
cv_.wait_until(lk, due);
continue;
}
kicked_ = false;
lk.unlock();
Refresh();
lk.lock();
last_ = std::chrono::steady_clock::now();
}
});
}
void SetPaused(bool paused) { paused_ = paused; }
// Unpausing (the pointer woke) reads the list again at once.
void SetPaused(bool paused) {
std::lock_guard<std::mutex> guard(lock_);
if (paused == paused_) return;
if (!paused) kicked_ = true, last_ = {};
paused_ = paused;
cv_.notify_one();
}
// The list may have changed: read it again soon (not while paused).
void Kick() {
std::lock_guard<std::mutex> guard(lock_);
kicked_ = true;
cv_.notify_one();
}
void Stop() {
running_ = false;
{
std::lock_guard<std::mutex> guard(lock_);
running_ = false;
cv_.notify_one();
}
if (thread_.joinable()) thread_.join();
}
std::vector<std::string> Keys() {
@@ -530,12 +582,25 @@ public:
for (const auto &e : entries_) keys.push_back(e.key);
return keys;
}
// A panel just made by another Frametop program (ft-screens: a floating window's menu, the
// keyboard): known at once, without waiting for the next read ("overlay <key>").
void Add(const std::string &key) {
{
std::lock_guard<std::mutex> guard(lock_);
for (const auto &e : entries_)
if (e.key == key) return;
entries_.push_back({key, key, true});
}
++generation_;
}
// Goes up by one each time the list is read, so the main loop knows to look the keys up again.
unsigned Generation() const { return generation_; }
// Answer `to` with {"t":"overlays","list":[{"key","name","visible"}...]} after the next refresh.
void Request(const sockaddr_un &to, socklen_t len) {
if (len <= offsetof(sockaddr_un, sun_path)) return;
std::lock_guard<std::mutex> guard(lock_);
if (waiting_.size() < 8) waiting_.push_back({to, len});
requested_ = true;
cv_.notify_one();
}
private:
@@ -544,7 +609,6 @@ private:
bool visible;
};
void Refresh() {
requested_ = false;
FILE *p = popen("LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null", "r");
if (!p) return;
std::vector<Entry> entries;
@@ -576,6 +640,7 @@ private:
entries_ = std::move(entries);
waiting.swap(waiting_);
}
++generation_;
if (waiting.empty()) return;
std::string msg = "{\"t\":\"overlays\",\"list\":[";
for (size_t i = 0; i < entries_.size(); ++i)
@@ -588,10 +653,11 @@ private:
std::thread thread_;
int out_ = -1;
std::atomic<bool> paused_{false};
std::atomic<bool> running_{true};
std::atomic<bool> requested_{false};
std::mutex lock_;
std::mutex lock_; // guards everything below
std::condition_variable cv_;
bool paused_ = false, running_ = true, kicked_ = false;
std::chrono::steady_clock::time_point last_{}; // the last read
std::atomic<unsigned> generation_{0};
std::vector<Entry> entries_; // written only by the thread; the lock guards readers
std::vector<std::pair<sockaddr_un, socklen_t>> waiting_;
};
@@ -619,6 +685,36 @@ bool Ignored(const std::vector<std::string> &patterns, const std::string &key) {
return false;
}
vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye);
// One of our dots, with what was last set on it: each setter goes to SteamVR only when its value
// changes (see "Unchanged frames" at the top).
struct DotOverlay {
vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
int shown = -1; // unknown at first
float alpha = -1, width = -1;
bool placed = false;
Vec3 at, eye;
void Show(bool on) {
if (shown == int(on)) return;
shown = on;
if (on) vr::VROverlay()->ShowOverlay(h);
else vr::VROverlay()->HideOverlay(h);
}
void Alpha(float a) {
if (a != alpha) vr::VROverlay()->SetOverlayAlpha(h, alpha = a);
}
// Width w, at `to`, facing `from`: left as it is for a change under 0.5% in width, and while
// it moved less than 0.2 mm and the eye less than 5 mm.
void Place(Vec3 to, Vec3 from, float w) {
if (std::fabs(w - width) > 0.005f * width) vr::VROverlay()->SetOverlayWidthInMeters(h, width = w);
if (placed && Length(to - at) < 0.0002 && Length(from - eye) < 0.005) return;
placed = true, at = to, eye = from;
const auto m = Billboard(to, from);
vr::VROverlay()->SetOverlayTransformAbsolute(h, vr::TrackingUniverseStanding, &m);
}
};
vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye) {
// Overlay faces +Z; point +Z at the eye, keep +Y roughly up.
const Vec3 z = Normalize(eye - at);
@@ -802,6 +898,8 @@ int main() {
below.m[1][3] = -50;
overlay->SetOverlayTransformTrackedDeviceRelative(laserMode, vr::k_unTrackedDeviceIndex_Hmd, &below);
bool laserModeShown = false;
DotOverlay cursorDot, markerDot; // the setters for `cursor` and `marker` (see "Unchanged frames" at the top)
cursorDot.h = cursor, markerDot.h = marker;
// Controller beams keep the user's width; nothing here changes it any more.
vr::VRSettings()->SetFloat("dashboard", "laserRayWidthScale", laserWidth);
@@ -821,12 +919,34 @@ int main() {
std::map<std::string, bool> sceneGraph; // no texture: plane test instead of ComputeOverlayIntersection
std::map<std::string, bool> visible; // refreshed every 50 ms
auto lastVisible = std::chrono::steady_clock::now();
auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
unsigned listGeneration = 0; // the overlay list the handles were looked up from
// The plane of the last panel the cursor was on, and the last point on it (panel edges).
Vec3 edgePoint, edgeNormal, edgeLast;
std::string edgeKey;
bool active = false, recenter = false, anchored = false;
using Clock = std::chrono::steady_clock;
// Unchanged frames (see the top): visibleGen goes up when an overlay shows or hides or the
// handles change; pass is the last collision pass, posed the last plain pose sent.
unsigned visibleGen = 0;
struct Pass {
bool valid = false;
Clock::time_point at{};
unsigned gen = 0;
double yaw = 0, pitch = 0;
Vec3 anchor, eye;
double best = 1e9, distance = 0;
std::string key;
bool scene = false, onEdge = false, occluded = false;
Vec3 point;
} pass;
struct Posed {
bool valid = false;
Clock::time_point at{};
Vec3 origin;
double yaw = 0, pitch = 0;
} posed;
Clock::time_point lastMouse{}, claimAt{}, claimRelease{}, wokeAt{}, noWakeUntil{};
bool claimPending = false, claimHeld = false;
// Last used wins: since when each controller has been moving (zero: it isn't).
@@ -863,6 +983,9 @@ int main() {
wokeAt = t;
active = true;
recenter = true;
// Overlay handles and visibility weren't looked at while it was off (see "Idle" at the top).
lastSlow = lastVisible = t - std::chrono::seconds(10);
pass.valid = posed.valid = false;
SendTo(out, "ft_pointer", "role " + role); // POINTER_ROLE, before it takes it
SendTo(out, "ft_pointer", "show");
claimPending = true; // take the laser without clicking, once SteamVR has bound the device
@@ -931,8 +1054,8 @@ int main() {
Clock::time_point handUsed{}; // a gesture began then (keeps the pointer, like gaze mode)
Clock::time_point lastTyping{}; // the relay's last "typing": a key on a keyboard
// Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
bool inGame = false, gazeAwake = false;
Clock::time_point inGameAt{}, gazeAwakeAt{};
bool inGame = false, gazeAwake = false, gameSent = false;
Clock::time_point inGameAt{}, gazeAwakeAt{}, gameSentAt{};
Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
// The left button, as sent to the driver; pressRight: the next press is the right button
// instead (gaze_right), heldButton: the one pressed.
@@ -964,6 +1087,8 @@ int main() {
if (debug) std::fflush(stdout);
}
nudging = confirmLesson = false;
// A click on nothing: maybe on an overlay that came up since the list was read.
if (lastHit.empty()) overlays.Kick();
leftHeld = true;
dragDistance = lastDistance;
pressKey.clear();
@@ -1087,7 +1212,6 @@ int main() {
return gazeOn && gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) &&
!aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease;
};
auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
// --- Panel placement (see "Placement" at the top of the file) ---
// Device pose, given in the standing universe, sent to the driver in raw space.
@@ -1103,6 +1227,7 @@ int main() {
char msg[200];
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", o.x, o.y, o.z, q[0], q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
posed.valid = false;
};
auto sleepMs = [](int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); };
auto headPos = [&] {
@@ -1114,8 +1239,8 @@ int main() {
auto borrow = [&] {
SendTo(out, "ft_pointer", "show");
overlay->ShowOverlay(laserMode);
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
sleepMs(active ? 50 : 400); // a fresh connect needs SteamVR to bind the device
SendTo(out, "ft_pointer", "btn a 1");
sleepMs(60);
@@ -1242,6 +1367,36 @@ int main() {
return msg;
};
// "move <dyaw> <dpitch>" from the relay.
auto mouseMove = [&](double a, double b) {
// A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
// the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
const bool heldBack = mouseMoveHeld();
if (!heldBack) lastMouse = Clock::now();
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
if (!active) wake(Clock::now());
if (heldBack) return;
if (tilting) {
tiltYaw += a;
tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
return;
}
lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode)
if (gazeOn && gazeOwns) {
// The mouse takes the pointer from the gaze, from where the gaze left it.
gazeOwns = false;
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
nudgeAt = Clock::now(), nudgeMoved = 0;
}
if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
if (!anchored) recenter = true;
yaw += a;
while (yaw > 180) yaw -= 360;
while (yaw < -180) yaw += 360;
pitch = std::clamp(pitch + b, -85.0, 85.0);
};
std::printf("ft-pointer running: free distance %.2f m, dot %.2f deg\n", freeDistance, cursorDeg);
std::fflush(stdout);
@@ -1256,8 +1411,8 @@ int main() {
if (headsetOff && active) {
active = false;
claimPending = claimHeld = false;
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
SendTo(out, "ft_pointer", "btn a 0");
SendTo(out, "ft_pointer", "hide");
std::printf("headset off: pointer released\n");
@@ -1283,6 +1438,13 @@ int main() {
gazeAwakeAt = t;
SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
}
// The relay pauses Frametop for VR games (input/game_pause.py). Repeated, so a relay
// that restarts learns it, and silence (SteamVR gone) ends the game there.
if (inGame != gameSent || t - gameSentAt > std::chrono::seconds(5)) {
gameSent = inGame;
gameSentAt = t;
SendTo(out, "frametop_relay", inGame ? "vrgame 1" : "vrgame 0");
}
}
// Commands from the relay.
@@ -1300,6 +1462,12 @@ int main() {
if (senderLen > offsetof(sockaddr_un, sun_path))
sendto(out, msg.data(), msg.size(), 0, reinterpret_cast<const sockaddr *>(&sender), senderLen);
};
// Mouse moves first, the most frequent command.
double mx, my;
if (std::strncmp(buf, "move ", 5) == 0 && std::sscanf(buf + 5, "%lf %lf", &mx, &my) == 2) {
mouseMove(mx, my);
continue;
}
// The gaze, from ft-gazed: not mouse input, it never wakes the pointer.
double g[4];
if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) {
@@ -1365,6 +1533,10 @@ int main() {
reply(controllerButtons.Status());
continue;
}
if (std::strncmp(buf, "overlay ", 8) == 0 && std::strncmp(buf + 8, "frametop.", 9) == 0) {
overlays.Add(buf + 8);
continue;
}
if (std::strncmp(buf, "overlays", 8) == 0) {
overlays.Request(sender, senderLen); // answered from the list's thread
continue;
@@ -1380,19 +1552,19 @@ int main() {
std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
std::fflush(stdout);
}
reply(gazeOn ? "ok on" : "ok off");
// "gaze ? headset" (the gaze service, which idles while nobody wears the headset)
// also says whether someone does.
if (std::strstr(arg, "headset"))
reply(gazeOn ? (headsetOff ? "ok on away" : "ok on worn") : (headsetOff ? "ok off away" : "ok off worn"));
else
reply(gazeOn ? "ok on" : "ok off");
continue;
}
const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 ||
std::strncmp(buf, "scroll", 6) == 0;
// A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
// the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
const bool moveHeldBack = std::strncmp(buf, "move", 4) == 0 && mouseMoveHeld();
if (mouseInput && !moveHeldBack) lastMouse = Clock::now();
// (Moves were taken first, above.)
const bool mouseInput = std::strncmp(buf, "btn", 3) == 0 || std::strncmp(buf, "scroll", 6) == 0;
if (mouseInput) lastMouse = Clock::now();
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
if (!active && mouseInput) wake(Clock::now());
if (moveHeldBack) continue;
double a, b;
char key[128];
double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1;
if (std::sscanf(buf, "grabprobe %127s", key) == 1) {
@@ -1451,31 +1623,12 @@ int main() {
tilting = swallowedRight = false;
continue;
}
if (tilting && std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
tiltYaw += a;
tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
continue;
}
if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode)
if (gazeOn && gazeOwns) {
// The mouse takes the pointer from the gaze, from where the gaze left it.
gazeOwns = false;
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
nudgeAt = Clock::now(), nudgeMoved = 0;
}
if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
if (!anchored) recenter = true;
yaw += a;
while (yaw > 180) yaw -= 360;
while (yaw < -180) yaw += 360;
pitch = std::clamp(pitch + b, -85.0, 85.0);
} else if (std::strncmp(buf, "recenter", 8) == 0) {
if (std::strncmp(buf, "recenter", 8) == 0) {
recenter = true;
} else if (std::strncmp(buf, "reload", 6) == 0) {
loadConfig();
lastSlow = Clock::now() - std::chrono::seconds(10); // apply POINTER_IGNORE now
pass.valid = false;
std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg, "
"controller pickup %.1fx, %zu ignored\n",
freeDistance, cursorDeg, originFraction, follow ? "on" : "off", leashDeg, pickupScale,
@@ -1493,8 +1646,8 @@ int main() {
if (!active) wake(Clock::now());
} else if (std::strncmp(buf, "hide", 4) == 0) {
active = false;
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
SendTo(out, "ft_pointer", "hide");
} else {
SendTo(out, "ft_pointer", buf); // btn, scroll
@@ -1536,8 +1689,8 @@ int main() {
sys->GetControllerRoleForTrackedDeviceIndex(ours) == vr::TrackedControllerRole_Invalid) {
active = false;
claimPending = claimHeld = false;
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
SendTo(out, "ft_pointer", "btn a 0");
SendTo(out, "ft_pointer", "hide");
noWakeUntil = tnow + std::chrono::seconds(2);
@@ -1565,8 +1718,8 @@ int main() {
if (tnow - movingSince[i] >= std::chrono::milliseconds(100)) {
active = false;
claimPending = claimHeld = false;
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
SendTo(out, "ft_pointer", "btn a 0");
SendTo(out, "ft_pointer", "hide");
std::printf("controller %u moved (%.2f m/s, %.1f rad/s): pointer released\n", i, speed, spin);
@@ -1616,7 +1769,7 @@ int main() {
}
// Hands (see the top): pinches and grips from ft-hands.
{
if (handsOn) {
vr::TrackedDevicePose_t h0;
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h0, 1);
if (h0.bPoseIsValid) poses.Add(tnow, h0.mDeviceToAbsoluteTracking);
@@ -1915,10 +2068,13 @@ int main() {
}
followAt = tnow;
// Slow work, once a second: overlay handles, our device index, laser width.
// Slow work, once a second and when the overlay list is new: overlay handles, our device
// index. Only while the pointer is awake (see "Idle" at the top).
const auto now = std::chrono::steady_clock::now();
if (now - lastSlow > std::chrono::seconds(1)) {
if (active && (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration)) {
lastSlow = now;
listGeneration = overlays.Generation();
const auto before = handles;
handles.clear();
for (const auto &key : overlays.Keys()) {
if (Ignored(ignore, key)) continue;
@@ -1938,19 +2094,21 @@ int main() {
sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute &&
key.rfind("frametop.", 0) != 0;
}
ours = vr::k_unTrackedDeviceIndexInvalid;
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
if (handles != before) ++visibleGen;
// Our device keeps its index; it's looked for again when a device is activated or
// deactivated (the events below).
for (vr::TrackedDeviceIndex_t i = 0; ours == vr::k_unTrackedDeviceIndexInvalid && i < vr::k_unMaxTrackedDeviceCount; ++i) {
char type[64] = "";
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
if (std::strcmp(type, "ft_pointer") == 0) ours = i;
}
}
if (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size()) {
if (active && (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size())) {
lastVisible = now;
visible.clear();
for (const auto &[key, h] : handles) visible[key] = overlay->IsOverlayVisible(h);
std::map<std::string, bool> seen;
for (const auto &[key, h] : handles) seen[key] = overlay->IsOverlayVisible(h);
if (seen != visible) visible.swap(seen), ++visibleGen;
}
if (active && anchored && hmd.bPoseIsValid && tilting) {
@@ -1960,8 +2118,8 @@ int main() {
tiltOrigin = lastOrigin;
tiltBasis = AimBasis(lastAim);
tiltStart = false; // angles carry on from any earlier tilt in this drag
overlay->HideOverlay(cursor);
overlay->HideOverlay(marker);
cursorDot.Show(false);
markerDot.Show(false);
}
const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
const Vec3 up{0, 1, 0}, side = tiltBasis.x;
@@ -1977,6 +2135,7 @@ int main() {
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", originRaw.x, originRaw.y,
originRaw.z, q[0], q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
posed.valid = false;
} else if (active && anchored && hmd.bPoseIsValid) {
const bool dragging = leftHeld || tnow < dropHoldUntil;
if (!dragging) tiltYaw = tiltPitch = 0; // drop finished: back to plain pointing
@@ -2020,85 +2179,98 @@ int main() {
}
return h;
};
Hit first;
if (!dragging) first = nearest(anchor, dir);
double best = first.along;
std::string bestKey = first.key;
bool bestScene = first.scene;
Vec3 bestPoint = first.point, bestNormal = first.normal;
bool onEdge = false;
if (!dragging && best < 1e8 && !bestScene) {
edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
} else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) {
// Just off a panel: stay on its plane (see "Panel edges" at the top).
const double denom = Dot(dir, edgeNormal);
if (std::fabs(denom) > 1e-4) {
const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
const Vec3 at = anchor + dir * along;
if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) {
best = along;
bestKey = edgeKey;
onEdge = true;
// Unchanged frames (see the top): the last pass still holds.
const bool reuse = !dragging && pass.valid && tnow - pass.at < std::chrono::milliseconds(100) &&
pass.gen == visibleGen && std::fabs(yaw - pass.yaw) < 1e-6 &&
std::fabs(pitch - pass.pitch) < 1e-6 && Length(anchor - pass.anchor) < 0.001 &&
Length(eye - pass.eye) < 0.005;
double best = pass.best, distance = pass.distance;
std::string bestKey = pass.key;
bool bestScene = pass.scene, onEdge = pass.onEdge, occluded = pass.occluded;
Vec3 point = pass.point;
if (!reuse) {
Hit first;
if (!dragging) first = nearest(anchor, dir);
best = first.along;
bestKey = first.key;
bestScene = first.scene;
Vec3 bestPoint = first.point, bestNormal = first.normal;
onEdge = false;
if (!dragging && best < 1e8 && !bestScene) {
edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
} else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible.count(edgeKey) &&
visible.at(edgeKey)) {
// Just off a panel: stay on its plane (see "Panel edges" at the top).
const double denom = Dot(dir, edgeNormal);
if (std::fabs(denom) > 1e-4) {
const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
const Vec3 at = anchor + dir * along;
if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) {
best = along;
bestKey = edgeKey;
onEdge = true;
}
}
}
}
// Held on one of ft-screens' panels that stays where it is: onto whichever of them
// the ray meets (see the top).
if (leftHeld && !pressKey.empty()) {
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t now{};
auto it = handles.find(pressKey);
bool still = it != handles.end() &&
overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
for (int i = 0; still && i < 3; ++i)
for (int j = 0; j < 4; ++j)
if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
if (still) {
Hit h;
for (const auto &[key, handle] : handles) {
if (!visible[key] || !FramePanel(key)) continue;
vr::VROverlayIntersectionParams_t params{};
params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
params.eOrigin = vr::TrackingUniverseStanding;
vr::VROverlayIntersectionResults_t r{};
if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
r.fDistance < h.along)
h.along = r.fDistance, h.key = key;
// Held on one of ft-screens' panels that stays where it is: onto whichever of them
// the ray meets (see the top).
if (leftHeld && !pressKey.empty()) {
vr::ETrackingUniverseOrigin uo;
vr::HmdMatrix34_t now{};
auto it = handles.find(pressKey);
bool still = it != handles.end() &&
overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
for (int i = 0; still && i < 3; ++i)
for (int j = 0; j < 4; ++j)
if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
if (still) {
Hit h;
for (const auto &[key, handle] : handles) {
if (!visible[key] || !FramePanel(key)) continue;
vr::VROverlayIntersectionParams_t params{};
params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
params.eOrigin = vr::TrackingUniverseStanding;
vr::VROverlayIntersectionResults_t r{};
if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
r.fDistance < h.along)
h.along = r.fDistance, h.key = key;
}
if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
} else {
pressKey.clear(); // carried: the lock holds for the rest of this press
}
if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
} else {
pressKey.clear(); // carried: the lock holds for the rest of this press
}
}
// While dragging: keep the press-time distance and show the non-interactive marker.
// On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
double distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance);
Vec3 point = anchor + dir * distance;
bool occluded = false;
if (!dragging) {
// The cursor lands on what you see under it: the ray above starts at the anchor,
// not the eye, so after leaning it can pick a panel that something nearer
// covers from where you are now (panels close together in view, at different
// depths). Anything in front of the point on the eye's line of sight wins.
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
const Hit front = nearest(eye, Normalize(point - eye));
if (front.along < toPoint - 0.02) {
occluded = true;
bestKey = front.key, bestScene = front.scene;
point = front.point;
if (!front.scene) edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
edgeLast = front.point;
distance = std::sqrt(Dot(point - anchor, point - anchor));
best = distance;
onEdge = false;
// While dragging: keep the press-time distance and show the non-interactive marker.
// On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance);
point = anchor + dir * distance;
occluded = false;
if (!dragging) {
// The cursor lands on what you see under it: the ray above starts at the anchor,
// not the eye, so after leaning it can pick a panel that something nearer
// covers from where you are now (panels close together in view, at different
// depths). Anything in front of the point on the eye's line of sight wins.
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
const Hit front = nearest(eye, Normalize(point - eye));
if (front.along < toPoint - 0.02) {
occluded = true;
bestKey = front.key, bestScene = front.scene;
point = front.point;
if (!front.scene)
edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
edgeLast = front.point;
distance = std::sqrt(Dot(point - anchor, point - anchor));
best = distance;
onEdge = false;
}
lastDistance = distance, lastHit = bestKey;
}
lastDistance = distance, lastHit = bestKey;
}
const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge);
const bool onPanel = dragging || (best < 1e8 && !onScene);
const Vec3 sight = Normalize(point - eye);
if (!dragging) {
if (!dragging && !reuse) {
// SteamVR Settings (see the top): the dashboard's main panel is hidden and its
// scene-graph panel shows the page.
onVrSettings = false;
@@ -2114,6 +2286,10 @@ int main() {
}
}
}
if (!reuse) {
pass = {!dragging, tnow, visibleGen, yaw, pitch, anchor, eye,
best, distance, bestKey, bestScene, onEdge, occluded, point};
}
if (onVrSettings != catcherHidesHit) {
overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings);
catcherHidesHit = onVrSettings;
@@ -2130,20 +2306,16 @@ int main() {
alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
}
if (tnow < calPanelUntil) alpha = 0;
overlay->SetOverlayAlpha(marker, float(alpha));
overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
auto mm = Billboard(near, eye);
overlay->SetOverlayTransformAbsolute(marker, vr::TrackingUniverseStanding, &mm);
overlay->SetOverlayAlpha(cursor, 0);
overlay->SetOverlayWidthInMeters(cursor, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
auto mc = Billboard(far, eye);
overlay->SetOverlayTransformAbsolute(cursor, vr::TrackingUniverseStanding, &mc);
overlay->ShowOverlay(marker);
overlay->ShowOverlay(cursor);
markerDot.Alpha(float(alpha));
markerDot.Place(near, eye, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
cursorDot.Alpha(0);
cursorDot.Place(far, eye, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
markerDot.Show(true);
cursorDot.Show(true);
} else {
// On a panel: the non-interactive marker, pulled 5 mm toward the eye so it
// draws on top. In free space: the interactive dot the laser lands on.
const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
DotOverlay &show = onPanel ? markerDot : cursorDot, &hide = onPanel ? cursorDot : markerDot;
const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
const double dist = std::sqrt(Dot(at - eye, at - eye));
// Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only
@@ -2160,12 +2332,10 @@ int main() {
}
}
if (tnow < calPanelUntil) alpha = 0; // the calibration panel is up (see the top)
overlay->SetOverlayAlpha(show, float(alpha));
overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
auto m = Billboard(at, eye);
overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m);
overlay->ShowOverlay(show);
overlay->HideOverlay(hide);
show.Alpha(float(alpha));
show.Place(at, eye, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
show.Show(true);
hide.Show(false);
}
// Controller ray: from the eye, aimed at the cursor point, converted from the
@@ -2210,10 +2380,15 @@ int main() {
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", oRaw.x, oRaw.y, oRaw.z, q[0],
q[1], q[2], q[3]);
SendTo(out, "ft_pointer", msg);
} else {
posed.valid = false;
} else if (!posed.valid || tnow - posed.at >= std::chrono::milliseconds(100) ||
Length(eyeRaw - posed.origin) >= 0.0002 || std::fabs(std::remainder(ayaw - posed.yaw, 360.0)) >= 0.04 ||
std::fabs(apitch - posed.pitch) >= 0.04) {
// The driver keeps the last pose: only a change goes out (see "Unchanged frames" at the top).
char msg[160];
std::snprintf(msg, sizeof msg, "pose %.5f %.5f %.5f %.4f %.4f", eyeRaw.x, eyeRaw.y, eyeRaw.z, ayaw, apitch);
SendTo(out, "ft_pointer", msg);
posed = {true, tnow, eyeRaw, ayaw, apitch};
}
}
@@ -2262,6 +2437,12 @@ int main() {
vr::VREvent_t ev;
while (sys->PollNextEvent(&ev, sizeof ev)) {
// The dashboard's overlays come and go with it, and a game brings its own.
if (ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated ||
ev.eventType == vr::VREvent_DashboardOverlayCreated || ev.eventType == vr::VREvent_SceneApplicationChanged)
overlays.Kick();
if (ev.eventType == vr::VREvent_TrackedDeviceActivated || ev.eventType == vr::VREvent_TrackedDeviceDeactivated)
ours = vr::k_unTrackedDeviceIndexInvalid;
if (ev.eventType == vr::VREvent_Quit) {
sys->AcknowledgeQuit_Exiting();
@@ -2270,6 +2451,15 @@ int main() {
return 0;
}
}
std::this_thread::sleep_for(std::chrono::milliseconds(8));
// Idle (see the top): wait for a command instead. Not with anything still to finish (the
// claim pulse, a click's release, a press).
const bool idle = !active && !handsOn && !claimPending && !claimHeld && !clickRelease && !leftHeld &&
hold.src == Src::None;
if (idle) {
pollfd p{in, POLLIN, 0};
poll(&p, 1, controllerButtons.Watching(inGame) ? 20 : 250);
} else {
std::this_thread::sleep_for(std::chrono::milliseconds(8));
}
}
}
+8
View File
@@ -111,6 +111,14 @@ class ControllerButtons {
}
}
// Whether Poll reads any button now (the main loop can't sleep long while it does).
bool Watching(bool inGame) const {
if (!ok_ || (inGame && !games_ && !capture_)) return false;
for (int i = 0; i < kCount; ++i)
if (bound_[i]) return true;
return false;
}
// {"manifest":true,"global":false,"bound":[...],"active":[...]}: active = bound and
// delivered (a controller that has the button is on, and SteamVR lets us have it).
std::string Status() const {
+72 -64
View File
@@ -306,14 +306,19 @@ int main() {
std::printf("displays on: %s\n", why.c_str());
};
// SteamVR is asked every 100 ms (events, the activity level, poses: IPC calls to
// vrserver). Input wakes the loop too, to wake the displays at once, but a moving mouse
// sends hundreds of events a second, so those wakes only drain the devices.
constexpr auto kTick = std::chrono::milliseconds(100);
auto lastVr = now - kTick;
std::vector<pollfd> fds;
while (!stopSignal) {
// Sleep until input or 100 ms: input wakes the displays at once, poses every tick.
fds.clear();
fds.push_back({ctl, POLLIN, 0});
for (auto &[path, d] : inputs)
if (d.fd >= 0) fds.push_back({d.fd, POLLIN, 0});
poll(fds.data(), fds.size(), 100);
const auto wait = std::chrono::ceil<std::chrono::milliseconds>(kTick - (Clock::now() - lastVr));
poll(fds.data(), fds.size(), std::clamp<int>(wait.count(), 0, kTick.count()));
now = Clock::now();
const bool grace = now < graceUntil;
std::string use; // why this tick counts as use
@@ -329,71 +334,74 @@ int main() {
lastConf = now;
}
vr::VREvent_t ev;
while (sys->PollNextEvent(&ev, sizeof ev)) {
if (ev.eventType == vr::VREvent_Quit) {
stopSignal = SIGTERM;
sys->AcknowledgeQuit_Exiting();
} else if (ev.eventType == vr::VREvent_TrackedDeviceActivated ||
ev.eventType == vr::VREvent_TrackedDeviceDeactivated) {
lastClasses = now - std::chrono::hours(1);
}
}
if (now - lastClasses > std::chrono::seconds(5)) {
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i)
classes[i] = sys->GetTrackedDeviceClass(i);
lastClasses = now;
}
// SteamVR's standby (headset taken off) turns the displays off itself; putting the
// headset on again (or the mount covering the sensor again) counts as use.
// A change counts once it has held for a second (the first reading after connecting
// is Idle).
const auto level = sys->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
const bool awayNow = level == vr::k_EDeviceActivityLevel_Idle || level == vr::k_EDeviceActivityLevel_Standby ||
level == vr::k_EDeviceActivityLevel_Idle_Timeout;
if (awayNow == away) awaySince = now;
if (awayNow != away && now - awaySince >= std::chrono::seconds(1)) {
away = awayNow;
std::printf("SteamVR: headset %s\n", away ? "off (SteamVR's standby has the displays)" : "on");
if (!away) use = "headset on again";
}
// Raw poses: recentering or a new play area doesn't move anything.
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, poses, vr::k_unMaxTrackedDeviceCount);
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
const auto c = classes[i];
if (c != vr::TrackedDeviceClass_HMD && c != vr::TrackedDeviceClass_Controller &&
c != vr::TrackedDeviceClass_GenericTracker)
continue;
const auto &p = poses[i];
if (!p.bPoseIsValid || p.eTrackingResult != vr::TrackingResult_Running_OK) {
anchors[i].set = false; // tracking back or a controller turned on: start over
continue;
}
if (!anchors[i].set || grace || now - anchors[i].at > kMoveWindow) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
continue;
}
double metres, degrees;
Distance(anchors[i], p.mDeviceToAbsoluteTracking, metres, degrees);
if (metres > moveMetres || degrees > moveDegrees) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
if (use.empty()) {
char why[96];
std::snprintf(why, sizeof why, "%s %u moved %.1f mm, %.2f deg",
c == vr::TrackedDeviceClass_HMD ? "headset" : "device", i, metres * 1000, degrees);
use = why;
if (now - lastVr >= kTick) {
lastVr = now;
vr::VREvent_t ev;
while (sys->PollNextEvent(&ev, sizeof ev)) {
if (ev.eventType == vr::VREvent_Quit) {
stopSignal = SIGTERM;
sys->AcknowledgeQuit_Exiting();
} else if (ev.eventType == vr::VREvent_TrackedDeviceActivated ||
ev.eventType == vr::VREvent_TrackedDeviceDeactivated) {
lastClasses = now - std::chrono::hours(1);
}
}
}
if (now - lastClasses > std::chrono::seconds(5)) {
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i)
classes[i] = sys->GetTrackedDeviceClass(i);
lastClasses = now;
}
if (off && ReadInt(kBacklight, -1) > 0) {
// Something else lit them (SteamVR leaving standby, the brightness setting).
off = false;
std::remove(SavedPath().c_str());
use = "turned on elsewhere";
std::printf("displays on: turned on elsewhere\n");
// SteamVR's standby (headset taken off) turns the displays off itself; putting the
// headset on again (or the mount covering the sensor again) counts as use.
// A change counts once it has held for a second (the first reading after connecting
// is Idle).
const auto level = sys->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
const bool awayNow = level == vr::k_EDeviceActivityLevel_Idle || level == vr::k_EDeviceActivityLevel_Standby ||
level == vr::k_EDeviceActivityLevel_Idle_Timeout;
if (awayNow == away) awaySince = now;
if (awayNow != away && now - awaySince >= std::chrono::seconds(1)) {
away = awayNow;
std::printf("SteamVR: headset %s\n", away ? "off (SteamVR's standby has the displays)" : "on");
if (!away) use = "headset on again";
}
// Raw poses: recentering or a new play area doesn't move anything.
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, poses, vr::k_unMaxTrackedDeviceCount);
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
const auto c = classes[i];
if (c != vr::TrackedDeviceClass_HMD && c != vr::TrackedDeviceClass_Controller &&
c != vr::TrackedDeviceClass_GenericTracker)
continue;
const auto &p = poses[i];
if (!p.bPoseIsValid || p.eTrackingResult != vr::TrackingResult_Running_OK) {
anchors[i].set = false; // tracking back or a controller turned on: start over
continue;
}
if (!anchors[i].set || grace || now - anchors[i].at > kMoveWindow) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
continue;
}
double metres, degrees;
Distance(anchors[i], p.mDeviceToAbsoluteTracking, metres, degrees);
if (metres > moveMetres || degrees > moveDegrees) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
if (use.empty()) {
char why[96];
std::snprintf(why, sizeof why, "%s %u moved %.1f mm, %.2f deg",
c == vr::TrackedDeviceClass_HMD ? "headset" : "device", i, metres * 1000, degrees);
use = why;
}
}
}
if (off && ReadInt(kBacklight, -1) > 0) {
// Something else lit them (SteamVR leaving standby, the brightness setting).
off = false;
std::remove(SavedPath().c_str());
use = "turned on elsewhere";
std::printf("displays on: turned on elsewhere\n");
}
}
if (!use.empty()) {
lastUse = now;
+17 -5
View File
@@ -83,6 +83,7 @@ class Window(Adw.ApplicationWindow):
self.status = {}
self.revealed = False
self.setting = False # the switch is being set from the status, not by the user
self.asking = False # a status check is running
self.toasts = Adw.ToastOverlay()
view = Adw.ToolbarView()
@@ -139,16 +140,27 @@ class Window(Adw.ApplicationWindow):
"/tmp/frametop-remote.log and /tmp/frametop-vnc.log.")
page.add(about)
# Each check runs remote-ctl.sh (bash, curl, and python3 for the tailnet name), so
# it runs every 5 seconds, when the window comes to the front, and after a change.
self.refresh()
GLib.timeout_add_seconds(2, self.refresh)
GLib.timeout_add_seconds(5, self.refresh)
self.connect("notify::is-active", lambda win, _param: win.is_active() and win.refresh())
# --- status ---
def refresh(self):
proc = Gio.Subprocess.new([CTL, "status"], Gio.SubprocessFlags.STDOUT_PIPE | Gio.SubprocessFlags.STDERR_SILENCE)
proc.communicate_utf8_async(None, None, self.on_status)
if not self.asking:
self.asking = True
proc = Gio.Subprocess.new([CTL, "status"], Gio.SubprocessFlags.STDOUT_PIPE | Gio.SubprocessFlags.STDERR_SILENCE)
proc.communicate_utf8_async(None, None, self.on_status)
return True
def refresh_soon(self):
"""Again in a moment: remote access takes a second or two to start or stop."""
self.refresh()
GLib.timeout_add_seconds(2, lambda: self.refresh() and False)
def on_status(self, proc, result):
self.asking = False
try:
_, out, _ = proc.communicate_utf8_finish(result)
except GLib.Error:
@@ -197,7 +209,7 @@ class Window(Adw.ApplicationWindow):
self.toast("Remote access turns on at the next desktop restart")
else:
self.toast("Remote access on")
self.refresh()
self.refresh_soon()
def on_reveal(self, button):
self.revealed = button.get_active()
@@ -221,7 +233,7 @@ class Window(Adw.ApplicationWindow):
if self.status.get("running") == "1":
subprocess.run([CTL, "restart"], stdin=subprocess.DEVNULL, capture_output=True)
self.toast("New password set" + (": remote access restarted" if self.status.get("running") == "1" else ""))
self.refresh()
self.refresh_soon()
def copy(self, text, done):
if text:
+1 -1
View File
@@ -23,6 +23,6 @@ $cxx -c -o build/handcut.o handcut.cpp
$cxx -c -o build/handtest.o handtest.cpp
vrlibs="$(pkg-config --libs egl glesv2 gbm) -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64"
g++ -o build/ft-screens build/compositor.o build/vr.o build/keyboard.o build/handcut.o \
$(pkg-config --libs wlroots-0.20 wayland-server xkbcommon) $vrlibs
$(pkg-config --libs wlroots-0.20 wayland-server xkbcommon pixman-1) $vrlibs
g++ -o build/ft-handtest build/handtest.o build/handcut.o $vrlibs
echo "built build/ft-screens build/ft-handtest"'
+186 -12
View File
@@ -13,7 +13,7 @@
// scale first ("scale <screen> <s>", from ft-layout).
//
// Usage: ft-screens [--socket NAME] [--control NAME] [--no-vr] [--screen WxH@METRES]...
// [--spares N] [-- COMMAND ARGS...]
// [--spares N] [--rates F,V,H] [-- COMMAND ARGS...]
// --socket Wayland socket name in $XDG_RUNTIME_DIR (default ft-screens-0)
// --control the control socket's abstract name (default ft_screens)
// --no-vr run without SteamVR, for tests next to the running desktop: no panels, no
@@ -22,6 +22,8 @@
// --screen one per screen, in KWin's order (default: 3440x1440@2.4)
// --spares KWin's outputs after the screens: spares for floating windows (ft-floatd
// turns them on and sizes them; see docs/floating-windows.md)
// --rates frame rates in Hz for screens you look at, the rest you see, and hidden ones
// (0: every display frame; default 0,15,1; see frame_interval)
// COMMAND run with WAYLAND_DISPLAY set to our socket (e.g. the Frametop session)
// Runs in the dev container (wlroots 0.20); KWin connects from the host.
#define _GNU_SOURCE
@@ -33,7 +35,9 @@
#include <string.h>
#include <stddef.h>
#include <sys/socket.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <sys/timerfd.h>
#include <sys/un.h>
#include <sys/wait.h>
#include <time.h>
@@ -61,6 +65,12 @@
#include "controller-click.h"
#define MAX_SCREENS 24 // screens and spare outputs
// A screen counts as playing a video while its last VIDEO_COMMITS commits each redrew at
// least VIDEO_AREA percent of it, at VIDEO_HZ or more; for VIDEO_HOLD ms after that stops.
#define VIDEO_COMMITS 8
#define VIDEO_AREA 6
#define VIDEO_HZ 10
#define VIDEO_HOLD 1500
struct config {
int width, height;
@@ -80,6 +90,13 @@ struct screen {
int buffer_width, buffer_height;
struct wlr_xdg_toplevel_decoration_v1 *decoration; // answered on the first commit
struct wl_listener commit, destroy, decoration_destroy, set_title;
// Its frame rate (see tick): when its last frame callback went (ms), and what its recent
// commits looked like, to tell a video playing on it.
uint32_t frame_sent;
uint32_t commit_ms[VIDEO_COMMITS];
unsigned commit_at;
unsigned big; // a bit per recent commit: it redrew at least VIDEO_AREA percent
uint32_t video_until; // counts as a video until then (ms)
};
// Per client buffer: forget its import when it goes away.
@@ -104,6 +121,14 @@ struct server {
int spares;
struct wl_list buffers; // tracked_buffer
struct wl_event_source *tick;
int tick_fd; // timerfd, at absolute times in step with the display's vsync (see schedule)
int64_t period_ns, base_ns, synced_ns; // the display's frame time, a tick time, last sync
double phase_ms; // ticks come this long after a vsync
// Frame rates in Hz for each attention level (0: every display frame), and a remote
// viewer's lease: until then (ms) every screen gets full rate ("watch").
int rate[3];
uint32_t watch_until;
uint32_t typed_ms; // the last key sent to the desktop: its screen counts as focused
struct screen *pointer_focus;
struct ft_controller_click controller_click;
pid_t child;
@@ -149,6 +174,25 @@ static void track_buffer(struct server *s, struct wlr_buffer *buffer) {
// ---------------------------------------------------------------- screens
// A video (or anything moving over a large area) on a screen you don't look at keeps the
// full frame rate (see frame_interval): its commits keep redrawing much of it, and keep
// coming as fast as its rate lets them. A cursor blinking or a spinner turning redraws a
// small part, and a page changing now and then doesn't keep coming.
static void note_damage(struct screen *sc, struct wlr_surface *surface, struct wlr_buffer *buffer) {
int n = 0;
const pixman_box32_t *r = pixman_region32_rectangles(&surface->buffer_damage, &n);
int64_t area = 0;
for (int i = 0; i < n; ++i) area += (int64_t)(r[i].x2 - r[i].x1) * (r[i].y2 - r[i].y1);
const bool big = area * 100 >= (int64_t)buffer->width * buffer->height * VIDEO_AREA;
const unsigned all = (1u << VIDEO_COMMITS) - 1;
sc->big = ((sc->big << 1) | big) & all;
const uint32_t t = now_ms();
const uint32_t oldest = sc->commit_ms[sc->commit_at]; // the commit VIDEO_COMMITS ago
sc->commit_ms[sc->commit_at] = t;
sc->commit_at = (sc->commit_at + 1) % VIDEO_COMMITS;
if (sc->big == all && t - oldest <= VIDEO_COMMITS * 1000 / VIDEO_HZ) sc->video_until = t + VIDEO_HOLD;
}
static void screen_commit(struct wl_listener *l, void *data) {
struct screen *sc = wl_container_of(l, sc, commit);
struct wlr_xdg_surface *xdg = sc->toplevel->base;
@@ -172,6 +216,7 @@ static void screen_commit(struct wl_listener *l, void *data) {
if (!buffer) return;
sc->frame_pending = true;
++sc->commits;
note_damage(sc, xdg->surface, buffer);
sc->buffer_width = buffer->width, sc->buffer_height = buffer->height;
if (buffer == sc->held) return;
struct wlr_dmabuf_attributes a;
@@ -301,6 +346,20 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
struct screen *sc = s->screens[e->screen];
struct wlr_surface *surface = sc->toplevel->base->surface;
if (e->type == FT_FRONT) {
// A spin brought this panel to the front: typing goes to it, as after a click there,
// and ft-floatd makes its window (or the top one on a screen) KWin's active window.
wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
s->keys_clicked = true;
char msg[32];
snprintf(msg, sizeof msg, "front %d", e->screen + 1);
struct sockaddr_un addr = {.sun_family = AF_UNIX};
const char name[] = "frametop_float";
memcpy(addr.sun_path + 1, name, sizeof name - 1);
sendto(s->relay_fd, msg, strlen(msg), MSG_DONTWAIT, (struct sockaddr *)&addr,
offsetof(struct sockaddr_un, sun_path) + 1 + sizeof name - 1);
return;
}
const uint32_t t = now_ms();
const double x = e->x / s->scale[e->screen], y = e->y / s->scale[e->screen]; // KWin's units
switch (e->type) {
@@ -381,9 +440,64 @@ static void keys_update(struct server *s) {
offsetof(struct sockaddr_un, sun_path) + 1 + sizeof name - 1);
}
// Every ~11 ms (90 Hz): SteamVR events, and frame callbacks for screens that committed.
static int tick(void *data) {
// How often a screen gets its frame callback (ms; 0 every tick). KWin draws a screen only
// after its callback, and its apps wait for theirs, so this is the screen's frame rate:
// full where you look (ft_vr_screen_attention) and for a video, lower for the rest of what
// you see, and about once a second for what you don't (hidden, behind you, paused for a VR
// game). A screen nothing changes on costs nothing at any rate: KWin commits only when
// something on it moved. A remote viewer ("watch") sees every screen at full rate.
static uint32_t frame_interval(struct server *s, struct screen *sc, uint32_t t) {
if ((int32_t)(s->watch_until - t) > 0) return 0;
enum ft_attention a = ft_vr_screen_attention(sc->index);
if (a == FT_IN_VIEW && (int32_t)(sc->video_until - t) > 0) a = FT_FOCUSED;
if (a != FT_FOCUSED && t - s->typed_ms < 1500 &&
s->seat->keyboard_state.focused_surface == sc->toplevel->base->surface)
a = FT_FOCUSED; // typing on it while looking elsewhere
const int hz = s->rate[a];
return hz > 0 ? 1000 / hz : 0;
}
static int64_t mono_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000000000LL + ts.tv_nsec;
}
// Ticks come once per display frame, phase_ms after its vsync: KWin gets its frame callbacks
// early in the frame and has the rest of it to draw before vrcompositor takes the panels.
// The vsync is read from SteamVR once a second, and the ticks run at absolute times between
// reads, so they keep their place (a timer set again after each tick, as before, slid
// through the frame and came about 85 times a second at 90 Hz). Without SteamVR's timing
// they still come at the display's rate (90 Hz until known). While paused, every 100 ms.
static void schedule(struct server *s) {
const int64_t now = mono_ns();
int64_t next;
if (ft_vr_paused()) {
next = now + 100000000LL;
} else {
double since, hz;
if (now - s->synced_ns >= 1000000000LL) {
s->synced_ns = now;
if (ft_vr_vsync(&since, &hz)) {
const int64_t period = (int64_t)(1e9 / hz);
if (llabs(period - s->period_ns) > 100000) wlr_log(WLR_INFO, "display at %.1f Hz", hz);
s->period_ns = period;
s->base_ns = now - (int64_t)(since * 1e9) + (int64_t)(s->phase_ms * 1e6);
while (s->base_ns > now) s->base_ns -= s->period_ns;
}
}
next = s->base_ns + ((now - s->base_ns) / s->period_ns + 1) * s->period_ns;
}
struct itimerspec its = {.it_value = {.tv_sec = next / 1000000000LL, .tv_nsec = next % 1000000000LL}};
timerfd_settime(s->tick_fd, TFD_TIMER_ABSTIME, &its, NULL);
}
// Each tick: SteamVR events, and frame callbacks for screens that committed and are due.
static int tick(int fd, uint32_t mask, void *data) {
struct server *s = data;
uint64_t expirations;
const ssize_t got = read(fd, &expirations, sizeof expirations); // clears it; how many doesn't matter
(void)got;
ft_vr_poll(handle_vr_event, s);
if (++s->ticks % 9 == 0) keys_update(s);
if (s->kb_close_at && s->ticks >= s->kb_close_at) {
@@ -395,14 +509,17 @@ static int tick(void *data) {
}
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
const uint32_t t = now_ms();
// Due within half a tick counts as due, so 15 Hz is every 6th tick at 90 Hz, not 7th.
const uint32_t slack = ft_vr_paused() ? 50 : (uint32_t)(s->period_ns / 2000000);
for (int i = 0; i < MAX_SCREENS; ++i) {
struct screen *sc = s->screens[i];
if (sc && sc->frame_pending) {
sc->frame_pending = false;
wlr_surface_send_frame_done(sc->toplevel->base->surface, &now);
}
if (!sc || !sc->frame_pending || t - sc->frame_sent + slack < frame_interval(s, sc, t)) continue;
sc->frame_pending = false;
sc->frame_sent = t;
wlr_surface_send_frame_done(sc->toplevel->base->surface, &now);
}
wl_event_source_timer_update(s->tick, 11);
schedule(s);
return 0;
}
@@ -427,6 +544,7 @@ static bool key_held(const struct wlr_keyboard *kb, uint32_t code) {
// One key to the focused screen, through the seat's keyboard so its xkb state and
// modifiers stay right.
static void send_key(struct server *s, uint32_t code, int pressed) {
s->typed_ms = now_ms();
struct wlr_keyboard_key_event ev = {
.time_msec = now_ms(), .keycode = code, .update_state = true,
.state = pressed ? WL_KEYBOARD_KEY_STATE_PRESSED : WL_KEYBOARD_KEY_STATE_RELEASED};
@@ -580,6 +698,38 @@ static int control_readable(int fd, uint32_t mask, void *data) {
handle_vr_event(&e, s);
snprintf(reply, sizeof reply, "ok");
}
} else if (sscanf(buf, "watch %d", &value) == 1) {
// A remote viewer is watching for that many seconds (vnc-bridge.sh renews it).
s->watch_until = now_ms() + (uint32_t)(value < 0 ? 0 : value > 60 ? 60 : value) * 1000;
snprintf(reply, sizeof reply, "ok");
} else if (sscanf(buf, "rates %d %d %d", &index, &w, &h) == 3) {
// Frame rates in Hz: focused, in view, hidden (0: every display frame).
if (index < 0 || w < 0 || h < 0 || index > 240 || w > 240 || h > 240) {
snprintf(reply, sizeof reply, "error rates <focused> <in view> <hidden> (Hz, 0 full)");
} else {
s->rate[FT_FOCUSED] = index, s->rate[FT_IN_VIEW] = w, s->rate[FT_HIDDEN] = h;
wlr_log(WLR_INFO, "frame rates: focused %d, in view %d, hidden %d (0 full)", index, w, h);
snprintf(reply, sizeof reply, "ok");
}
} else if (sscanf(buf, "phase %lf", &scale) == 1) {
// Ticks this long after the vsync (ms), for tuning.
if (!(scale >= 0 && scale < 20)) snprintf(reply, sizeof reply, "error phase <ms after vsync>");
else s->phase_ms = scale, s->synced_ns = 0, snprintf(reply, sizeof reply, "ok");
} else if (strcmp(buf, "rates?") == 0) {
// "ok <focused> <in view> <hidden> <display Hz> <phase ms> <watched>" and a line per
// screen: "<screen> hidden|view|focused <ms between frames> <video 0|1>".
static const char *names[] = {"hidden", "view", "focused"};
const uint32_t t = now_ms();
int at = snprintf(reply, sizeof reply, "ok %d %d %d %.1f %.1f %d", s->rate[FT_FOCUSED],
s->rate[FT_IN_VIEW], s->rate[FT_HIDDEN], 1e9 / s->period_ns, s->phase_ms,
(int32_t)(s->watch_until - t) > 0);
for (int i = 0; i < MAX_SCREENS && at < (int)sizeof reply; ++i) {
struct screen *sc = s->screens[i];
if (!sc) continue;
at += snprintf(reply + at, sizeof reply - at, "\n%d %s %u %d", i + 1,
names[ft_vr_screen_attention(i)], frame_interval(s, sc, t),
(int32_t)(sc->video_until - t) > 0);
}
} else if (strcmp(buf, "toplevels") == 0) {
int n = 0, at = 0;
for (int i = 0; i < MAX_SCREENS; ++i) n += s->screens[i] != NULL;
@@ -659,9 +809,14 @@ static bool setup_dmabuf(struct server *s) {
int main(int argc, char **argv) {
struct server s = {0};
s.controller_click.threshold = 8;
// About 0.9 degrees on a 3.4 m wide 3440-pixel screen 2 m away; 8 (the first default,
// about 0.2 degrees) needed a very still hand to click (headset test 2026-10-03).
s.controller_click.threshold = 32;
for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
s.kb_screen = -1;
s.rate[FT_FOCUSED] = 0, s.rate[FT_IN_VIEW] = 15, s.rate[FT_HIDDEN] = 1;
s.period_ns = 1000000000LL / 90;
s.phase_ms = 1;
const char *socket_name = "ft-screens-0", *control_name = "ft_screens";
char **command = NULL;
s.vr = true;
@@ -674,6 +829,11 @@ int main(int argc, char **argv) {
s.spares = atoi(argv[++i]);
} else if (strcmp(argv[i], "--no-vr") == 0) {
s.vr = false;
} else if (strcmp(argv[i], "--rates") == 0 && i + 1 < argc) {
if (sscanf(argv[++i], "%d,%d,%d", &s.rate[FT_FOCUSED], &s.rate[FT_IN_VIEW], &s.rate[FT_HIDDEN]) != 3) {
fprintf(stderr, "bad --rates %s (want FOCUSED,IN_VIEW,HIDDEN in Hz, 0 full)\n", argv[i]);
return 2;
}
} else if (strcmp(argv[i], "--screen") == 0 && i + 1 < argc && s.n_config < MAX_SCREENS) {
struct config *c = &s.config[s.n_config];
c->metres = 0;
@@ -689,7 +849,7 @@ int main(int argc, char **argv) {
} else {
fprintf(stderr,
"usage: %s [--socket NAME] [--control NAME] [--no-vr] [--screen WxH@METRES]... "
"[--spares N] [-- COMMAND ARGS...]\n",
"[--spares N] [--rates F,V,H] [-- COMMAND ARGS...]\n",
argv[0]);
return 2;
}
@@ -741,8 +901,10 @@ int main(int argc, char **argv) {
s.relay_fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
wl_event_loop_add_fd(s.loop, control, WL_EVENT_READABLE, control_readable, &s);
s.tick = wl_event_loop_add_timer(s.loop, tick, &s);
wl_event_source_timer_update(s.tick, 11);
s.tick_fd = timerfd_create(CLOCK_MONOTONIC, TFD_NONBLOCK | TFD_CLOEXEC);
s.base_ns = mono_ns();
s.tick = wl_event_loop_add_fd(s.loop, s.tick_fd, WL_EVENT_READABLE, tick, &s);
schedule(&s);
wl_event_loop_add_signal(s.loop, SIGINT, stop, &s);
wl_event_loop_add_signal(s.loop, SIGTERM, stop, &s);
wl_event_loop_add_signal(s.loop, SIGCHLD, child_exited, &s);
@@ -757,6 +919,18 @@ int main(int argc, char **argv) {
}
}
// Ahead of the desktop's programs for the CPU, after the session started (it would
// inherit it): KWin waits for our ticks to draw. SteamOS lets users go to nice -8
// (RLIMIT_NICE 28) once they raise their soft limit; without that, as before.
if (s.vr) {
struct rlimit rl;
if (getrlimit(RLIMIT_NICE, &rl) == 0 && rl.rlim_cur < rl.rlim_max) {
rl.rlim_cur = rl.rlim_max;
setrlimit(RLIMIT_NICE, &rl);
}
if (setpriority(PRIO_PROCESS, 0, -5) != 0) wlr_log(WLR_INFO, "can't raise our priority (nice -5); running at 0");
}
wl_display_run(s.display);
wlr_log(WLR_INFO, "stopping");
+15 -1
View File
@@ -459,7 +459,21 @@ void Hide() {
bool Shown() { return g_shown; }
void SetLasers(bool on) {
if (Create()) vr::VROverlay()->SetOverlayFlag(g_overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, on);
static vr::VROverlayHandle_t set = vr::k_ulOverlayHandleInvalid; // the overlay `was` is for
static bool was = false;
if (!Create() || (set == g_overlay && was == on)) return;
set = g_overlay, was = on;
vr::VROverlay()->SetOverlayFlag(g_overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, on);
}
bool Aimed(const vr::HmdMatrix34_t &laser) {
if (!g_shown) return false;
vr::VROverlayIntersectionParams_t in{};
in.eOrigin = vr::TrackingUniverseStanding;
in.vSource = {laser.m[0][3], laser.m[1][3], laser.m[2][3]};
in.vDirection = {-laser.m[0][2], -laser.m[1][2], -laser.m[2][2]};
vr::VROverlayIntersectionResults_t out{};
return vr::VROverlay()->ComputeOverlayIntersection(g_overlay, &in, &out);
}
void Poll(void (*handle)(const Event &, void *), void *data) {
+2
View File
@@ -26,6 +26,8 @@ const vr::HmdMatrix34_t &Pose();
void EndDragBy(uint32_t device);
// Controllers' lasers work the panel with the dashboard closed, like the screens'.
void SetLasers(bool on);
// A laser (its pose, aiming along -z) points at the panel.
bool Aimed(const vr::HmdMatrix34_t &laser);
// The panel's input: key presses and releases, and Closed for its Close key.
void Poll(void (*handle)(const Event &, void *), void *data);
void Destroy();
+491 -41
View File
@@ -11,6 +11,8 @@
// its centre (it snaps level within kRollSnap), or scroll on it for kRollStep steps.
// - a resize tab on the bottom right corner: drag it to set the width (the height
// follows the screen's resolution).
// - a reset button left of the bar: every screen back in its layout, around where you
// are now (`ft-layout apply`, like Meta+Shift+R).
// The controls are translucent, like SteamVR's own, and brighten under a laser. They
// are invisible until a laser (a controller's, or the 3D mouse's) lands on or passes very close to
// one of them (UpdateControls).
@@ -40,9 +42,17 @@
// default it's off while a game (a scene app) runs: the screens stay up over the game,
// the controllers stay in it, and the 3D mouse (its own laser mode) or the dashboard
// works the screens. Modes: always, outside_games (default), dashboard (never on its
// own; also for flatscreen games, which aren't scene apps).
// own; also for flatscreen games, which aren't scene apps). Where the mode leaves the
// controllers to the game, pointing a controller at a panel (a screen, a floating window
// and its popups, their controls, the keyboard) turns the laser on for it until you
// point away, like SteamVR's own floating windows (UpdateAim).
// - during a VR game the screens hide unless the dashboard is open (g_inGames, default),
// or stay visible over it; the hotkey still shows them.
// - paused ("pause on", from the input relay when Frametop pauses for a VR game,
// input/game_pause.py): every screen and floating window hides whatever the mode, the
// hotkey, or the dashboard says, and compositor.c gives KWin a frame callback once a
// second, as for any hidden screen, so KWin and its apps hardly draw. "pause off" undoes
// it.
// - hand cutouts (handcut.cpp): where ft-hands (hands/) tracks a hand between an eye and a
// screen, that eye sees through the screen (to Room View). Only then is the screen
// drawn by us, into a side-by-side buffer (one half per eye); otherwise its client
@@ -180,8 +190,12 @@ Mat TipOffset(vr::TrackedDeviceIndex_t dev) {
Tip tip{model, Identity(), false, now};
vr::RenderModel_ControllerMode_State_t mode{};
vr::RenderModel_ComponentState_t state{};
if (model[0] && vr::VRRenderModels()->GetComponentStateForDevicePath(model, vr::k_pch_Controller_Component_Tip,
vr::k_ulInvalidInputValueHandle, &mode, &state))
// GetComponentState, not GetComponentStateForDevicePath: without an input source handle
// the latter fails for every component while a VR game runs, and the rays came from the
// pose, 40 degrees above the laser. The tip doesn't move with the buttons.
vr::VRControllerState_t buttons{};
if (model[0] && vr::VRRenderModels()->GetComponentState(model, vr::k_pch_Controller_Component_Tip, &buttons, &mode,
&state))
tip.offset = state.mTrackingToComponentLocal, tip.found = true;
cache[dev] = tip;
return tip.offset;
@@ -228,6 +242,7 @@ constexpr double kRollSnap = 2.5; // degrees from level where rolling snaps l
constexpr double kRollStep = 5; // degrees per scroll notch on the roll button
constexpr float kChromeIdle = 0.55f; // the controls' opacity without a laser on them
constexpr long kControlsLinger = 35; // ticks (~0.4 s) the controls stay after a laser leaves
constexpr long kAimLinger = 25; // ticks (~0.3 s) a panel keeps the laser on after the aim leaves it
long g_tick = 0; // ft_vr_poll calls
bool g_vr = false; // connected to SteamVR (ft-screens --no-vr runs without it)
constexpr vr::TrackedDeviceIndex_t kNone = vr::k_unTrackedDeviceIndexInvalid;
@@ -242,7 +257,8 @@ struct Screen {
vr::VROverlayHandle_t overlay = vr::k_ulOverlayHandleInvalid, bar = vr::k_ulOverlayHandleInvalid,
handle = vr::k_ulOverlayHandleInvalid, curveButton = vr::k_ulOverlayHandleInvalid,
rollButton = vr::k_ulOverlayHandleInvalid, dockButton = vr::k_ulOverlayHandleInvalid,
closeButton = vr::k_ulOverlayHandleInvalid; // the last two: floating windows
closeButton = vr::k_ulOverlayHandleInvalid, // the last two: floating windows
resetButton = vr::k_ulOverlayHandleInvalid; // desktop screens only
int width = 0, height = 0; // current buffer size (mouse scale)
double metres = 1;
double curve = 0; // cylinder radius in metres; 0 = flat
@@ -259,11 +275,12 @@ struct Screen {
double grabX = 0, grabY = 0; // resize: the grab point relative to the corner
Mat rollFrom = Identity(); // roll: the pose at the press (pinRel when pinned)
double rollAngle = 0; // roll: the laser's angle around the centre then
bool hover[6] = {}; // a laser is on the bar, curve, roll, resize, dock, close control
bool hover[7] = {}; // a laser is on the bar, curve, roll, resize, dock, close, reset control
bool lasers = true; // MakeOverlaysInteractiveIfVisible is set
float controls = 0; // the controls' fade, 0 (hidden) .. 1
bool controlsUp = false; // the controls' overlays are shown
long nearUntil = 0; // a laser was near the controls until this tick
long aimUntil = 0; // a hand controller pointed at it until this tick (UpdateAim)
vr::TrackedDeviceIndex_t pinTarget = kNone; // moving: rides on this controller when let go
vr::TrackedDeviceIndex_t onWrist = kNone; // moving: the laser is in this controller's ring
bool barLit = false;
@@ -287,6 +304,11 @@ struct Screen {
long resizeSent = 0; // g_tick of the last resize request (they're throttled)
int resizeW = 0, resizeH = 0; // ...and its size
std::map<int, Sub> subs;
// Attention (UpdateAttention): what ft_vr_screen_attention answers, and until when (ms)
// it stays focused or in view after the last reason for it.
ft_attention attention = FT_FOCUSED;
int64_t inputMs = INT64_MIN / 2; // the last pointer event on it or its controls
int64_t focusUntil = 0, viewUntil = 0;
double heightMetres() const {
if (floating && cropW > 0) return metres * cropH / cropW;
return width > 0 ? metres * height / width : metres * 9 / 16;
@@ -294,11 +316,11 @@ struct Screen {
// Buffer pixels from OpenVR's mouse position on the panel (its origin is bottom left).
// A cropped panel too: SteamVR gives the position in the whole texture, not the crop.
void ToBuffer(double mx, double my, double *x, double *y) const { *x = mx, *y = height - my; }
std::array<vr::VROverlayHandle_t, 6> Controls() const {
return {bar, curveButton, rollButton, handle, dockButton, closeButton};
std::array<vr::VROverlayHandle_t, 7> Controls() const {
return {bar, curveButton, rollButton, handle, dockButton, closeButton, resetButton};
}
std::array<vr::VROverlayHandle_t, 7> All() const {
return {overlay, bar, curveButton, rollButton, handle, dockButton, closeButton};
std::array<vr::VROverlayHandle_t, 8> All() const {
return {overlay, bar, curveButton, rollButton, handle, dockButton, closeButton, resetButton};
}
};
std::map<int, Screen> g_screens;
@@ -320,6 +342,7 @@ double g_gestureAngle = 20; // gesture: look within this of the controller
std::string g_gestureHand = "left";
Lasers g_lasers = Lasers::OutsideGames; // when controllers' lasers work the screens (see the top)
bool g_gameRunning = false; // a scene app (VR game) is running
bool g_paused = false; // Frametop paused for a VR game: everything hidden (see the top)
InGames g_inGames = InGames::Hide; // during a VR game, the always mode acts like the dashboard mode
// ---------------------------------------------------------------- chrome (bar, button, handle)
@@ -437,6 +460,19 @@ std::vector<uint8_t> DockTexture(int n) {
DiscRim);
}
std::vector<uint8_t> ResetTexture(int n) {
// A reticle: "put the screens back around you" (like a recenter).
return ControlTexture(
n, InDisc,
[](double u, double v) {
const double r = std::hypot(u, v);
if (std::fabs(r - 0.4) < 0.07 || r < 0.13) return true; // the ring and the centre
return (std::fabs(u) < 0.06 && std::fabs(v) > 0.47 && std::fabs(v) < 0.72) ||
(std::fabs(v) < 0.06 && std::fabs(u) > 0.47 && std::fabs(u) < 0.72); // the ticks
},
DiscRim);
}
vr::VROverlayHandle_t MakeChrome(const char *key, const char *name, const std::vector<uint8_t> &px, int w, int h) {
vr::VROverlayHandle_t o = vr::k_ulOverlayHandleInvalid;
if (vr::VROverlay()->CreateOverlay(key, name, &o) != vr::VROverlayError_None) return o;
@@ -589,16 +625,18 @@ Mat BarOffset(const Screen &s) { return OnSurface(s, 0, BarY(s), 0.003); }
// Put the bar, the curve button, and the corner tab under the screen (same parent: the
// room or the controller), sized for the screen and its distance, and on its surface.
// Where each control sits, relative to the screen: bar, curve, roll, resize tab, and a
// floating window's dock and close buttons (left of the bar).
std::array<Mat, 6> ControlOffsets(const Screen &s) {
// Where each control sits, relative to the screen: bar, curve, roll, resize tab, a
// floating window's dock and close buttons (left of the bar), and a desktop screen's reset
// button (left of the bar, where a floating window has its dock button).
std::array<Mat, 7> ControlOffsets(const Screen &s) {
const double h = s.heightMetres(), bar = s.chrome, button = s.grip, gap = bar * 0.06;
return {BarOffset(s), OnSurface(s, bar / 2 + gap + button / 2, BarY(s), 0.003),
OnSurface(s, bar / 2 + gap * 2 + button * 1.5, BarY(s), 0.003),
// The tab's top left corner is the screen's bottom right corner.
OnSurface(s, s.metres / 2 + s.grip / 2, -(h / 2 + s.grip / 2), 0.003),
OnSurface(s, -(bar / 2 + gap + button / 2), BarY(s), 0.003),
OnSurface(s, -(bar / 2 + gap * 2 + button * 1.5), BarY(s), 0.003)};
OnSurface(s, -(bar / 2 + gap * 2 + button * 1.5), BarY(s), 0.003),
OnSurface(s, -(bar / 2 + gap + button / 2), BarY(s), 0.003)};
}
// A floating window's popups and dialogs, a few millimetres in front of it, where they are
@@ -633,12 +671,15 @@ void PlaceChrome(Screen &s) {
if (s.floating) {
vr::VROverlay()->SetOverlayWidthInMeters(s.dockButton, float(button));
vr::VROverlay()->SetOverlayWidthInMeters(s.closeButton, float(button));
} else {
vr::VROverlay()->SetOverlayWidthInMeters(s.resetButton, float(button));
}
// Curved, the bar bends with the screen's bottom edge.
vr::VROverlay()->SetOverlayCurvature(s.bar, s.curve > 0 ? float(std::min(1.0, bar / (2 * M_PI * s.curve))) : 0.f);
const std::pair<vr::VROverlayHandle_t, Mat> parts[] = {
{s.bar, offsets[0]}, {s.curveButton, offsets[1]}, {s.rollButton, offsets[2]},
{s.handle, offsets[3]}, {s.dockButton, offsets[4]}, {s.closeButton, offsets[5]}};
{s.handle, offsets[3]}, {s.dockButton, offsets[4]}, {s.closeButton, offsets[5]},
{s.resetButton, offsets[6]}};
PlaceSubs(s);
if (s.pinned != kNone) {
for (const auto &[o, off] : parts) {
@@ -734,6 +775,7 @@ void UpdateGame() {
}
bool ModeVisible() {
if (g_paused) return false;
switch (EffectiveMode()) {
case Mode::Always: return !g_manual;
case Mode::Toggle: return g_manual;
@@ -759,10 +801,10 @@ bool ModeVisible() {
void ApplyAlpha(const Screen &s) {
vr::VROverlay()->SetOverlayAlpha(s.overlay, s.alpha);
for (const auto &[k, sub] : s.subs) vr::VROverlay()->SetOverlayAlpha(sub.overlay, s.alpha);
const bool active[6] = {s.hover[0] || s.drag == Drag::Move, s.hover[1], s.hover[2] || s.drag == Drag::Roll,
s.hover[3] || s.drag == Drag::Resize, s.hover[4], s.hover[5]};
const bool active[7] = {s.hover[0] || s.drag == Drag::Move, s.hover[1], s.hover[2] || s.drag == Drag::Roll,
s.hover[3] || s.drag == Drag::Resize, s.hover[4], s.hover[5], s.hover[6]};
const auto controls = s.Controls();
for (int k = 0; k < 6; ++k)
for (int k = 0; k < 7; ++k)
if (controls[k] != vr::k_ulOverlayHandleInvalid)
vr::VROverlay()->SetOverlayAlpha(controls[k], s.alpha * s.controls * (active[k] ? 1.f : kChromeIdle));
}
@@ -792,7 +834,7 @@ void UpdateVisibility() {
Mat head;
const bool haveHead = DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head);
for (auto &[i, s] : g_screens) {
bool visible = s.shown && (shared || s.drag != Drag::None) && !s.alone;
bool visible = !g_paused && s.shown && (shared || s.drag != Drag::None) && !s.alone;
// A floating window's panel: while a window floats on it, its output is on, and the
// window isn't minimized (and once it has a crop).
if (s.floating) visible = visible && s.floatOn && s.outputOn && !s.minimized && s.cropW > 0;
@@ -810,16 +852,92 @@ void UpdateVisibility() {
}
}
// Attention, for each screen's frame rate (compositor.c gives KWin frame callbacks at a
// rate for each level): focused while you look at the screen (within kFocusAngle of where
// your head points), a laser or the mouse is on it, or it's being carried; in view while
// any of it is within kViewAngle; hidden otherwise, or while it isn't shown. A level stays
// for a moment after its reason goes, so a glance away doesn't make it stutter, and goes up
// at once.
constexpr double kFocusAngle = 12, kViewAngle = 60; // degrees
constexpr int64_t kFocusLinger = 1500, kViewLinger = 500, kInputFocus = 1500; // ms
bool RayOnPlane(const Mat &p, const Mat &d, double *x, double *y);
int64_t NowMs() {
return std::chrono::duration_cast<std::chrono::milliseconds>(Clock::now().time_since_epoch()).count();
}
// The smallest angle between where the head points and the screen: to the point of its
// rectangle nearest where the head's ray meets its plane, its centre, and its corners. A
// curved screen counts as flat; the angles hardly differ.
double AngleToScreen(const Screen &s, const Mat &p, const Mat &head) {
const double hw = s.metres / 2, hh = s.heightMetres() / 2;
double f[3];
Column(head, 2, f); // the head's +Z points backward
auto angleTo = [&](double u, double v) {
double to[3];
for (int i = 0; i < 3; ++i) to[i] = p.m[i][3] + u * p.m[i][0] + v * p.m[i][1] - head.m[i][3];
const double len = std::sqrt(Dot3(to, to)) + 1e-9;
return std::acos(std::clamp(-Dot3(f, to) / len, -1.0, 1.0)) * 180 / M_PI;
};
double best = 180, x, y;
if (RayOnPlane(p, head, &x, &y)) best = angleTo(std::clamp(x, -hw, hw), std::clamp(y, -hh, hh));
for (double u : {-hw, 0.0, hw})
for (double v : {-hh, 0.0, hh}) best = std::min(best, angleTo(u, v));
return best;
}
void UpdateAttention() {
const int64_t now = NowMs();
Mat head;
const bool haveHead = DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head);
for (auto &[i, s] : g_screens) {
if (!s.visible) {
s.attention = FT_HIDDEN;
s.focusUntil = s.viewUntil = 0;
continue;
}
bool focus = s.drag != Drag::None || now - s.inputMs < kInputFocus, view = focus;
Mat p;
if (!haveHead) {
view = true; // nothing to go by
} else if (ScreenPose(s, &p)) {
const double a = AngleToScreen(s, p, head);
focus = focus || a <= kFocusAngle;
view = view || a <= kViewAngle;
}
if (focus) s.focusUntil = now + kFocusLinger;
if (view) s.viewUntil = now + kViewLinger;
s.attention = now < s.focusUntil ? FT_FOCUSED : now < s.viewUntil ? FT_IN_VIEW : FT_HIDDEN;
}
}
// Controllers' lasers on the screens (see the top): the flag follows the mode and whether a
// VR game runs.
// VR game runs, and where the mode leaves the controllers to the game, whether one points at
// the panel (UpdateAim).
bool LasersByMode() { return g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning); }
long g_keyboardAimUntil = 0;
void UpdateLasers() {
const bool want = g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning);
const bool byMode = LasersByMode();
for (auto &[i, s] : g_screens) {
const bool want = byMode || (s.visible && g_tick < s.aimUntil);
if (s.lasers == want) continue;
s.lasers = want;
vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, want);
}
if (keyboard::Shown()) keyboard::SetLasers(byMode || g_tick < g_keyboardAimUntil);
}
// The distance from a laser's line to a point ahead of it, or -1 when it's behind.
double RayDistance(const Mat &d, const Mat &c) {
const double o[3] = {d.m[0][3], d.m[1][3], d.m[2][3]}, dir[3] = {-d.m[0][2], -d.m[1][2], -d.m[2][2]};
const double v[3] = {c.m[0][3] - o[0], c.m[1][3] - o[1], c.m[2][3] - o[2]};
const double t = Dot3(v, dir);
if (t <= 0) return -1;
const double q[3] = {v[0] - dir[0] * t, v[1] - dir[1] * t, v[2] - dir[2] * t};
return std::sqrt(Dot3(q, q));
}
// The controls are invisible until a laser is on one of them (SteamVR's hover event) or
@@ -842,18 +960,14 @@ void UpdateControls() {
for (double f : {-0.5, -0.25, 0.0, 0.25, 0.5})
spots.push_back(Mul(p, Mul(offsets[0], Translation(f * s.chrome, 0, 0))));
const auto controls = s.Controls();
for (int k = 1; k < 6; ++k)
for (int k = 1; k < 7; ++k)
if (controls[k] != vr::k_ulOverlayHandleInvalid) spots.push_back(Mul(p, offsets[k]));
const double reach = std::max(s.grip * 1.5, s.chrome * 0.12);
for (const Mat &d : lasers) {
const double o[3] = {d.m[0][3], d.m[1][3], d.m[2][3]}, dir[3] = {-d.m[0][2], -d.m[1][2], -d.m[2][2]};
bool close = false;
for (const Mat &c : spots) {
const double v[3] = {c.m[0][3] - o[0], c.m[1][3] - o[1], c.m[2][3] - o[2]};
const double t = Dot3(v, dir);
if (t <= 0) continue;
const double q[3] = {v[0] - dir[0] * t, v[1] - dir[1] * t, v[2] - dir[2] * t};
if (Dot3(q, q) <= reach * reach) close = true;
const double r = RayDistance(d, c);
if (r >= 0 && r <= reach) close = true;
}
if (close) {
s.nearUntil = g_tick + kControlsLinger;
@@ -894,6 +1008,21 @@ void SendFloat(const std::string &msg) {
std::printf("to ft-floatd: %s\n", msg.c_str());
}
// To the pointer helper (@ft_pointer_helper), from an unbound socket.
void SendPointer(const std::string &msg) {
static const int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
sockaddr_un addr{};
addr.sun_family = AF_UNIX;
const char name[] = "ft_pointer_helper";
std::memcpy(addr.sun_path + 1, name, sizeof name - 1);
sendto(fd, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<sockaddr *>(&addr),
socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1));
}
// A new panel: the pointer helper reads SteamVR's list of panels only every 20 s, so it's told
// at once ("overlay <key>"), or the mouse couldn't click a menu until then.
void AnnounceOverlay(const char *key) { SendPointer(std::string("overlay ") + key); }
// Where a device's ray meets the screen's plane, in the screen's x (right) and y (up),
// metres from its centre.
bool RayOnPlane(const Mat &p, const Mat &d, double *x, double *y) {
@@ -1017,15 +1146,8 @@ void EndDrag(Screen &s) {
ApplyAlpha(s);
}
// KWin's outputs follow where the screens are, so the pointer and dragged windows cross
// to the screen you see next to this one: `ft-layout scale` runs once a move has settled.
long g_arrangeAt = -1; // g_tick to run it at, -1 = not pending
void ArrangeDesktopSoon() { g_arrangeAt = g_tick + 45; } // about half a second
void UpdateArrange() {
if (g_arrangeAt < 0 || g_tick < g_arrangeAt) return;
g_arrangeAt = -1;
// Run `ft-layout <cmd>` in the background, logging to /tmp/frametop-layout.log.
void RunLayout(const char *cmd) {
char exe[PATH_MAX];
if (!realpath("/proc/self/exe", exe)) return;
std::string layout(exe); // <repo>/screens/build/ft-screens -> <repo>/layout/ft-layout
@@ -1036,14 +1158,26 @@ void UpdateArrange() {
posix_spawn_file_actions_addopen(&io, 0, "/dev/null", O_RDONLY, 0);
posix_spawn_file_actions_addopen(&io, 1, "/tmp/frametop-layout.log", O_WRONLY | O_CREAT | O_APPEND, 0644);
posix_spawn_file_actions_adddup2(&io, 1, 2);
char scale[] = "scale";
char *argv[] = {layout.data(), scale, nullptr};
std::string arg(cmd);
char *argv[] = {layout.data(), arg.data(), nullptr};
pid_t pid; // reaped by the compositor's SIGCHLD handler
if (posix_spawn(&pid, layout.c_str(), &io, nullptr, argv, environ) != 0)
std::printf("can't run %s\n", layout.c_str());
posix_spawn_file_actions_destroy(&io);
}
// KWin's outputs follow where the screens are, so the pointer and dragged windows cross
// to the screen you see next to this one: `ft-layout scale` runs once a move has settled.
long g_arrangeAt = -1; // g_tick to run it at, -1 = not pending
void ArrangeDesktopSoon() { g_arrangeAt = g_tick + 45; } // about half a second
void UpdateArrange() {
if (g_arrangeAt < 0 || g_tick < g_arrangeAt) return;
g_arrangeAt = -1;
RunLayout("scale");
}
// Let go: pin to the armed wrist, as the screen is now.
void FinishDrag(Screen &s, int index) {
const bool moved = s.drag == Drag::Move;
@@ -1241,6 +1375,7 @@ void SetSub(Screen &s, int index, int k, int x, int y, int w, int h) {
vr::VROverlay()->SetOverlayFlag(sub.overlay, vr::VROverlayFlags_SendVRDiscreteScrollEvents, true);
vr::VROverlay()->SetOverlayFlag(sub.overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, s.lasers);
vr::VROverlay()->SetOverlaySortOrder(sub.overlay, 5);
AnnounceOverlay(key);
it = s.subs.emplace(k, sub).first;
if (s.shown) {
auto imp = g_imports.find(s.shown);
@@ -1274,6 +1409,7 @@ struct Press {
double x = 0, y = 0; // ...and where on it, in buffer pixels
double distance = 1; // from the laser's start to the last panel it met
long upAt = -1; // the pointer helper saw left come up: release it at this tick
int64_t idleSince = -1; // the pressing controller has held nothing since (ms, ReleaseStuck)
};
Press g_press;
vr::VROverlayHandle_t g_catcher = vr::k_ulOverlayHandleInvalid;
@@ -1310,6 +1446,47 @@ void ReleaseAway(uint32_t button, void (*handle)(const struct ft_event *, void *
handle(&e, data);
}
// Whether a hand controller holds anything (a button down): 1 yes, 0 no, -1 unknown. SteamVR
// answers overlay apps only while a VR game runs (checked 2026-10-04); outside games it can't
// say. The Frame controller's axes have no types, so the buttons are all there is.
int ControllerHolds(vr::TrackedDeviceIndex_t dev) {
vr::VRControllerState_t st{};
if (!vr::VRSystem()->GetControllerState(dev, &st, sizeof st)) return -1;
return st.ulButtonPressed ? 1 : 0;
}
// Releases SteamVR never sends. Pausing, or hiding the screen a button went down on, takes the
// laser off it mid-click (the pause gesture's second thumbstick click does that), and SteamVR's
// laser mouse then forgets the button: no release comes, the catcher stayed up, and the game
// never got its controllers back (2026-10-04). So a held button is released here when it has
// no visible screen to come up on, or when its hand controller has held nothing for kStuckMs
// (only known during VR games, where the lost release took the game's controllers).
// The pointer helper's virtual controller has its own word for that ("up").
constexpr int64_t kStuckMs = 1000;
void ReleaseStuck(void (*handle)(const struct ft_event *, void *), void *data) {
if (!g_press.buttons) {
g_press.idleSince = -1;
return;
}
const char *why = nullptr;
const auto it = g_screens.find(g_press.screen);
if (g_paused) why = "paused";
else if (g_press.screen >= 0 && (it == g_screens.end() || !it->second.visible)) why = "its screen hid";
else if (IsHandController(g_press.device)) {
const int64_t now = NowMs();
if (ControllerHolds(g_press.device) != 0) g_press.idleSince = -1;
else if (g_press.idleSince < 0) g_press.idleSince = now;
else if (now - g_press.idleSince >= kStuckMs) why = "the controller holds nothing";
}
if (!why) return;
std::printf("a held button can't come up on a screen (%s): released\n", why);
const vr::TrackedDeviceIndex_t dev = g_press.device;
for (uint32_t b = 0; b < 32; ++b)
if (g_press.buttons & (1u << b)) ReleaseAway(BTN_LEFT + b, handle, data);
g_press.buttons = 0, g_press.upAt = -1, g_press.idleSince = -1;
EndDragsBy(dev);
}
void ShowCatcher(bool on) {
if (on == g_catcherShown || g_catcher == vr::k_ulOverlayHandleInvalid) return;
g_catcherShown = on;
@@ -1355,6 +1532,137 @@ Screen *Find(int one_based) {
return it == g_screens.end() ? nullptr : &it->second;
}
// ---------------------------------------------------------------- the lazy susan
// "spin next|prev|<degrees>": the panels in the room (screens and floating windows, not
// pinned ones) turn together about a vertical axis through your head, so the next panel to
// your right (next) or left (prev) glides to straight ahead, or the ring turns by that many
// degrees (positive turns it left, like next). Their arrangement stays as it is: the room
// turns instead of you. A spin that arrives during one adds to it, from where the panels are
// headed, so quick taps carry on smoothly. Grabbing a panel, or a command that places it,
// takes it out of the spin where it is. The 3D mouse's pointer goes to straight ahead.
constexpr double kSpinSeconds = 0.3; // how long a spin takes
constexpr double kSpinAhead = 8; // degrees: a panel this near straight ahead is the current one
constexpr double kSpinFocus = 30; // degrees: when a spin settles, the panel this near ahead gets typing
struct Spin {
bool on = false;
double cx = 0, cz = 0; // the axis
double from = 0, to = 0; // radians, turned from the poses in base (positive: to the left)
Clock::time_point start;
std::map<int, Mat> base; // index -> its pose before the spin
int front = -1; // settled: this panel came to the front (ft_vr_poll reports it)
} g_spin;
// p turned a radians about the vertical axis through (cx, cz); positive turns it to the left.
Mat Turned(const Mat &p, double a, double cx, double cz) {
const double c = std::cos(a), s = std::sin(a);
Mat r = Identity();
r.m[0][0] = float(c), r.m[0][2] = float(s);
r.m[2][0] = float(-s), r.m[2][2] = float(c);
r.m[0][3] = float(cx - c * cx - s * cz);
r.m[2][3] = float(cz + s * cx - c * cz);
return Mul(r, p);
}
bool Spinnable(const Screen &s) { return s.pinned == kNone && (!s.floating || s.floatOn) && s.drag == Drag::None; }
double SpinNow() {
if (!g_spin.on) return g_spin.to;
const double t = std::min(1.0, std::chrono::duration<double>(Clock::now() - g_spin.start).count() / kSpinSeconds);
return g_spin.from + (g_spin.to - g_spin.from) * t * t * (3 - 2 * t);
}
void UpdateSpin() {
if (!g_spin.on) return;
const double a = SpinNow();
const bool done = Clock::now() - g_spin.start >= std::chrono::duration<double>(kSpinSeconds);
for (auto it = g_spin.base.begin(); it != g_spin.base.end();) {
auto s = g_screens.find(it->first);
if (s == g_screens.end() || !Spinnable(s->second)) {
it = g_spin.base.erase(it); // grabbed, pinned, or gone: it stays where it is now
continue;
}
SetAbsolute(s->second, Turned(it->second, a, g_spin.cx, g_spin.cz));
++it;
}
if (done) {
// The panel now nearest straight ahead (of where you face) gets typing and the active window.
Mat head;
double nearest = kSpinFocus;
if (DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head)) {
for (const auto &[i, p] : g_spin.base) {
const auto it = g_screens.find(i);
Mat q;
if (it == g_screens.end() || !it->second.visible || !ScreenPose(it->second, &q)) continue;
double f[3] = {-head.m[0][2], 0, -head.m[2][2]};
double d[3] = {q.m[0][3] - head.m[0][3], 0, q.m[2][3] - head.m[2][3]};
const double fl = std::sqrt(Dot3(f, f)), dl = std::sqrt(Dot3(d, d));
if (fl < 1e-6 || dl < 1e-6) continue;
const double a = std::acos(std::clamp(Dot3(f, d) / (fl * dl), -1.0, 1.0)) * 180 / M_PI;
if (a < nearest) nearest = a, g_spin.front = i;
}
}
g_spin.on = false;
g_spin.base.clear();
ArrangeDesktopSoon(); // KWin's outputs follow where the screens are now
}
}
void SpinCommand(const char *arg, char *reply, int size) {
Mat head;
if (!DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head))
return (void)std::snprintf(reply, size, "error no head pose (headset off?)");
if (g_spin.on) {
g_spin.from = SpinNow(); // carry on from where the panels are now
} else {
g_spin.base.clear();
for (auto &[i, s] : g_screens) {
Mat p;
if (Spinnable(s) && ScreenPose(s, &p)) g_spin.base[i] = p;
}
g_spin.cx = head.m[0][3], g_spin.cz = head.m[2][3];
g_spin.from = g_spin.to = 0;
}
if (g_spin.base.empty()) return (void)std::snprintf(reply, size, "error nothing to spin");
double turn; // degrees, positive to the left
const bool next = !std::strcmp(arg, "next");
if (next || !std::strcmp(arg, "prev")) {
// Each visible panel's bearing from where you face, to the right positive, as it will
// be when the spin so far ends; the nearest one past straight ahead comes to the front.
double fx = -head.m[0][2], fz = -head.m[2][2];
const double n = std::sqrt(fx * fx + fz * fz) + 1e-9;
fx /= n, fz /= n;
const double rx = -fz, rz = fx;
double best = 0;
bool found = false;
for (const auto &[i, p] : g_spin.base) {
const auto s = g_screens.find(i);
if (s == g_screens.end() || !s->second.visible) continue;
const Mat q = Turned(p, g_spin.to, g_spin.cx, g_spin.cz);
const double dx = q.m[0][3] - head.m[0][3], dz = q.m[2][3] - head.m[2][3];
double a = std::atan2(dx * rx + dz * rz, dx * fx + dz * fz) * 180 / M_PI;
if (!next) a = -a;
if (a <= kSpinAhead) a += 360;
if (!found || a < best) best = a, found = true;
}
if (!found || best >= 360 - kSpinAhead) {
if (!g_spin.on) g_spin.base.clear();
return (void)std::snprintf(reply, size, "ok 0 (no other panel)");
}
if (best > 180) best -= 360; // the short way round
turn = next ? best : -best;
} else {
char *end;
turn = std::strtod(arg, &end);
if (end == arg || *end) return (void)std::snprintf(reply, size, "error spin next|prev|<degrees>");
}
g_spin.to += turn * M_PI / 180;
g_spin.start = Clock::now();
g_spin.on = true;
SendPointer("recenter"); // the 3D mouse's pointer stays in front of you, on what comes there
std::snprintf(reply, size, "ok %.1f", turn);
}
uint32_t LinuxButton(uint32_t vrButton) {
switch (vrButton) {
case vr::VRMouseButton_Right: return BTN_RIGHT;
@@ -1369,6 +1677,81 @@ void ReleaseAwayBy(vr::TrackedDeviceIndex_t dev, uint32_t vrButton, void (*handl
if (g_press.buttons && dev == g_press.device) ReleaseAway(LinuxButton(vrButton), handle, data);
}
// Where a laser meets a panel's surface, in the panel's u (metres along it from the centre,
// along the arc when curved) and v (up). OpenVR curves a screen into a cylinder toward its
// front, centred `curve` metres in front of it (see OnSurface).
bool RayOnSurface(const Screen &s, const Mat &p, const Mat &laser, double *u, double *v) {
const Mat inv = Inverse(p);
const double o[3] = {inv.m[0][0] * laser.m[0][3] + inv.m[0][1] * laser.m[1][3] + inv.m[0][2] * laser.m[2][3] + inv.m[0][3],
inv.m[1][0] * laser.m[0][3] + inv.m[1][1] * laser.m[1][3] + inv.m[1][2] * laser.m[2][3] + inv.m[1][3],
inv.m[2][0] * laser.m[0][3] + inv.m[2][1] * laser.m[1][3] + inv.m[2][2] * laser.m[2][3] + inv.m[2][3]};
double d[3];
for (int i = 0; i < 3; ++i) d[i] = -(inv.m[i][0] * laser.m[0][2] + inv.m[i][1] * laser.m[1][2] + inv.m[i][2] * laser.m[2][2]);
if (s.curve <= 0) {
if (std::fabs(d[2]) < 1e-6) return false;
const double t = -o[2] / d[2];
if (t <= 0) return false;
*u = o[0] + d[0] * t, *v = o[1] + d[1] * t;
return true;
}
// x^2 + (z - r)^2 = r^2, on the screen's side of the axis (z < r).
const double r = s.curve, oz = o[2] - r;
const double a = d[0] * d[0] + d[2] * d[2], b = 2 * (o[0] * d[0] + oz * d[2]), c = o[0] * o[0] + oz * oz - r * r;
const double disc = b * b - 4 * a * c;
if (a < 1e-9 || disc < 0) return false;
for (double t : {(-b - std::sqrt(disc)) / (2 * a), (-b + std::sqrt(disc)) / (2 * a)}) {
const double x = o[0] + d[0] * t, z = oz + d[2] * t;
if (t <= 0 || z >= 0) continue;
*u = r * std::atan2(x, -z), *v = o[1] + d[1] * t;
return true;
}
return false;
}
// Where the mode leaves the controllers to a VR game (see the top): a hand controller
// pointing at a panel, its controls, or a floating window's popups keeps that panel's laser
// on (UpdateLasers) until kAimLinger ticks after it points away, like SteamVR's own floating
// windows. Leaving takes a wider margin than arriving, and a drag or a held button keeps it
// on. The keyboard is one overlay, so SteamVR's own intersection test does there.
void UpdateAim() {
if (LasersByMode()) return;
std::vector<Mat> lasers;
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
Mat d;
if (IsHandController(i) && LaserPose(i, &d)) lasers.push_back(d);
}
for (auto &[index, s] : g_screens) {
Mat p;
if (!s.visible || !ScreenPose(s, &p)) continue;
if (s.drag != Drag::None || (g_press.buttons && g_press.screen == index)) {
s.aimUntil = g_tick + kAimLinger;
continue;
}
const double m = s.grip * (g_tick < s.aimUntil ? 2.0 : 0.25), h = s.heightMetres();
// The panel and its controls: the bar row under it, the resize tab off its corner.
const double halfW = std::max(s.metres / 2 + s.grip, s.chrome / 2 + s.chrome * 0.12 + s.grip * 2) + m;
const double top = h / 2 + m, bottom = std::min(BarY(s) - s.grip, -(h / 2 + s.grip)) - m;
for (const Mat &l : lasers) {
double u, v;
if (!RayOnSurface(s, p, l, &u, &v)) continue;
bool on = std::fabs(u) <= halfW && v <= top && v >= bottom;
for (const auto &[k, sub] : s.subs) {
if (on || s.cropW <= 0) break;
const double su = (sub.x + sub.w / 2.0 - (s.cropX + s.cropW / 2.0)) * s.mpp;
const double sv = -(sub.y + sub.h / 2.0 - (s.cropY + s.cropH / 2.0)) * s.mpp;
on = std::fabs(u - su) <= sub.w * s.mpp / 2 + m && std::fabs(v - sv) <= sub.h * s.mpp / 2 + m;
}
if (on) {
s.aimUntil = g_tick + kAimLinger;
break;
}
}
}
if (keyboard::Shown())
for (const Mat &l : lasers)
if (keyboard::Aimed(l)) g_keyboardAimUntil = g_tick + kAimLinger;
}
const char *LasersName() {
switch (g_lasers) {
case Lasers::Always: return "always";
@@ -1501,7 +1884,7 @@ void UpdateSteamInFront() {
g_keyboardAside = true;
} else if (!front && g_keyboardAside) {
g_keyboardAside = false;
keyboard::Show(g_asidePose);
if (keyboard::Show(g_asidePose)) AnnounceOverlay("frametop.keyboard");
}
}
@@ -1569,6 +1952,25 @@ int ft_vr_modifiers(uint32_t format, uint64_t *out, int max) {
}
bool ft_vr_screens_shown(void) { return g_vr && ModeVisible(); }
bool ft_vr_paused(void) { return g_paused; }
enum ft_attention ft_vr_screen_attention(int index) {
const auto it = g_screens.find(index);
return g_vr && it != g_screens.end() ? it->second.attention : FT_FOCUSED;
}
bool ft_vr_vsync(double *since, double *hz) {
if (!g_vr) return false;
float s = 0;
uint64_t frame = 0;
if (!vr::VRSystem()->GetTimeSinceLastVsync(&s, &frame)) return false;
vr::ETrackedPropertyError err = vr::TrackedProp_Success;
const float f = vr::VRSystem()->GetFloatTrackedDeviceProperty(vr::k_unTrackedDeviceIndex_Hmd,
vr::Prop_DisplayFrequency_Float, &err);
if (err != vr::TrackedProp_Success || !(f >= 30 && f <= 240) || !(s >= 0 && s < 1)) return false;
*since = s, *hz = f;
return true;
}
} // extern "C"
@@ -1608,6 +2010,9 @@ bool MakePanel(Screen &s, const char *prefix, const char *label) {
static const auto close = CloseTexture(64);
s.dockButton = chrome("dock", "back to the desktop", dock, 64, 64);
s.closeButton = chrome("close", "close", close, 64, 64);
} else {
static const auto reset = ResetTexture(64);
s.resetButton = chrome("reset", "reset the layout", reset, 64, 64);
}
ApplyAlpha(s);
return true;
@@ -1728,6 +2133,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
auto panelEvent = [&](const vr::VREvent_t &ev, bool sub) {
ft_event e{};
e.screen = index;
if (ev.eventType != vr::VREvent_FocusLeave) s.inputMs = NowMs();
auto at = [&] { s.ToBuffer(ev.data.mouse.x, ev.data.mouse.y, &e.x, &e.y); };
switch (ev.eventType) {
case vr::VREvent_MouseMove:
@@ -1780,6 +2186,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
// The controls light up under a laser.
auto hover = [&](int k) {
const bool on = ev.eventType == vr::VREvent_MouseMove || ev.eventType == vr::VREvent_FocusEnter;
if (on) s.inputMs = NowMs();
if (!on && ev.eventType != vr::VREvent_FocusLeave) return;
if (s.hover[k] != on) s.hover[k] = on, ApplyAlpha(s);
};
@@ -1829,6 +2236,19 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
}
}
}
// The reset button: every screen back in the layout, around where you are now
// (`ft-layout apply`, like Meta+Shift+R; it refuses a second copy).
while (s.resetButton != vr::k_ulOverlayHandleInvalid &&
vr::VROverlay()->PollNextOverlayEvent(s.resetButton, &ev, sizeof ev)) {
hover(6);
if (ev.eventType == vr::VREvent_MouseButtonDown && ev.data.mouse.button == vr::VRMouseButton_Left) {
std::printf("screen %d: reset the layout\n", index + 1);
RunLayout("apply");
} else if (ev.eventType == vr::VREvent_MouseButtonUp) {
EndDragsBy(ev.trackedDeviceIndex);
ReleaseAwayBy(ev.trackedDeviceIndex, ev.data.mouse.button, handle, data);
}
}
// The roll button: drag around like a knob, or scroll.
while (vr::VROverlay()->PollNextOverlayEvent(s.rollButton, &ev, sizeof ev)) {
hover(2);
@@ -1854,6 +2274,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
if (ev.eventType == vr::VREvent_MouseButtonUp)
ReleaseAwayBy(ev.trackedDeviceIndex, ev.data.mouse.button, handle, data);
if (g_press.upAt >= 0 && g_tick >= g_press.upAt) ReleaseAway(BTN_LEFT, handle, data);
ReleaseStuck(handle, data);
RefreshChrome();
while (vr::VRSystem()->PollNextEvent(&ev, sizeof ev)) {
if (ev.eventType == vr::VREvent_Quit) {
@@ -1889,9 +2310,19 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
handle(&e, data);
}
++g_tick;
UpdateSpin();
if (g_spin.front >= 0) {
ft_event e{};
e.type = FT_FRONT;
e.screen = g_spin.front;
g_spin.front = -1;
handle(&e, data);
}
UpdateGame();
UpdateArrange();
UpdateVisibility();
UpdateAttention();
UpdateAim();
UpdateLasers();
UpdateControls();
UpdateGuides();
@@ -1918,7 +2349,7 @@ bool ft_vr_keyboard_show(int index) {
const double fx = -head.m[0][2], fz = -head.m[2][2], n = std::sqrt(fx * fx + fz * fz) + 1e-9;
const double at[3] = {head.m[0][3] + fx / n * kKeyboardAhead, head.m[1][3] - kKeyboardBelow,
head.m[2][3] + fz / n * kKeyboardAhead};
keyboard::SetLasers(g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning));
keyboard::SetLasers(LasersByMode());
g_steamInFront = SteamInFront();
if (g_steamInFront) {
g_asidePose = FacingPose(at, head);
@@ -1926,7 +2357,9 @@ bool ft_vr_keyboard_show(int index) {
std::printf("keyboard: waiting for Steam to close\n");
return true;
}
return keyboard::Show(FacingPose(at, head));
if (!keyboard::Show(FacingPose(at, head))) return false;
AnnounceOverlay("frametop.keyboard");
return true;
}
void ft_vr_keyboard_hide(void) {
@@ -1961,6 +2394,8 @@ void ft_vr_keyboard_hide(void) {
// ~100 ms (it landed on something else), KWin gets it anyway
// state -> "ok <mode> <manual 0|1> <wrist deg> <gesture hand> <gesture deg>
// <controllers> <game running 0|1> <ingames>"
// spin next|prev|<degrees> turn every panel in the room about your head (see the lazy
// susan) -> "ok <degrees turned>"
// cutouts on|off|state hand cutouts (see handcut.h) -> "ok <on|off> <ready|idle|unavailable>
// <last composite ms> ms, predict <on|off> lead <ms> ms"
// cutouts predict on|off move the hands ahead along their velocity (on by default)
@@ -1998,6 +2433,7 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
Screen *s = Find(n);
if (!s) return (void)std::snprintf(reply, size, "error no screen %d", n);
EndDrag(*s);
g_spin.base.erase(n - 1);
SetAbsolute(*s, PanelPose(x, y, z, yaw, pitch, roll));
std::snprintf(reply, size, "ok");
} else if (std::sscanf(cmd, "width %d %lf", &n, &w) == 2) {
@@ -2104,6 +2540,17 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
g_manual = always ? !want : want;
UpdateVisibility();
std::snprintf(reply, size, "ok %s", want ? "shown" : "hidden");
} else if (std::sscanf(cmd, "pause %15s", word) == 1) {
if (!std::strcmp(word, "on") || !std::strcmp(word, "off")) {
const bool on = !std::strcmp(word, "on");
if (on != g_paused)
std::printf("%s\n", on ? "paused for a VR game: everything hidden, KWin slowed down" : "resumed");
g_paused = on;
UpdateVisibility();
} else if (std::strcmp(word, "state") != 0) {
return (void)std::snprintf(reply, size, "error pause on|off|state");
}
std::snprintf(reply, size, "ok %s", g_paused ? "paused" : "running");
} else if (std::sscanf(cmd, "ingames %15s", word) == 1) {
if (!std::strcmp(word, "hide")) g_inGames = InGames::Hide;
else if (!std::strcmp(word, "visible")) g_inGames = InGames::Visible;
@@ -2153,6 +2600,7 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
Mat m{};
for (int k = 0; k < 12; ++k) m.m[k / 4][k % 4] = r[k];
EndDrag(*s);
g_spin.base.erase(n - 1);
SetAbsolute(*s, m);
std::snprintf(reply, size, "ok");
} else if (int k0; std::sscanf(cmd, "sub %d %d %d %d %d %d", &n, &k0, &x0, &y0, &w0, &h0) == 6 ||
@@ -2180,6 +2628,8 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
if ((g_press.buttons & ButtonBit(BTN_LEFT)) && g_press.device != kNone && !IsHandController(g_press.device))
g_press.upAt = g_tick + 9;
std::snprintf(reply, size, "ok");
} else if (std::sscanf(cmd, "spin %15s", word) == 1) {
SpinCommand(word, reply, size);
} else if (std::strncmp(cmd, "state", 5) == 0) {
std::snprintf(reply, size, "ok %s %d %.0f %s %.0f %s %d %s", ModeName(), g_manual ? 1 : 0, g_wristAngle,
g_gestureHand.c_str(), g_gestureAngle, LasersName(), g_gameRunning ? 1 : 0,
+12 -1
View File
@@ -17,7 +17,8 @@ struct ft_dmabuf {
int fd[4];
};
enum ft_event_type { FT_MOTION, FT_BUTTON, FT_SCROLL, FT_LEAVE, FT_QUIT, FT_KEY, FT_KEYBOARD_CLOSED };
// FT_FRONT: a spin (the lazy susan) brought panel `screen` to straight ahead: typing goes there.
enum ft_event_type { FT_MOTION, FT_BUTTON, FT_SCROLL, FT_LEAVE, FT_QUIT, FT_KEY, FT_KEYBOARD_CLOSED, FT_FRONT };
struct ft_event {
enum ft_event_type type;
@@ -36,6 +37,16 @@ void ft_vr_shutdown(void);
int ft_vr_modifiers(uint32_t format, uint64_t *out, int max);
// The screens are showing (by the visibility mode; not counting a wrist-pinned screen).
bool ft_vr_screens_shown(void);
// Frametop is paused for a VR game ("pause on"): everything is hidden, and KWin slows down.
bool ft_vr_paused(void);
// How much of a screen you see, for its frame rate (compositor.c): hidden (or out of view),
// in view, or focused (you look at it, or a laser or the mouse is on it). Focused without
// SteamVR (--no-vr) or for an unknown screen.
enum ft_attention { FT_HIDDEN, FT_IN_VIEW, FT_FOCUSED };
enum ft_attention ft_vr_screen_attention(int index);
// The display's refresh rate and the time since its last vsync, in seconds. False without
// them (no SteamVR, or the headset isn't reporting).
bool ft_vr_vsync(double *since, double *hz);
// A panel for screen `index`, width in metres, placed in a row in front of the head.
void ft_vr_screen_create(int index, double metres, int count);
void ft_vr_screen_destroy(int index);
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Update the lines of ~/.config/frametop.conf that still read exactly as an older
# frametop.conf.example wrote them. A line you changed stays as it is. install.sh and
# hands/rec/install.sh run this on every install and update.
# HANDS_SWAP_SIDES=0: the example's value until 2026-10-05. It made ft-hands keep ft-camd's side
# camera names even when they're backwards (some SteamVR restarts swap them), so hands landed
# beside their cutouts and the hand recorder labelled the side cameras wrong. auto tells from
# the hands.
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
. "$root/scripts/_env.sh"
on_frame_script <<'EOF'
f=~/.config/frametop.conf
[ -f "$f" ] || exit 0
old="HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)"
new="HANDS_SWAP_SIDES=auto # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them"
if grep -qxF "$old" "$f"; then
tmp=$(mktemp)
awk -v old="$old" -v new="$new" '$0 == old { print new; next } { print }' "$f" > "$tmp"
cat "$tmp" > "$f" # in place: the file keeps its owner and mode
rm -f "$tmp"
echo "settings: HANDS_SWAP_SIDES=0 (the old default) is now auto: hand tracking tells the side cameras apart itself"
fi
EOF
+4
View File
@@ -36,6 +36,10 @@ check "distrobox" on_frame 'test -x ~/.local/bin/distrobox && ~/.local/bin/distr
check "container $FRAME_BOX" on_frame "podman ps -a --filter name=^$FRAME_BOX\$ --format '{{.Image}} {{.Status}}' | grep ."
check "repo on the Frame" on_frame 'pwd'
check "free space in ~" on_frame "df -h ~ | awk 'NR==2{print \$4\" free\"}'"
# A taskbar saved on a screen the desktop doesn't have is hidden; the desktop's next start
# moves it to the first screen (session/fix-panels.py).
check "taskbar on a screen" on_frame 'set -o pipefail; [ -f session/fix-panels.py ] || { echo "not checked (older checkout)"; exit 0; }
python3 session/fix-panels.py --check | paste -sd ";" | sed "s/;/; /g"'
echo "what Frametop needs from SteamOS:"
if [ "$FRAME_LOCAL" = 1 ]; then
+44 -1
View File
@@ -25,7 +25,9 @@ for u in frametop-input-relay frametop-pointer frametop-power; do
echo "$u: $(systemctl --user is-enabled $u 2>/dev/null) / $(systemctl --user is-active $u 2>/dev/null)"
done
echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
# SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md).
XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
section "What Frametop needs from SteamOS (scripts/update-check.py)"
python3 "$repo/scripts/update-check.py" 2>&1 || true
@@ -41,12 +43,53 @@ for i, s in enumerate(d.get("screens", []), 1):
print("visibility:", d.get("visibility"))
PY
section "Plasma panels and outputs"
python3 "$repo/session/fix-panels.py" --check 2>&1 || true
python3 - "$repo" <<'PY' 2>&1 || true
import json, subprocess, sys
sys.path.insert(0, sys.argv[1] + "/layout")
import ft_layout
env = ft_layout.nested_env()
if not env:
sys.exit(print("desktop not running: no live outputs or panels"))
outs = json.loads(subprocess.run(["kscreen-doctor", "-j"], capture_output=True, text=True, env=env, timeout=10).stdout or "{}")
for o in outs.get("outputs", []):
size = o.get("size") or {}
print(f"output {o.get('name')}: enabled={o.get('enabled')} priority={o.get('priority')} {size.get('width')}x{size.get('height')}")
js = "print(JSON.stringify(panels().map(p => ({id: p.id, screen: p.screen, location: p.location}))))"
r = subprocess.run(["qdbus6", "org.kde.plasmashell", "/PlasmaShell", "org.kde.PlasmaShell.evaluateScript", js],
capture_output=True, text=True, env=env, timeout=10)
print("live panels:", (r.stdout or r.stderr).strip())
PY
section "Input relay (last 60 lines)"
journalctl --user -u frametop-input-relay -n 60 --no-pager -o short 2>/dev/null
section "Pointer helper (last 60 lines)"
journalctl --user -u frametop-pointer -n 60 --no-pager -o short 2>/dev/null
section "Power service (last 30 lines)"
journalctl --user -u frametop-power -n 30 --no-pager -o short 2>/dev/null
section "Gaze"
echo "frametop-gaze: $(systemctl --user is-enabled frametop-gaze 2>/dev/null) / $(systemctl --user is-active frametop-gaze 2>/dev/null)"
echo "frametop-eyegrab (our eye tracker's frame grabber): $(systemctl is-enabled frametop-eyegrab 2>/dev/null) / $(systemctl is-active frametop-eyegrab 2>/dev/null)"
for f in calibration.json eyes/calibration.json; do # SteamVR's correction, our tracker's calibration
p=~/.local/state/frametop/gaze/$f
echo "$f: $([ -f "$p" ] && date -r "$p" '+%F %T' || echo none)"
done
python3 - "$repo" <<'PY' 2>&1 || true
import json, subprocess, sys
out = subprocess.run([sys.executable, sys.argv[1] + "/gaze/ft-gazectl", "status"], capture_output=True, text=True, timeout=10)
text = (out.stdout or out.stderr).strip()
try:
print("status:", json.dumps(json.loads(text), separators=(",", ":")))
except ValueError:
print("status:", text or "no answer from the gaze service")
PY
section "Gaze checks and calibration dots (last 10)"
tail -n 10 ~/.local/state/frametop/gaze/checks.jsonl 2>/dev/null | cut -c1-400 || true
section "Gaze service (last 60 lines, podman's left out)"
journalctl --user -u frametop-gaze -n 300 --no-pager -o short 2>/dev/null | grep -v ' podman\[' | tail -n 60
section "Our eye tracker's frame grabber (last 20 lines)"
journalctl -u frametop-eyegrab -n 20 --no-pager -o short 2>/dev/null
for f in /tmp/frametop-session.log /tmp/frametop-screens.log /tmp/frametop-layout.log; do
section "$f (last 60 lines)"
tail -n 60 "$f" 2>/dev/null || echo "missing"
+117 -9
View File
@@ -12,15 +12,18 @@ This checks each one, and compares the versions with the last ones recorded as w
From a PC: ssh frame python3 - [--mark-good] < scripts/update-check.py
It only reads. It connects to SteamVR as a background app (which never starts SteamVR),
runs `vrcmd --overlays` as the pointer helper does, and maps the eye tracker's shared
memory read-only. It never starts, stops, or restarts anything.
runs `vrcmd --overlays` as the pointer helper does, maps the eye tracker's shared memory
read-only, and follows a controller on vrserver's web socket for a moment. It never starts,
stops, or restarts anything.
"""
import base64
import json
import math
import mmap
import os
import platform
import re
import socket
import struct
import subprocess
import sys
@@ -32,8 +35,13 @@ KNOWN_GOOD = os.path.join(HOME, ".local/state/frametop/known-good.json")
STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64"
LAUNCHER = os.path.join(HOME, ".local/share/applications/deckard-nested-desktop.desktop")
VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath")
# The Frametop desktop has its own XDG_CONFIG_HOME (session/frametop-session.sh). An install from a
# terminal there, before pointer/driver/install.sh set SteamVR's, left vrpathreg's registry here.
STRAY_VRPATHS = os.path.join(HOME, ".config/frametop/openvr/openvrpaths.vrpath")
BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness" # as in power/ft-powerd.cpp
EYE_MMAP = "/dev/shm/eye-server.mmap"
VRSERVER = "http://127.0.0.1:27062" # its web socket: input/vrws.py
PAUSE_STATE = f"/run/user/{os.getuid()}/frametop-pause.json" # input/game_pause.py
HOST_GLIBC = (2, 39) # the newest the pointer driver may need (pointer/driver/build.sh)
# Packages in the OS image that Frametop depends on, and what to try by hand when one changes.
@@ -41,8 +49,11 @@ PACKAGES = {
"deckard-steamvr-rel": "the 3D mouse on the dashboard, on SteamVR Settings, and on a SteamVR "
"window's grab bar; picking up a controller; mapped controller buttons; gaze",
"kwin": "clicks near the far edge of a screen whose scale isn't 1; every screen comes back "
"after a desktop restart; floating a window",
"plasma-workspace": "the taskbar and panels after a desktop restart",
"after a desktop restart; floating a window; no blur behind the taskbar's menus",
"plasma-workspace": "the taskbar and panels after a desktop restart; no DiscoverNotifier or "
"ibus-daemon inside the desktop",
"at-spi2-core": "an AT-SPI-aware app appears on the nested desktop's accessibility bus; "
"the registry stops and comes back after a desktop restart",
"gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked",
"bluez": "a Bluetooth mouse reconnecting after it sleeps",
}
@@ -156,6 +167,14 @@ def check_host():
("/usr/bin/kwin_wayland_wrapper", "FAIL", "the desktop can't start KWin"),
("/usr/bin/startplasma-wayland", "FAIL", "the desktop can't start Plasma"),
("/usr/bin/dbus-run-session", "FAIL", "the desktop can't start its session bus"),
("/usr/bin/python3", "warn", "nested accessibility can't run its startup helper"),
("/usr/bin/gdbus", "warn", "nested accessibility can't discover or check its bus"),
("/usr/lib/at-spi-bus-launcher", "warn", "nested accessibility can't start its bus"),
("/usr/lib/at-spi2-registryd", "warn", "nested accessibility has no fallback registry"),
("/usr/share/dbus-1/services/org.a11y.Bus.service", "warn",
"nested accessibility can't activate its bus"),
("/usr/share/dbus-1/accessibility-services/org.a11y.atspi.Registry.service", "warn",
"nested accessibility can't activate its registry natively"),
(f"{STEAMVR_BIN}/vrcmd", "FAIL", "the 3D mouse can't find panels"),
(f"{STEAMVR_BIN}/vrpathreg", "warn", "the pointer driver can't be installed or removed"),
("/usr/share/deckard/mesavars.sh", "warn", "the desktop starts without SteamOS's Mesa settings"),
@@ -171,14 +190,21 @@ def check_host():
try:
with open("/usr/bin/kwin_wayland", "rb") as f:
# Qt keeps the option name as a UTF-16 string literal.
output_count = "output-count".encode("utf-16-le") in f.read()
kwin = f.read()
except OSError:
output_count = False
if output_count:
kwin = b""
# Qt keeps the option name as a UTF-16 string literal.
if "output-count".encode("utf-16-le") in kwin:
report("ok", "KWin", "has --output-count (one output per screen)")
else:
report("FAIL", "KWin", "no --output-count option: the desktop gets one screen at most")
# The session turns these built-in effects off by id (blurEnabled, contrastEnabled in kwinrc).
gone = [e for e in ("blur", "contrast") if b"KWin::%s_factory" % e.encode() not in kwin]
if not gone:
report("ok", "KWin effects", "blur and contrast, which the desktop turns off, keep their ids")
else:
report("warn", "KWin effects", f"no built-in {', '.join(gone)} effect: renamed? The desktop's "
"kwinrc may no longer turn it off (session/frametop-session.sh)")
if systemctl("cat", "steamvr.service"):
report("ok", "steamvr.service", "Frametop's services start and stop with it")
@@ -204,6 +230,14 @@ def check_host():
else:
report("FAIL", "pointer driver", "not registered with SteamVR; run pointer/driver/install.sh install, "
"then restart SteamVR")
try:
with open(STRAY_VRPATHS) as f:
stray = json.load(f).get("runtime") is None
except (OSError, ValueError, AttributeError):
stray = False
if stray:
report("warn", "OpenVR path registry", f"{STRAY_VRPATHS} has no SteamVR in it, and hides SteamVR from "
"OpenVR programs started in the Frametop desktop; pointer/driver/install.sh install removes it")
session = launcher_session()
if session is None:
@@ -297,11 +331,19 @@ def check_overlays(desktop_up):
def check_services(sockets, desktop_up):
try:
with open(PAUSE_STATE) as f:
pause = json.load(f)
paused = pause["stopped"]["units"] if pause.get("paused") else []
except (OSError, ValueError, KeyError, TypeError):
paused = []
for unit, socket in UNITS.items():
if systemctl("is-enabled", unit) != "enabled":
continue
state = systemctl("is-active", unit)
if state != "active":
if state != "active" and unit + ".service" in paused:
report("skip", unit, "stopped while Frametop is paused for a VR game (input/ft-pause off resumes it)")
elif state != "active":
report("FAIL", unit, f"{state}; see journalctl --user -u {unit}")
elif socket and socket not in sockets:
report("FAIL", unit, f"runs, but hasn't opened {socket}")
@@ -345,6 +387,69 @@ def check_steam_ui():
report("warn", "Steam menu shortcut", f"{detail}; a Meta tap won't open the Steam menu (steam/ft-steam)")
def check_vr_socket():
"""vrserver's web socket, which the pause gesture reads the controllers from (input/vrws.py,
input/game_pause.py). It's undocumented: SteamVR's controller binding page uses it."""
label, headers = "vrserver web socket", {"Referer": f"{VRSERVER}/dashboard/controllerbinding.html"}
try:
import urllib.request
with urllib.request.urlopen(urllib.request.Request(f"{VRSERVER}/input/getstate.json", headers=headers),
timeout=3) as r:
devices = json.load(r).get("devices", [])
except (OSError, ValueError) as e:
report("FAIL", label, f"/input/getstate.json didn't answer ({e}); the pause gesture can't read the controllers")
return
found = [d for d in devices if d.get("controller_type") == "frame_controller" and d.get("root_path")
and d.get("side") in ("left", "right")]
if not found:
report("skip", label, "no Frame controller is on, so it wasn't checked")
return
if not all(any(c.get("path") == "/input/thumbstick/click" for c in d.get("components", [])) for d in found):
report("warn", label, "a controller lists no /input/thumbstick/click; the default pause gesture needs it")
data = b""
try:
with socket.create_connection(("127.0.0.1", 27062), timeout=3) as s:
key = base64.b64encode(os.urandom(16)).decode()
s.sendall((f"GET / HTTP/1.1\r\nHost: 127.0.0.1:27062\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n"
f"Sec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\nOrigin: {VRSERVER}\r\n"
f"Referer: {headers['Referer']}\r\n\r\n").encode())
while b"\r\n\r\n" not in data:
chunk = s.recv(4096)
if not chunk:
break
data += chunk
if b" 101 " not in data.split(b"\r\n", 1)[0] + b" ":
report("FAIL", label, "vrserver refused the web socket; the pause gesture can't read the controllers")
return
mailbox = f"frametop_check_{os.getpid()}"
# Every controller: one lying still (or asleep) may send nothing at all.
for text in [f"mailbox_open {mailbox}"] + ["mailbox_send input_server " + json.dumps(
{"type": "request_input_state_updates", "device_path": d["root_path"], "returnAddress": mailbox})
for d in found]:
payload, mask = text.encode(), os.urandom(4)
size = bytes([0x80 | len(payload)]) if len(payload) < 126 else bytes([0x80 | 126]) + struct.pack(">H", len(payload))
s.sendall(b"\x81" + size + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
deadline = time.monotonic() + 3
try:
while b"update_component_states" not in data and time.monotonic() < deadline:
chunk = s.recv(65536)
if not chunk:
break
data += chunk
except socket.timeout:
report("skip", label, "it took the subscription, but no controller sent anything in 3 s "
"(lying still or asleep?): pick one up and check again")
return
except OSError as e:
report("FAIL", label, f"{e}; the pause gesture can't read the controllers")
return
if b"update_component_states" in data and b"/click" in data:
report("ok", label, "it still streams the controllers' buttons, which the pause gesture reads")
else:
report("FAIL", label, "no button states came, so the pause gesture can't read the controllers "
"(input/vrws.py: the message format changed?)")
def check_eye_tracker():
gaze = systemctl("is-enabled", "frametop-gaze") == "enabled"
try:
@@ -380,6 +485,8 @@ def main():
# A terminal in the desktop has its session's runtime dir and bus; systemctl needs the real ones.
os.environ["XDG_RUNTIME_DIR"] = f"/run/user/{UID}"
os.environ["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path=/run/user/{UID}/bus"
# And its own config folder, where SteamVR's path registry isn't: OpenVR and vrcmd need ours.
os.environ["XDG_CONFIG_HOME"] = os.path.join(HOME, ".config")
versions = current_versions()
check_versions(versions)
@@ -394,6 +501,7 @@ def main():
check_overlays(desktop_up)
check_services(sockets, desktop_up)
check_steam_ui()
check_vr_socket()
check_eye_tracker()
if args == ["--mark-good"]:
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env python3
"""Bring back a taskbar that Plasma saved against a screen this desktop doesn't have.
Plasma 6.2.5 ties each panel to a screen number (lastScreen in its containment), and the
numbers rank the enabled outputs by priority: 0 is the primary screen. A panel whose
number is past the screen count gets no view and stays hidden; Plasma never moves it,
even on a later start (sanitizeScreenLayout() only remaps a panel whose number has no
desktop, and the Frametop desktop keeps a desktop for every spare output it has seen).
So a taskbar saved on a spare output (issue #18), or on a screen that a smaller layout
dropped, was lost until the config was deleted.
The session runs this before Plasma starts, so nothing races Plasma for the file. Each
panel numbered at or past the screen count moves to screen 0, with its system tray's
containment, and keeps its widgets and settings. A panel is left where it is when screen
0 already has a panel on that edge: it comes back by itself if the screens do. The file
is backed up once per repair (<file>.ft-bak), and the changes go through kwriteconfig6.
fix-panels.py [--screens N] [--file APPLETSRC] [--check]
--screens the desktop's screen count (default: the configured layout's)
--file default: $XDG_CONFIG_HOME/plasma-org.kde.plasma.desktop-appletsrc, with
XDG_CONFIG_HOME defaulting to the Frametop desktop's ~/.config/frametop
--check change nothing; list the panels, and exit 1 if one is lost
"""
import argparse
import os
import re
import shutil
import subprocess
import sys
EDGES = {3: "top", 4: "bottom", 5: "left", 6: "right"} # Plasma::Types::Location
SYSTRAY = "org.kde.plasma.private.systemtray"
GROUP = re.compile(r"\[([^\]]*)\]")
def parse(text):
"""KConfig text -> {(group, subgroup, ...): {key: value}}."""
groups, cur = {}, None
for line in text.splitlines():
line = line.strip()
if line.startswith("["):
cur = tuple(GROUP.findall(line))
groups.setdefault(cur, {})
elif cur is not None and "=" in line and not line.startswith("#"):
k, v = line.split("=", 1)
groups[cur][re.sub(r"\[\$.*\]$", "", k.strip())] = v.strip()
return groups
def number(v, default=-1):
try:
return int(v)
except (TypeError, ValueError):
return default
def panels(groups):
"""The panels, by containment id: location, lastScreen, and the ids of their system
trays' own containments (which sit on the same edge and screen as the panel)."""
trays, found = {}, {}
for g, keys in groups.items():
if len(g) == 5 and g[0] == "Containments" and g[2] == "Applets" and g[4] == "Configuration":
tray = keys.get("SystrayContainmentId")
if tray:
trays.setdefault(g[1], []).append(tray)
owned = {t for ts in trays.values() for t in ts}
for g, keys in groups.items():
if len(g) != 2 or g[0] != "Containments" or number(g[1]) <= 0 or g[1] in owned:
continue
loc = number(keys.get("location"), 0)
if loc in EDGES and keys.get("plugin") != SYSTRAY:
found[g[1]] = {"location": loc, "screen": number(keys.get("lastScreen")),
"plugin": keys.get("plugin", "?"), "trays": trays.get(g[1], [])}
return dict(sorted(found.items(), key=lambda kv: number(kv[0])))
def plan(groups, screens):
"""(moves, kept): moves are (panel id, from screen, containment ids to put on screen 0);
kept are (panel id, from screen, why) for lost panels left alone."""
found = panels(groups)
taken = {(p["screen"], p["location"]) for p in found.values() if 0 <= p["screen"] < screens}
moves, kept = [], []
for pid, p in found.items():
if p["screen"] < screens:
continue # on a screen this desktop has (Plasma puts -1 on the first one itself)
if (0, p["location"]) in taken:
kept.append((pid, p["screen"], f"the first screen already has a {EDGES[p['location']]} panel"))
continue
taken.add((0, p["location"]))
moves.append((pid, p["screen"], [pid, *p["trays"]]))
return moves, kept
def default_screens():
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "layout"))
import ft_layout
return ft_layout.screen_count()
def main(argv):
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--screens", type=int)
ap.add_argument("--file")
ap.add_argument("--check", action="store_true")
a = ap.parse_args(argv)
config = os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config/frametop")
path = a.file or os.path.join(config, "plasma-org.kde.plasma.desktop-appletsrc")
screens = a.screens if a.screens else default_screens()
try:
with open(path) as f:
groups = parse(f.read())
except FileNotFoundError:
if a.check:
print("no Plasma config yet (Plasma makes the default taskbar)")
return 0
moves, kept = plan(groups, screens)
if a.check:
found = panels(groups)
for pid, p in found.items():
lost = " (lost: no such screen)" if p["screen"] >= screens else ""
print(f"panel {pid}: screen {p['screen']}, {EDGES[p['location']]}{lost}")
if not found:
print("no panels")
print(f"{screens} screen(s)")
return 1 if moves or kept else 0
if moves:
shutil.copy2(path, path + ".ft-bak")
for pid, was, ids in moves:
for cid in ids:
subprocess.run(["kwriteconfig6", "--file", os.path.abspath(path), "--group", "Containments",
"--group", cid, "--key", "lastScreen", "0"], check=True)
print(f"frametop: panel {pid} was saved on screen {was}, which this desktop doesn't have "
f"({screens} screen(s)); moved it to the first screen (backup: {path}.ft-bak)", file=sys.stderr)
for pid, was, why in kept:
print(f"frametop: panel {pid} is saved on screen {was}, which this desktop doesn't have; "
f"left there: {why}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+55 -1
View File
@@ -29,7 +29,7 @@ for var in $(compgen -e); do
case $var in
LD_LIBRARY_PATH | LD_PRELOAD | STEAM_* | Steam* | SRT_* | PRESSURE_VESSEL_* | MANGOHUD_* | \
ENABLE_VK_LAYER_VALVE_steam_overlay_* | STEAMVIDEOTOKEN | QT_IM_MODULE | GTK_IM_MODULE | \
XMODIFIERS) unset "$var" ;;
XMODIFIERS | AT_SPI_BUS_ADDRESS) unset "$var" ;;
esac
done
@@ -190,6 +190,21 @@ if [ "$remote" = 1 ]; then
"$here/remote-ctl.sh" start
fi
# AT-SPI: start the registry inside Plasma's autostart, after KWin has published the
# nested displays. Starting it before startplasma could bind to the host's X display.
# This config belongs only to Frametop; the host desktop's autostart is unchanged.
autostart=$XDG_CONFIG_HOME/autostart/frametop-atspi.desktop
mkdir -p "$(dirname "$autostart")"
cat > "$autostart" <<EOF
[Desktop Entry]
Type=Application
Name=Frametop accessibility
Exec="$here/ft-atspi"
X-KDE-autostart-phase=2
OnlyShowIn=KDE;
NoDisplay=true
EOF
# ft-floatd (floating windows) runs inside the Plasma session, on its D-Bus: started from
# the session's autostart, which only this desktop reads (XDG_CONFIG_HOME above).
autostart=$XDG_CONFIG_HOME/autostart/frametop-floatd.desktop
@@ -234,8 +249,47 @@ elif [ "$(kreadconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete
fi
# KWin draws through zink on the headset's GPU, which vrcompositor needs, so the blur and
# background contrast behind panels and menus and the window animations cost frames for
# little. They're off by default in this desktop. Each is written once, before KWin first
# reads it, and only if this desktop's config doesn't have it yet; the marker keeps it
# from coming back, since System Settings deletes a setting put back to KDE's default.
# docs/reference.md says how to turn them on again.
default() { # file group key value
[ -n "$(kreadconfig6 --file "$1" --group "$2" --key "$3")" ] || kwriteconfig6 --file "$1" --group "$2" --key "$3" "$4"
}
frametoprc=$XDG_CONFIG_HOME/frametoprc
if [ "$(kreadconfig6 --file "$frametoprc" --group Defaults --key effects)" != 1 ]; then
default "$kwinrc" Plugins blurEnabled false
default "$kwinrc" Plugins contrastEnabled false
default "$XDG_CONFIG_HOME/kdeglobals" KDE AnimationDurationFactor 0
kwriteconfig6 --file "$frametoprc" --group Defaults --key effects 1
fi
# System autostart entries this desktop doesn't need, hidden for it alone by a copy with
# Hidden=true in its own autostart folder, once and unless there's a file of that name
# already. Discover's update notifier starts Discover itself to check for updates (about
# 600 MB and a share of a core, with Flatpak's helper and AppStream behind it); updates
# come with SteamOS and from Discover in the stock desktop. IBus can't reach the
# desktop's apps: KWin's input method is ft-textinput, and the session drops the
# variables that point apps at IBus or XIM. Deleting the copy brings an entry back.
if [ "$(kreadconfig6 --file "$frametoprc" --group Defaults --key autostart)" != 1 ]; then
for entry in org.kde.discover.notifier ibus; do
src=/etc/xdg/autostart/$entry.desktop dst=$XDG_CONFIG_HOME/autostart/$entry.desktop
[ -r "$src" ] && [ ! -e "$dst" ] || continue
mkdir -p "$(dirname "$dst")"
sed '/^\[Desktop Entry\]$/a Hidden=true' "$src" > "$dst"
done
kwriteconfig6 --file "$frametoprc" --group Defaults --key autostart 1
fi
# Profiles reopen apps (docs/profiles.md), so Plasma's own session restore stays off here;
# with both, apps would open twice.
kwriteconfig6 --file "$XDG_CONFIG_HOME/ksmserverrc" --group General --key loginMode emptySession
# Plasma keeps a panel on a screen number, and never moves one whose screen this desktop
# doesn't have, like a spare output or a screen a smaller layout dropped. Put such a panel
# back on the first (primary) screen before Plasma reads the file (session/fix-panels.py).
python3 "$here/fix-panels.py" --screens "$screens" || true
dbus-run-session startplasma-wayland
+1 -1
View File
@@ -40,7 +40,7 @@ POINTER_HEAD_DEADZONE=0.5 # keyboard clicks at the gaze (Meta+J, Meta+K): degre
POINTER_KEY_TAP=0.25 # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
HANDS_SWAP_SIDES=auto # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them
HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on
HANDS_CAMERAS=auto # hand tracking: auto (by the light) | mono (the IR cameras; also keeps ft-camd off the colour ones) | color | all
HANDS_BRIGHT=all # hand tracking, auto: the cameras in bright light, all | color
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/python3
"""Start AT-SPI on this nested desktop's live bus, after KWin has set its displays."""
import ast
import os
import signal
import subprocess
import sys
import time
DBUS = "org.freedesktop.DBus"
DBUS_PATH = "/org/freedesktop/DBus"
REGISTRY = "org.a11y.atspi.Registry"
def call(address, destination, path, method, *args):
return subprocess.run(
["gdbus", "call", "--address", address, "--dest", destination,
"--object-path", path, "--method", method, "--timeout", "5", *args],
check=True, capture_output=True, text=True, timeout=6).stdout.strip()
def stop(child):
if child.poll() is None:
child.terminate()
try:
child.wait(timeout=3)
except subprocess.TimeoutExpired:
child.kill()
child.wait()
def run_registry(address, session):
# SteamOS's registryd stays alive after a bus disconnect. Keep only our own
# child tied to both private buses; no host PID lookup or broad process kill.
registry = subprocess.Popen(["/usr/lib/at-spi2-registryd"])
try:
while registry.poll() is None:
try:
call(session, DBUS, DBUS_PATH, DBUS + ".GetId")
call(address, DBUS, DBUS_PATH, DBUS + ".GetId")
except subprocess.SubprocessError:
break # Normal session shutdown or loss of its accessibility bus.
# Each check starts two gdbus processes, and SteamVR needs the CPU.
time.sleep(5)
finally:
stop(registry)
def start():
session = os.environ.get("DBUS_SESSION_BUS_ADDRESS")
runtime = os.environ.get("XDG_RUNTIME_DIR", "")
if not session or os.path.basename(runtime) != "frametop":
return # Do not autolaunch a bus or touch the host desktop.
os.environ.pop("AT_SPI_BUS_ADDRESS", None)
reply = call(session, "org.a11y.Bus", "/org/a11y/bus", "org.a11y.Bus.GetAddress")
address, = ast.literal_eval(reply)
if not isinstance(address, str) or not address.startswith("unix:"):
raise ValueError("org.a11y.Bus returned no local accessibility bus")
if call(address, DBUS, DBUS_PATH, DBUS + ".NameHasOwner", REGISTRY) == "(true,)":
return
# Activation otherwise inherits the accessibility daemon's old environment (and
# DBUS_SESSION_BUS_ADDRESS is the accessibility bus itself). Pin the live bus.
env = {key: os.environ.get(key, "") for key in (
"DISPLAY", "WAYLAND_DISPLAY", "XAUTHORITY", "XDG_RUNTIME_DIR",
"XDG_CONFIG_HOME", "DBUS_SESSION_BUS_ADDRESS")}
env["AT_SPI_BUS_ADDRESS"] = address
call(address, DBUS, DBUS_PATH, DBUS + ".UpdateActivationEnvironment", repr(env))
try:
call(address, DBUS, DBUS_PATH, DBUS + ".StartServiceByName", REGISTRY, "0")
except subprocess.CalledProcessError:
# SteamOS's launcher can select dbus-broker because we are in a user unit,
# but the private session bus has no systemd activation manager. Reuse its
# bus rather than starting another launcher/bus. Never replace an owner.
if call(address, DBUS, DBUS_PATH, DBUS + ".NameHasOwner", REGISTRY) == "(true,)":
return
os.environ["AT_SPI_BUS_ADDRESS"] = address
# registryd itself refuses to queue behind an existing registry, including
# races with native activation. The watcher cleans up only our own child.
run_registry(address, session)
if __name__ == "__main__":
signal.signal(signal.SIGTERM, lambda signum, frame: sys.exit(0))
try:
start()
except (OSError, ValueError, SyntaxError, subprocess.SubprocessError) as error:
print(f"frametop: accessibility unavailable: {error}", file=sys.stderr)
+1 -1
View File
@@ -10,7 +10,7 @@ cg=$(systemctl --user show -p ControlGroup --value "$unit" 2>/dev/null)
# The desktop's own processes stay in the unit and stop with it: the session, KWin,
# Plasma, and the session services it started (portals, input methods, kded, wallet...).
own='^(frametop-sessi|dbus-|startplasma|plasma|kwin|Xwayland|ksmserver|krdpserver|Xvnc|xfreerdp|ft-layout|'
own='^(frametop-sessi|dbus-|startplasma|plasma|kwin|Xwayland|ksmserver|krdpserver|Xvnc|xfreerdp|ft-layout|ft-atspi|'
own+='kded|kactivitymanage|kaccess|kglobalaccel|kscreen|kwalletd|ksecretd|polkit-kde|org_kde_|baloo|'
own+='xembedsniproxy|gmenudbusmenu|DiscoverNotifie|kimpanel|ibus|xdg-|at-spi|dconf-service|fusermount|agent)'
keep=()
File diff suppressed because it is too large. Load diff
+162
View File
@@ -0,0 +1,162 @@
#!/usr/bin/env python3
"""Tests for session/fix-panels.py: issue #18's config (the taskbar saved on spare output 8
of a 3-screen desktop), panels that are fine, an edge that's taken, and a real run of
kwriteconfig6 on a copy. Nothing here touches the running desktop or its config.
python3 session/tests/test_fix_panels.py
"""
import importlib.util
import os
import subprocess
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.realpath(__file__))
SCRIPT = os.path.join(HERE, "..", "fix-panels.py")
spec = importlib.util.spec_from_file_location("fix_panels", SCRIPT)
fp = importlib.util.module_from_spec(spec)
spec.loader.exec_module(fp)
def desktop(cid, screen):
return f"""[Containments][{cid}]
activityId=7e1d1a3c-0000-4000-8000-000000000000
formfactor=0
immutability=1
lastScreen={screen}
location=0
plugin=org.kde.plasma.folder
wallpaperplugin=org.kde.image
"""
def panel(cid, screen, location=4, tray=None):
text = f"""[Containments][{cid}]
activityId=
formfactor=2
immutability=1
lastScreen={screen}
location={location}
plugin=org.kde.panel
wallpaperplugin=org.kde.image
[Containments][{cid}][Applets][{cid}1]
immutability=1
plugin=org.kde.plasma.kickoff
"""
if tray:
text += f"""
[Containments][{cid}][Applets][{cid}2]
immutability=1
plugin=org.kde.plasma.systemtray
[Containments][{cid}][Applets][{cid}2][Configuration]
PreloadWeight=90
SystrayContainmentId={tray}
[Containments][{tray}]
activityId=
formfactor=2
immutability=1
lastScreen={screen}
location={location}
plugin=org.kde.plasma.private.systemtray
popupHeight=432
"""
return text
ISSUE_18 = "\n".join([desktop(1, 0), desktop(18, 1), desktop(19, 2)] + [desktop(20 + i, 3 + i) for i in range(8)]
+ [panel(10, 8, tray=11), "[ScreenMapping]\nitemsOnDisabledScreens=\n"])
class Plan(unittest.TestCase):
def test_issue_18(self):
moves, kept = fp.plan(fp.parse(ISSUE_18), 3)
self.assertEqual(moves, [("10", 8, ["10", "11"])])
self.assertEqual(kept, [])
def test_panel_on_a_screen_stays(self):
for screen in (0, 2, -1):
moves, kept = fp.plan(fp.parse(desktop(1, 0) + panel(2, screen, tray=8)), 3)
self.assertEqual((moves, kept), ([], []), screen)
def test_tray_is_not_a_panel(self):
found = fp.panels(fp.parse(panel(2, 0, tray=8)))
self.assertEqual(list(found), ["2"])
self.assertEqual(found["2"]["trays"], ["8"])
def test_desktops_never_move(self):
moves, _ = fp.plan(fp.parse(ISSUE_18), 1)
self.assertEqual([m[0] for m in moves], ["10"])
def test_edge_taken_on_first_screen(self):
text = panel(2, 0, tray=8) + panel(10, 8, tray=11)
moves, kept = fp.plan(fp.parse(text), 3)
self.assertEqual(moves, [])
self.assertEqual([k[:2] for k in kept], [("10", 8)])
def test_other_edge_is_free(self):
moves, kept = fp.plan(fp.parse(panel(2, 0) + panel(10, 4, location=3)), 3)
self.assertEqual(moves, [("10", 4, ["10"])])
self.assertEqual(kept, [])
def test_two_lost_on_one_edge(self):
moves, kept = fp.plan(fp.parse(panel(10, 5) + panel(12, 8)), 3)
self.assertEqual([m[0] for m in moves], ["10"])
self.assertEqual([k[0] for k in kept], ["12"])
def test_screen_count_went_down(self):
moves, _ = fp.plan(fp.parse(panel(2, 1, location=3)), 1)
self.assertEqual(moves, [("2", 1, ["2"])])
class Run(unittest.TestCase):
def setUp(self):
self.dir = tempfile.TemporaryDirectory()
self.path = os.path.join(self.dir.name, "plasma-org.kde.plasma.desktop-appletsrc")
with open(self.path, "w") as f:
f.write(ISSUE_18)
def tearDown(self):
self.dir.cleanup()
def run_script(self, *args):
return subprocess.run([sys.executable, SCRIPT, "--file", self.path, "--screens", "3", *args],
capture_output=True, text=True)
def test_check_reports_and_changes_nothing(self):
r = self.run_script("--check")
self.assertEqual(r.returncode, 1)
self.assertIn("panel 10: screen 8, bottom (lost", r.stdout)
with open(self.path) as f:
self.assertEqual(f.read(), ISSUE_18)
def test_repair(self):
r = self.run_script()
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn("moved it to the first screen", r.stderr)
with open(self.path) as f:
groups = fp.parse(f.read())
self.assertEqual(groups[("Containments", "10")]["lastScreen"], "0")
self.assertEqual(groups[("Containments", "11")]["lastScreen"], "0")
self.assertEqual(groups[("Containments", "10", "Applets", "101")]["plugin"], "org.kde.plasma.kickoff")
self.assertEqual(groups[("Containments", "25")]["lastScreen"], "8") # a spare's desktop
with open(self.path + ".ft-bak") as f:
self.assertEqual(f.read(), ISSUE_18)
# A second run finds nothing to do and leaves the backup alone.
os.remove(self.path + ".ft-bak")
r = self.run_script()
self.assertEqual((r.returncode, r.stderr), (0, ""))
self.assertFalse(os.path.exists(self.path + ".ft-bak"))
self.assertEqual(self.run_script("--check").returncode, 0)
def test_no_config_yet(self):
os.remove(self.path)
self.assertEqual(self.run_script().returncode, 0)
self.assertEqual(self.run_script("--check").returncode, 0)
if __name__ == "__main__":
unittest.main()
+105 -15
View File
@@ -11,6 +11,16 @@
# screen placed lower or further right. Instead the VNC screen is the primary's size, and
# the workspace-sized RDP window inside it is shifted so the primary fills it. The pointer
# maps 1:1. When the layout changes, the VNC screen resizes and the RDP client reconnects.
#
# The RDP client runs only while someone watches. While connected, krdp captures and
# H.264-encodes every redraw in software, about 60% of a core, with FreeRDP and Xvnc adding
# about 30% more, even with no VNC viewer. krdp starts its capture per RDP connection and
# stops it when the connection closes, so it idles without one. So FreeRDP starts when a
# VNC client connects (the screen is black for the few seconds that takes) and stops
# VNC_IDLE_SEC (45) seconds after the last one leaves. Xvnc has no hook for clients, so ss
# counts them: whenever Xvnc writes to its log (it logs each connection), every second
# while FreeRDP runs, and every 5 seconds otherwise. The log only wakes this script up;
# what it says doesn't matter.
set -eu
here=$(dirname "$(readlink -f "$0")")
@@ -57,16 +67,51 @@ stop_rdp() { pkill -f "[x]freerdp /v:127.0.0.1:$rdp_port " 2>/dev/null || true;
trap 'stop_rdp; pkill -f "[X]vnc $display " 2>/dev/null || true' EXIT
box bash -c 'vncpasswd -f < "$1/vnc-password" > "$1/vnc-passwd.bin" && chmod 600 "$1/vnc-passwd.bin"' - "$creds"
box Xvnc "$display" -geometry "${w}x${h}" -depth 24 \
exec {xlog}< <(box Xvnc "$display" -geometry "${w}x${h}" -depth 24 \
-interface "$addr" -rfbport "$vnc_port" \
-SecurityTypes VncAuth -PasswordFile "$creds/vnc-passwd.bin" \
-AlwaysShared -desktop "Steam Frame (Frametop)" &
-AlwaysShared -desktop "Steam Frame (Frametop)" 2>&1)
xvnc=$!
sleep 2
# Keep an RDP connection open inside the VNC screen. Reconnect if it drops or the layout changes.
# Wait up to $1 seconds, less if Xvnc logs something; its lines go on to this log.
nap() {
local line rc=0
IFS= read -rt "$1" -u "$xlog" line || rc=$?
if [ $rc -eq 0 ]; then
printf '%s\n' "$line"
while IFS= read -rt 0.1 -u "$xlog" line; do printf '%s\n' "$line"; done
elif [ $rc -le 128 ]; then
sleep 1 # Xvnc's output closed: it's exiting
fi
return 0
}
nap 2
# A connected VNC client: an established TCP connection to Xvnc's port. Any connection
# counts, authenticated or not; it's on the tailnet only.
clients() { [ -n "$(ss -Htn state established "( sport = :$vnc_port )" 2>/dev/null)" ]; }
# ft-screens drops screens you aren't looking at to a low frame rate, and krdp would
# stream that. "watch SECONDS" asks it for full rate on every screen for that long: sent
# when a client connects and renewed every few seconds while one stays, so it lapses by
# itself if this script dies. An older ft-screens just answers that it doesn't know it.
watch() {
if command -v socat >/dev/null; then
printf 'watch 15' | socat -u - ABSTRACT-SENDTO:ft_screens 2>/dev/null
else
python3 -c 'import socket; socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM).sendto(b"watch 15", "\0ft_screens")' 2>/dev/null
fi || true
}
# What remote-view depends on: KWin's saved outputs (positions, scales, primary) and
# Frametop's layout (screen sizes). It's read again only after one of these changes, and
# once a minute in case a change didn't touch them, and only while FreeRDP runs.
layout_files=("$HOME/.config/frametop/kwinoutputconfig.json" "$HOME/.config/frametop-layout.json")
stamp() { stat -c %y "${layout_files[@]}" 2>/dev/null || true; }
# Start FreeRDP inside the VNC screen, sized and shifted for $v.
# /cert:ignore is fine here: the connection never leaves this host.
while kill -0 $xvnc 2>/dev/null; do
start_rdp() {
read -r x y w h ww wh <<< "$v"
box env DISPLAY=$display bash -c '
size=$1 x=$2 y=$3 ww=$4 wh=$5 creds=$6 rdp_port=$7
@@ -80,6 +125,7 @@ while kill -0 $xvnc 2>/dev/null; do
rdp=$!
# FreeRDP takes no negative position, so move its window once it is up.
for _ in $(seq 60); do
kill -0 $rdp 2>/dev/null || break
win=$(xdotool search --class xfreerdp 2>/dev/null | tail -1)
[ -n "$win" ] && break
sleep 0.5
@@ -88,14 +134,58 @@ while kill -0 $xvnc 2>/dev/null; do
wait $rdp
' vnc-rdp "${w}x$h" "$x" "$y" "$ww" "$wh" "$creds" "$rdp_port" || true &
rdp=$!
while kill -0 $rdp 2>/dev/null; do
sleep 5
now=$(view) || continue
[ -n "$now" ] && [ "$now" != "$v" ] || continue
echo "layout changed: $v -> $now"
v=$now
stop_rdp
done
wait $rdp 2>/dev/null || true
sleep 2
}
idle_sec=${VNC_IDLE_SEC:-45}
rdp= # FreeRDP's job while it runs
seen=0 # when a client was last seen ($SECONDS)
watched=-99 # when "watch" was last sent
stamp_v= # stamp() when $v was read
recheck=0 # read the layout every 2 seconds until then
next_view=0 # next time to read it
while kill -0 $xvnc 2>/dev/null; do
if clients; then
if [ $((SECONDS - watched)) -ge 5 ]; then watch; watched=$SECONDS; fi
seen=$SECONDS
if [ -z "$rdp" ]; then
echo "VNC client connected, starting the RDP client"
now=$(view) && [ -n "$now" ] && v=$now
stamp_v=$(stamp) recheck=0 next_view=$((SECONDS + 60))
start_rdp
fi
elif [ "$seen" -ne 0 ]; then
watched=-99
if [ $((SECONDS - seen)) -ge "$idle_sec" ]; then
seen=0
if [ -n "$rdp" ]; then
echo "no VNC client for ${idle_sec}s, stopping the RDP client"
stop_rdp
wait "$rdp" 2>/dev/null || true
rdp=
fi
fi
fi
if [ -n "$rdp" ] && ! kill -0 "$rdp" 2>/dev/null; then
# It dropped, or the layout changed: reconnect next round if a client is still there.
wait "$rdp" 2>/dev/null || true
rdp=
nap 2
continue
fi
if [ -n "$rdp" ]; then
# A layout change shows up in KWin's outputs a few seconds after the files change.
s=$(stamp)
[ "$s" = "$stamp_v" ] || { stamp_v=$s; recheck=$((SECONDS + 10)) next_view=$SECONDS; }
if [ "$SECONDS" -ge "$next_view" ]; then
next_view=$((SECONDS + 60))
[ "$SECONDS" -ge "$recheck" ] || next_view=$((SECONDS + 2))
now=$(view) || now=
if [ -n "$now" ] && [ "$now" != "$v" ]; then
echo "layout changed: $v -> $now"
v=$now
stop_rdp
fi
fi
fi
if [ -n "$rdp" ] || [ "$seen" -ne 0 ]; then nap 1; else nap 5; fi
done
Executable
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env bash
# Uninstall Frametop from the Steam Frame. In a terminal on the headset (Konsole in the desktop,
# or over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
#
# or ~/frametop/uninstall.sh. It doesn't use the rest of the repo, so it also works when
# ~/frametop is gone or broken.
#
# It never stops what you're using now: the input relay carries the keyboard and mouse, and the
# desktop runs from the repo. So it goes in two steps:
# 1. Frametop stops starting. The launcher's Desktop entry goes back to the stock desktop, and
# Frametop's services, SteamVR driver, menu entries, and system files (our eye tracker's
# frame grabber and the Bluetooth fixes, with sudo) are removed. What runs now keeps running
# until you restart the headset.
# 2. After the restart, run it again. It deletes the code (~/frametop) and, if you want, your
# settings and the build container.
# When nothing of Frametop is running, one run does both.
#
# Options (piped, they go after "bash -s --"):
# --dir DIR where the repo is, if not ~/frametop
# --dry-run show what it would do, and change nothing
set -euo pipefail
shopt -s nullglob
usage() {
cat <<'EOF'
usage: uninstall.sh [--dir DIR] [--dry-run]
piped: curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- [options]
EOF
}
dry=0
apps=$HOME/.local/share/applications
override=$apps/deckard-nested-desktop.desktop
relay_unit=$HOME/.config/systemd/user/frametop-input-relay.service
driver=$HOME/.local/share/frametop/ft_pointer
vrpathreg=/opt/steamvr/bin/linuxarm64/vrpathreg
handsctl=$HOME/.local/bin/ft-handsctl
eyegrab_files=(/etc/systemd/system/frametop-eyegrab.service /etc/frametop/ft-eyegrab)
bt_files=(/etc/systemd/system/steamframe-bt-fixups.service /etc/systemd/system/bluetooth.service.d/steamframe.conf
/etc/steamframe/bt-fixups.sh)
step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
run() { # run a command, or with --dry-run, show it
if [ "$dry" = 1 ]; then printf ' would run: %s\n' "$*"; else "$@"; fi
}
ask() { # ask "question" default(y|n)
local hint answer
if [ "$dry" = 1 ]; then echo "$1 [dry run: yes]"; return 0; fi
hint=$([ "$2" = y ] && echo "Y/n" || echo "y/N")
read -r -p "$1 [$hint] " answer </dev/tty || answer=
answer=${answer:-$2}
[[ $answer =~ ^[Yy] ]]
}
exists() { local f; for f in "$@"; do [ -e "$f" ] && return 0; done; return 1; }
size() { du -shc "$@" 2>/dev/null | tail -1 | cut -f1; }
# Is this folder Frametop's code? Only then is it deleted.
is_repo() { [ "$1" != "$HOME" ] && [ -f "$1/desktops.sh" ] && [ -f "$1/session/frametop-session.sh" ]; }
find_repo() {
local p
[ -n "$dir" ] && { echo "$dir"; return; }
# The launcher entry and the relay's unit point into the repo, until step 1 removes them.
p=$(sed -n 's#^Exec=\(.*\)/session/frametop-session\.sh.*#\1#p' "$override" 2>/dev/null | head -1)
[ -z "$p" ] && p=$(sed -n 's#^ExecStart=/usr/bin/python3 \(.*\)/input/input-relay\.py.*#\1#p' "$relay_unit" 2>/dev/null | head -1)
[ -z "$p" ] && [ -f "${BASH_SOURCE[0]:-}" ] && p=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
echo "${p:-$HOME/frametop}"
}
# Frametop's programs that are running now (step 1 leaves them running until the restart).
running() {
local n out=()
for n in ft-screens ft-pointer ft-powerd ft-eyegrab ft-camd ft-hands; do
pgrep -x "$n" >/dev/null && out+=("$n")
done
pgrep -f '[i]nput/input-relay\.py' >/dev/null && out+=("input relay")
pgrep -f '[g]aze/ft-gazed' >/dev/null && out+=("gaze service")
pgrep -f '[f]rametop-session\.sh' >/dev/null && out+=("desktop session")
echo "${out[*]}"
}
main() {
local units=() entries=() names=() sys=0 repo now f
while [ $# -gt 0 ]; do
case $1 in
--dir) dir=${2:?--dir needs a folder}; shift ;;
--dry-run) dry=1 ;;
-h|--help) usage; return 0 ;;
*) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
esac
shift
done
if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
echo "This uninstalls Frametop from a Steam Frame (SteamOS, VR variant). Run it in a terminal on the headset." >&2
return 1
fi
if [ "$dry" = 0 ] && ! { : </dev/tty; } 2>/dev/null; then
echo "This asks questions, and there's no terminal to ask in. Run it in one (over SSH: ssh -t)." >&2
return 1
fi
[ "$dry" = 1 ] && echo "Dry run: nothing changes."
repo=$(find_repo)
# Step 1: what makes Frametop start. Nothing here stops a running program.
[ -f "$override" ] && grep -q 'Frametop' "$override" || override=
units=("$HOME"/.config/systemd/user/frametop-*.service)
for f in ft-input-settings ft-display-settings ft-layout-reset ft-screens-toggle ft-remote-settings ft-gazeprobe; do
[ -e "$apps/$f.desktop" ] && entries+=("$apps/$f.desktop")
done
[ -e "$apps/frametop-handrec.desktop" ] && entries+=("$apps/frametop-handrec.desktop")
entries+=("$apps"/frametop-profile-*.desktop) # one per layout profile (layout/ft_layout.py)
[ -L "$handsctl" ] || handsctl=
exists "${eyegrab_files[@]}" "${bt_files[@]}" && sys=1
if [ -n "$override" ] || [ ${#units[@]} -gt 0 ] || [ ${#entries[@]} -gt 0 ] || [ -d "$driver" ] ||
[ -n "$handsctl" ] || [ "$sys" = 1 ]; then
step "Step 1 of 2: stop Frametop from starting"
echo "This removes:"
[ -n "$override" ] && echo " - the launcher's Desktop entry (Launch a program -> Desktop opens the stock desktop again)"
for f in "${units[@]}"; do echo " - the service $(basename "$f")"; done
[ -d "$driver" ] && echo " - the 3D mouse's SteamVR driver (ft_pointer)"
[ ${#entries[@]} -gt 0 ] && echo " - ${#entries[@]} menu entries (Frametop Display Settings, Input Settings, ...)"
[ -n "$handsctl" ] && echo " - $handsctl"
exists "${eyegrab_files[@]}" && echo " - our eye tracker's frame grabber (a system service: needs your password)"
exists "${bt_files[@]}" && echo " - the Bluetooth fixes (system files: needs your password)"
echo "What runs now keeps running until you restart the headset, so your keyboard, mouse, and"
echo "this terminal keep working. Your settings and the code stay for now."
ask "Uninstall Frametop?" n || { echo "Nothing changed."; return 0; }
[ -n "$override" ] && run rm -f "$override"
if [ ${#units[@]} -gt 0 ]; then
for f in "${units[@]}"; do names+=("$(basename "$f")"); done
run systemctl --user disable "${names[@]}" 2>/dev/null || true
run rm -f "${units[@]}"
run systemctl --user daemon-reload
fi
if [ -d "$driver" ]; then
if [ -x "$vrpathreg" ]; then
LD_LIBRARY_PATH=$(dirname "$vrpathreg") run "$vrpathreg" removedriver "$driver" ||
echo "warning: SteamVR's vrpathreg couldn't unregister the driver; SteamVR may log that it's missing" >&2
fi
run rm -rf "$driver"
fi
[ ${#entries[@]} -gt 0 ] && run rm -f "${entries[@]}"
[ -n "$handsctl" ] && run rm -f "$handsctl"
if [ "$sys" = 1 ]; then
echo "The system files need your password (sudo)."
if ! run sudo bash -c '
for u in frametop-eyegrab steamframe-bt-fixups; do systemctl disable $u.service 2>/dev/null; done
rm -f "$@"
rmdir /etc/frametop /etc/steamframe /etc/systemd/system/bluetooth.service.d 2>/dev/null
systemctl daemon-reload; true' sys "${eyegrab_files[@]}" "${bt_files[@]}"; then
echo "warning: the system files weren't removed (no password?). Run this again to retry." >&2
fi
fi
[ "$dry" = 1 ] || echo "Frametop no longer starts."
fi
# Step 2: delete what's left, once nothing of Frametop runs.
now=$(running)
if [ -n "$now" ]; then
step "Restart the headset to finish"
echo "Still running from before: $now."
echo "They stop when the headset restarts. After that, run this again to delete the code"
echo "($repo) and, if you want, your settings and the build container:"
echo
if [ "$repo" = "$HOME/frametop" ]; then
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash"
else
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- --dir $(printf %q "$repo")"
fi
echo
if ask "Restart the headset now? This closes everything open, in VR and on the desktop." n; then
run systemctl reboot || echo "Couldn't restart it from here: restart the headset from Steam's power menu." >&2
fi
[ "$dry" = 1 ] || return 0
echo "Dry run: after the restart, step 2 would go like this."
fi
step "Step 2 of 2: delete what's left"
if [ -d "$repo" ] && is_repo "$repo"; then
if [ -f "$repo/.git" ]; then
echo "Leaving $repo: it's a git worktree. Remove it with git worktree remove."
else
local def=y
if [ -n "$(git -C "$repo" status --porcelain --untracked-files=no 2>/dev/null)" ] ||
[ -n "$(git -C "$repo" log --branches --not --remotes --oneline 2>/dev/null | head -1)" ]; then
echo "$repo has changes of its own (git -C $repo status)."
def=n
fi
if ask "Delete the Frametop code in $repo ($(size "$repo"))?" "$def"; then
cd "$HOME"
run rm -rf "$repo"
fi
fi
elif [ -e "$repo" ]; then
echo "Leaving $repo: it doesn't look like Frametop's code."
fi
# Rebuilt by the desktop at each start, so nothing to keep.
run rm -rf "$HOME/.local/share/frametop/apps" "$HOME/.local/share/kwin/decorations/kwin4_decoration_qml_frametop" \
"$HOME/.cache/frametop"
local settings=("$HOME"/.config/frametop.conf* "$HOME"/.config/frametop-*.json* "$HOME/.config/frametop-remote"
"$HOME/.config/frametop" "$HOME/.local/state/frametop")
local kept=()
for f in "${settings[@]}"; do [ -e "$f" ] && kept+=("$f"); done
if [ ${#kept[@]} -gt 0 ]; then
echo "Your settings: the screen layout and profiles, button maps, gaze calibration, the remote"
echo "desktop password, and the Frametop desktop's own Plasma setup (${kept[*]/#$HOME/\~})."
ask "Delete your settings too? Keep them to pick up where you left off if you reinstall." n &&
run rm -rf "${kept[@]}"
fi
local recs=()
for f in "$HOME/.local/share/frametop/eyes" "$HOME/.local/share/frametop/hands"; do [ -e "$f" ] && recs+=("$f"); done
if [ ${#recs[@]} -gt 0 ]; then
ask "Delete your eye and hand recordings in ~/.local/share/frametop ($(size "${recs[@]}"))?" n &&
run rm -rf "${recs[@]}"
fi
[ "$dry" = 1 ] || rmdir "$HOME/.local/share/frametop" 2>/dev/null || true
if command -v podman >/dev/null && podman container exists dev 2>/dev/null; then
echo "The build container (dev) holds Frametop's compilers and libraries, 1-2 GB. If you put"
echo "anything else in it, that goes too."
if ask "Delete the build container?" n; then
run "$HOME/.local/bin/distrobox" rm --force dev </dev/null
run podman image rm registry.fedoraproject.org/fedora-toolbox:44 >/dev/null 2>&1 || true
echo "distrobox stays in ~/.local/bin, for any other containers. To remove it too:"
echo " ~/dev/src/distrobox/uninstall --prefix ~/.local"
fi
fi
step "Done"
echo "Frametop is uninstalled."
}
dir=
main "$@"