Compare commits

...
48 Commits
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 0821b1013e Settings: HANDS_SWAP_SIDES defaults to auto, and old untouched configs move to it
frametop.conf.example said HANDS_SWAP_SIDES=0, and desktops.sh copies it on a
fresh install, so every new install forced ft-camd's side camera names and
turned the hand tracker's own side check off. Some SteamVR restarts swap those
names, and then hands land beside their cutouts and recordings are mislabelled.

The example now says auto. scripts/conf-migrate.sh, run by install.sh and
hands/rec/install.sh, replaces the old line only where it's still exactly as
the example wrote it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:54:03 -06:00
DeeJanuzandClaude Opus 5.5 3cccad3525 Hands: label side cameras by the hands when a forced HANDS_SWAP_SIDES disagrees
With HANDS_SWAP_SIDES set to 0 or 1, ft-hands kept the forced naming as the
published truth even after the hands showed it was backwards. The hand recorder
took that as the session's decision, so the export labelled slam_left and
slam_right the wrong way round (dataset PR #4: every take swapped).

ft-hands now publishes the hands' answer once they disagree (state "forced,
disagrees"); tracking keeps the forced names. sides.read_live corrects the same
case from an ft-hands built before this, and the recorder's log says to use auto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:54:03 -06:00
DeeJanuzandClaude Opus 5.5 276c1409e8 Pages: the hand recorder's fix ships in Frametop 0.2.1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:46:35 -06:00
DeeJanuzandClaude Opus 5.5 a0fe3bb55e Merge experimental: eye tracker first calibration, accessibility registry, hand recorder without the colour module (#37)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:38:43 -06:00
DeeJanuzandClaude Opus 5.5 13198bc203 Pages: the hand recorder works without the colour module now
The known-issue notice becomes a fixed notice: the fix (98bd6ce, 662919b,
67a3c02) reaches main with this release. Anyone who hit "2 of 4 mono
cameras" is pointed at Update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:34:49 -06:00
DeeJanuzandClaude Opus 5.5 42a26753d3 Merge main into experimental: get.sh --branch and the hand recorder page
PR #29 was squash-merged, so main and experimental last shared history at
1ebf369 and 11 files conflicted. Main's copy of each was the PR #29 head
(ec870d2), which experimental already has, so experimental's side is kept
for all of them. The result is experimental plus main's get.sh and
hand-recorder.md, and main can take experimental with a merge commit again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:34:35 -06:00
DeeJanuzandClaude Opus 5.5 0f0674ff36 Merge branch gaze-double-cal (the first calibration no longer runs twice) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:28:23 -06:00
DeeJanuzandClaude Opus 5.5 3a796b5013 Gaze: the first calibration no longer runs twice
Turning gaze mode on wakes our tracker, so its eyes come back just as the
first calibration opens. DON_DELAY later, "the headset went on" re-armed the
automatic calibration while it was still open, and when it ended, ft-eyes'
status was still a moment old (not calibrated), so a second one opened at
once. Seen live on 2026-10-05: 26 of 27 dots, then 27 more.

The headset going on re-arms it only when no check is open.
first-calibration-test.py now has the headset go on mid-calibration and
checks that only one opens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:28:15 -06:00
DeeJanuz 67a3c02e31 ft-camd: map a camera whose dark frames are all zeros
Through the ISP (no colour module), the near-black exposures come out
all zeros, identical every time, so their dequeues change no buffer and
ft-camd never finished learning the side cameras' buffers ("can't tell
which buffers are this camera's yet"): the side pair published nothing.
Such an index is now left unmapped and its frames counted as dark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 89ec1608d9)
2026-10-05 13:55:18 -06:00
DeeJanuz 662919babf camcheck: no nested parentheses in the missing cameras' message
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 917ef9d29c)
2026-10-05 13:55:17 -06:00
DeeJanuz 98bd6ce21c Hand recorder: headsets without the colour module
Without the Arcturus module, XRService runs the side cameras through the
ISP ("ISP enabled for tracking cameras (main VFE available)"): NV12 on
vfe0 and vfe1, pitch 1152. ft-camd published only grey cameras, so it
dropped them, and the recorder stopped at "ft-camd publishes only 2 of 4
mono cameras" whatever was restarted.

- ft-camd reads a tracking camera's NV12 luma as its grey image.
- ft-hands and check_sides name the cameras by XRService's numbering in
  its log (TrackingCameraInit index 0-3), not by capture pipe, which
  moves with the module; a camera whose size isn't its calibration's is
  left out. sides.json says which device each camera was.
- camcheck reports the camera map, the ISP routing and which cameras
  ft-camd is missing; the recorder says so instead of "restart it",
  restarts its own ft-camd once when that's the only problem, and keeps
  the map in session.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4db5266cf7)
2026-10-05 13:55:17 -06:00
DeeJanuzandClaude Opus 5.5 608ad6034a get.sh: --branch NAME, to test a fix before it's released
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:49:20 -06:00
DeeJanuzandClaude Opus 5.5 336a60ce32 Pages: hand recorder needs the color module until the fix
Headsets without the Arcturus color passthrough module stop at the
camera check (ft-camd publishes only 2 of 4 mono cameras). Say so on
the install page, with the fix due by October 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 12:09:28 -06:00
DeeJanuz e1f27958ca Merge remote-tracking branch 'origin/experimental' into experimental 2026-10-05 11:12:07 -06:00
DeeJanuzandClaude Opus 5.5 c969963b83 A one-command uninstaller that keeps the headset usable
uninstall.sh replaces the README's thirteen uninstall commands. Run from a
Frametop desktop, those commands stopped the input relay second, which took
the keyboard and mouse away before the rest could be typed. And deleting
~/frametop first left Launch a program -> Desktop pointing at a missing
script.

The script works in two steps. Step 1 stops Frametop from starting: the
launcher entry, the user services (disabled, not stopped), the SteamVR
driver, the menu entries, and the eye grabber and Bluetooth fixes under
/etc (one sudo). Everything running keeps running until the headset
restarts, and the script offers to restart it. Step 2 runs once none of
Frametop's programs run. It deletes the code, and asks before deleting the
settings, recordings, and the dev container.

It doesn't use the rest of the repo, so the Pages one-liner works even when
~/frametop is gone. --dry-run shows each command without running it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 11:11:57 -06:00
DeeJanuzandClaude Opus 5.5 9f2ee5ad68 A one-command uninstaller that keeps the headset usable
uninstall.sh replaces the README's thirteen uninstall commands. Run from a
Frametop desktop, those commands stopped the input relay second, which took
the keyboard and mouse away before the rest could be typed. And deleting
~/frametop first left Launch a program -> Desktop pointing at a missing
script.

The script works in two steps. Step 1 stops Frametop from starting: the
launcher entry, the user services (disabled, not stopped), the SteamVR
driver, the menu entries, and the eye grabber and Bluetooth fixes under
/etc (one sudo). Everything running keeps running until the headset
restarts, and the script offers to restart it. Step 2 runs once none of
Frametop's programs run. It deletes the code, and asks before deleting the
settings, recordings, and the dev container.

It doesn't use the rest of the repo, so the Pages one-liner works even when
~/frametop is gone. --dry-run shows each command without running it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 11:11:38 -06:00
DeeJanuzandClaude Opus 5.5 dfc8e6eb5f Merge branch fresh-install (our eye tracker's first calibration; SteamVR's tools from a terminal in the desktop; gaze in the report) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:55:22 -06:00
DeeJanuzandClaude Opus 5.5 1037d4e125 Report: the gaze service, our eye tracker, and their logs
A gaze problem couldn't be told from a report: it had no gaze section. A user's report of
2026-10-05 showed only that the gaze service ran and our tracker was picked.

The report now has a Gaze section: whether the gaze service and our tracker's frame grabber
are enabled and running, when SteamVR's correction and our tracker's calibration were made
(or "none"), and the service's status (ft-gazectl status, on one line: the tracker in use,
whether it's awake and why not, checks.problem). Then the last 10 checks and calibration dots
(checks.jsonl: each dot's reply and whether it was taken), the gaze service's last 60 lines
without podman's exec lines, and the frame grabber's last 20. Nothing in them is an eye image.

Tested on this Frame: each section fills in, and the status is one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:54:57 -06:00
DeeJanuzandClaude Opus 5.5 307e2b7308 SteamVR's tools find SteamVR from a terminal in the Frametop desktop
The desktop has its own XDG_CONFIG_HOME (~/.config/frametop), and SteamVR's tools and OpenVR
read their path registry from there. So from a terminal in the desktop:

- pointer/driver/install.sh (install.sh step 4, every reinstall or update from Konsole) wrote
  a new registry, ~/.config/frametop/openvr/openvrpaths.vrpath, with only our driver and
  "runtime": null, and never registered the driver with SteamVR. That stray file then hid
  SteamVR from every OpenVR program started in the desktop.
- scripts/update-check.py (and so doctor.sh and report.sh) reported OpenVR "can't connect to
  SteamVR as a background app" (VRInitError_Init_PathRegistryNotFound, or
  InstallationNotFound with the stray file) and "vrcmd --overlays lists no overlays". A user's
  report of 2026-10-05 showed both, with SteamVR and Frametop's services fine.
- ft-layout's vrcmd calls (the gamescope backend) found no overlays.

They now run with XDG_CONFIG_HOME=~/.config: the driver installer (install, uninstall, probe,
aimhere), update-check.py (for all its checks: nothing there reads the desktop's own config),
report.sh's vrpathreg show, and ft-layout's vrcmd. report.sh doesn't set it for the whole
report, since session/fix-panels.py --check reads the desktop's Plasma config through it.

The driver installer also removes the stray registry (only vrpathreg's, with no runtime), and
update-check.py warns about one. And vrpathreg runs `xdg-open vrmonitor://driverinstalled` to
tell SteamVR's desktop monitor, which the Frame doesn't have: in the desktop that showed "could
not read file vrmonitor://driverinstalled" on every install. A stand-in xdg-open on its PATH
takes it, so install.sh no longer filters the step's output.

Tested in a fake HOME with a copy of SteamVR's registry, a stray one, and the desktop's
XDG_CONFIG_HOME: the new installer removes the stray file, registers the driver in the copy,
and never reaches xdg-open; the old one wrote the stray file, left the copy alone, and ran
xdg-open. update-check.py from that environment: OpenVR and vrcmd ok (the old one fails both);
its stray-registry warning fires on a seeded file. ft-layout's vrcmd: 119 overlays (was 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:53:59 -06:00
DeeJanuzandClaude Opus 5.5 d39fbb9146 Gaze: our eye tracker can take its first calibration
A fresh install that chose our tracker (the installer's default since 12ad93d) could never
calibrate it, so gaze mode never worked. ft-eyes maps pupils to a gaze only once it has a
calibration, so before its first one ft-gaze's "own" source is empty. ft-gazed then never
counted a sample, Checks.can_run() stayed false, and the calibration refused to open: "Not
calibrated, and the calibration can't open: the eye tracker isn't sending", or "the headset is
off or the tracker isn't sending" from Calibrate. This Frame never hit it, because its
calibration came from the gaze probe. Reported 2026-10-05 on SteamOS stable.

- With our tracker in use, running, and uncalibrated ("blind"), SteamVR's tracker seeing an
  eye is enough to open the full calibration (ft-eyes answering is what makes calibrated()
  False rather than None).
- Each dot stands in for the gaze, so a click takes the CHECK_WINDOW up to it. ft-eyes already
  checks, in calib-point, that each pupil was seen in enough frames and held still then, and
  its reason for a refused dot shows in the panel's note as before.
- A quick or five-dot check is refused until then ("use Calibrate"): ours can't take a click
  without a calibration.
- With no eyes seen, the reason given is that, not "the eye tracker isn't sending".

gaze/test/first-calibration-test.py runs the service against a fake ft-gaze, ft-eyes and
pointer helper, in a temp HOME: the calibration opens by itself when gaze mode comes on, each
click sends ft-eyes the 0.6 s before it, a refused dot's reason reaches the panel, and
calib-fit leaves the tracker calibrated with gaze mode still on. It passes here and fails on
the previous code. gaze/test/idle-test.py still passes. Not yet tested in the headset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 10:51:07 -06:00
DeeJanuzandClaude Opus 5.5 7fe5d8c0dd Merge branch nested-atspi (PR #28: the desktop starts an accessibility (AT-SPI) registry, #27) into experimental
JakeGreen2145's PR plus our fix 6df6f97: the watcher checks its buses
every 5 seconds instead of every second (about 1% of a core), and
design.md says the watcher is the only cleanup when the VR launcher
runs the desktop in steam.service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:27:26 -06:00
DeeJanuzandClaude Opus 5.5 6df6f97937 Session: AT-SPI watcher checks its buses every 5 seconds
Each check starts two gdbus processes (about 5.5 ms of CPU each on the
Frame), so checking once a second cost about 1% of a core for as long as
the desktop ran. On SteamOS 0.3.0 native activation fails, so the watcher
always runs. A registry left behind now goes within 5 seconds instead of 1.

design.md also says where the watcher matters: started from the VR
launcher, the desktop runs in steam.service, which doesn't stop with it,
so keep-apps.sh and the unit's stop don't clean up there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:23:51 -06:00
DeeJanuzandClaude Opus 5.5 ca9492be00 Pages: link the Frametop Discord at the top and bottom of the hand recorder page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 19:50:12 -06:00
DeeJanuzandClaude Opus 5.5 41b5b68a41 Pages: hand recorder install page
hand-recorder.md, served at deejanuz.github.io/frametop/hand-recorder.html:
the Konsole commands to install, update and uninstall the Hand Recorder,
who can take part, and what it needs. Frametop comes first for now, since
the recorder runs in its desktop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 19:47:13 -06:00
9d7c8a0b91 Experimental (#29)
* Input relay: typing with the pointer helper down no longer ends the relay

Typing on a pass-through keyboard tells the helper "typing". With the
helper not running (SteamVR off), that send raised ConnectionRefusedError
and the relay exited, dropping every grab until systemd restarted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-steam: open Steam's menu through Steam's own UI

steam/ft-steam menu opens the SteamVR dashboard on Steam's menu, or closes
the dashboard if it's up, without pointer mode: it asks Steam's UI over its
debugging port to show its dashboard overlay (ShowVROverlay, what Steam
calls itself) and focus the Steam frame's left menu (MenuStore.OpenMainMenu).
ft-steam check says whether those calls still exist, and update-check.py
runs it, since a Steam client update can rename them.

The CDP client moves from display-settings/steam_settings.py to
steam/steamui.py, so both use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Shortcuts: Steam menu, commands, and modifier taps

Key combinations (and mouse and controller buttons) get two new actions:
- steam_menu: Open Steam menu / close dashboard (steam/ft-steam menu).
- command:CMD: run CMD with sh -c, as the relay's service, with layout/,
  float/ and steam/ on its PATH. Input Settings offers it for key
  combinations as Run a command...
Both work without pointer mode.

A modifier on its own is now a key combination too: a tap, pressed and
released with no other key, mouse button, or scroll in between. A bound
tap sends the desktop F24 before the release, so Plasma's launcher stays
shut. The defaults gain a Meta tap for the Steam menu; this replaces
META_DASHBOARD, which only worked in pointer mode.

input/test/keys-test.py runs the relay against fake devices with every
outgoing socket renamed, so it's safe next to the live relay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Relay: share a key combination's Meta release with frame-voice

A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release
from the desktop, and the relay skipped share_key for it too. frame-voice
saw Meta go down on @frametop_keys and never come up, so it held all
dictated text back, waiting for that release. Keys of grabbed keyboards
are now shared as pressed, before key_binding() decides what the desktop
gets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-cutouts, the hand cutouts without pinches and grips

hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient
user units with ft-hands' new --no-gestures: hands are published for
ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks
or drags and a closing hand doesn't raise the tracking rate. It needs a
build and ft-camd's capabilities, not hands/run.sh install. Its units
conflict with ft-handsctl's, so each stops the other, and they stop with
SteamVR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-cutouts status: only the current run's tracker lines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: say why gaze mode can't work yet, and open the calibration whenever it's missing

Turning gaze mode on without a calibration opened Calibrate only on the
off-to-on change, and only if it could open right then. With the headset
off, the eye tracker silent, or the panel not built, or with gaze mode
already on when the gaze service started, nothing opened and nothing said
why: the pointer just stayed a mouse.

- The gaze service now checks every second: gaze mode on, no calibration
  for the tracker in use, eyes seen -> the full calibration opens. One
  that closes unfinished opens again only after the headset comes off and
  on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that
  fails retries every 10 s.
- Its status says why gaze mode can't work yet (checks.problem): not
  calibrated and opening, open, closed unfinished, or can't open and why.
- Input Settings shows that under the Gaze pointer switch, along with the
  gaze service not installed or not running and our tracker missing its
  frame grabber.
- ft-gazectl on notes a missing calibration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken

A user on a fresh install got "Calibration failed: only 0 of 21 dots" with
no reason. The installer never installed our tracker, so gaze used
SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an
eye for most of the look. The panel just showed a red ring.

- install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh)
  after gaze mode, yes by default; it needs sudo, so --yes runs it only
  when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker.
  Configs that still say GAZE_TRACKER=steam (the old template) are asked
  whether to switch.
- GAZE_TRACKER=auto, the new default: ours when it's installed (the
  frame grabber, its unit, and ft-eyes' Python), else SteamVR's.
  ft-gazed rechecks every second, so installing it switches over. Input
  Settings lists Own tracker first as recommended, and says how to
  install it when it's missing (checking the host's /etc through
  /run/host from the dev container).
- The calibration panel has a note line, orange over the instructions:
  why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for
  SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply
  for ours), what a click is still waiting for after 1.5 s, and a failed
  calibration's most common reason, which the Gaze page shows too.
  steady_samples keeps the same samples as before (checked on 2037
  windows of recordings); a lost eye is named before a blink, since its
  openness reads 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* README: link the Frametop Discord

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Wait for a new container to finish setting up before entering it

container-up.sh starts the dev container in a scope of its own, so
distrobox enter finds it running and skips its wait for distrobox-init.
On a fresh install, init was still setting up passwordless sudo when
dev-container.sh ran sudo dnf install, and sudo asked for a password
with no terminal to read it from. container-up.sh now waits for
container_setup_done itself, and the container's sudo calls use -n,
so a password prompt fails at once with a clear message.

Fixes #9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Prevent small desktop overlay pointer movements from starting a drag

* Clear reported drag state on controller release

* Click stability: only a hand controller's press starts it

The 3D mouse drives SteamVR's laser through the ft_pointer virtual
controller, so its events reach the screens the same way a controller's
do. The filter held every press, which turned the mouse's short drags
(selecting a character or two, nudging a slider) into clicks. Mark
button events from hand controllers and start the filter only on those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: retry a new device until udev gives it to the input group

A new /dev/input node is root:root 0600 until udev applies GROUP=input. The
scan probed each new node once and marked it seen even when the open failed, so
a node caught in that gap was never opened. Behind a KVM, a switch brings back a
hub of devices at once: on the Frame, four nodes failed with EACCES in one switch,
the keyboard was never grabbed, and its keys went to gamescope instead of the
desktop screens. A node that isn't readable yet now waits for the next scan.

* Screens: take a screen's overlays from one copy in the catcher

While a button pressed on a screen is held, UpdateCatcher checks every tick
whether the laser is still on one of the screen's overlays. It built that list
from s.All().begin() and s.All().end(), but All() returns a std::array by
value: iterators into two different temporaries, which is undefined behaviour.
A clang build of ft-screens got a garbage length, threw std::length_error, and
aborted on the first click, taking KWin and the desktop with it.

* Gaze: leave SteamVR's gaze action alone during VR games

From curiousjtuber's PR #13: with the gaze service running, SteamVR
restarted its eye tracker every 10 to 13 s of Beat Saber, as if the
headset came off, and each restart took input focus from the game.
The PR stopped every read in a game. Only the action path reaches
SteamVR (UpdateActionState on the gaze set at overlay-global priority,
then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are
read-only files. So only the action is skipped while a scene app runs,
and gaze keeps moving the pointer over the dashboard in a game. The
action source is only used with --source action.

Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: --record-hz, and the hand recorder's design (hands/rec/DESIGN.md)

ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10).
DESIGN.md lays out the recorder: the headset panel, the session runner and its script,
the files, review and export, consent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-handpanel, the hand recorder's headset panel

A head-locked SteamVR overlay for the hand recorder (hands/rec/DESIGN.md):
1.2 m ahead, 12 degrees up, 36 degrees wide, drawn with stb_truetype
into three shared DMA-BUFs as ft-gazepanel does. It shows the title,
step, wrapped instruction, note, countdown, hand chips, near/far bar
and a "Paused" cover, driven over @ft_handpanel.

It also places the touch target, a 2 cm dot in its own overlay fixed
in the room where the head was at the first command for that point,
and logs head and controller poses to poses.jsonl at 250 Hz from a
thread of its own. Both threads take one lock around OpenVR calls.

--no-vr prints each picture's state to stdout (and --dump writes the
pictures), for testing without a headset. hands/rec/build.sh builds it
in the dev container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder's worn check goes by the panel's backlight, as frame-job does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the hand recorder's session runner and guided script

hands/rec/session.py runs a recording session from script.json: it starts
ft-camd and a tracking ft-hands as transient units only if they aren't
running, records each section as one take (ft-hands --record-only at
10 sets/s, a new sets-N.bin after each pause), drives ft-handpanel, and
writes session.json, calibration.json (identifying fields removed),
prompts.jsonl and take.json. Feedback comes from the live hands file and,
in the controller sections, from the panel's device poll. It also runs
from the command line (--dry-run, --speed, --ring, --no-start).

hands/rec/script.json: 11 sections, about 9 minutes without the object
and controller sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the hand recorder's window, review and export

hands/rec/ft_handrec.py + main.qml (Kirigami, dev container; host launcher
hands/rec/ft-handrec): consent (CONSENT.md, asked again when its version
changes; profile.json with a random contributor id), the before-you-start
checklist with the lighting and free-space checks, the session controls
(Space pauses, Esc stops), review with a frame-set viewer that deletes
ranges, takes and sessions, export with progress and cancel (warns while
the headset is worn), and the upload page (UPLOAD.md, the
huggingface-cli command; HF_DATASET is a placeholder). --dry-run runs
sessions without processes, for testing.

hands/rec/takes.py (standard library): indexes sets.bin and sets-N.bin
without reading pixels, reads one set's cameras, keeps deleted ranges in
take.json, and exports: deleted sets left out, zstd -10 -T2 at nice 19,
manifest.json and SHA256SUMS, nothing left behind on cancel.

CONSENT.md and UPLOAD.md are drafts pending a legal review; the window
says contributions aren't open yet. The dev container gains zstd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: upload from the hand recorder's window, export checks, a rehearsal

hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+)
checks an export before upload and when it's received: SHA256SUMS, an
allow-list of files, the manifest's schema and keys, the consent version,
a uuid4 contributor, no identifying fields in calibration.json or
device.json, every sets.bin.zst decompressed to its end as a stream with
each FHSET01 header checked against the manifest, jsonl lines, the total
size. It decompresses with compression.zstd, zstandard or the zstd
program. validate.py DIR [--json].

hands/rec/hub.py uploads an export with huggingface_hub, as a pull
request to contributions/<contributor>/<session>: validate first, refuse
a repeat of the same export, check the login (whoami) and access
(auth_check), upload_folder(create_pr=True) with the manifest summary as
the description, then record the PR under "uploads" in session.json.
Errors are explained (terms not accepted, not found, 401/403, network).
--dry-run makes no network calls. FT_HANDREC_DATASET overrides
HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real
upload needs FT_HANDREC_ALLOW_UPLOAD=1.

The Upload page shows the login with "Check again" and how to run
hf auth login in a terminal (the token never enters the window), then
Upload with a phase, progress and Cancel (hub.py as a child process), the
PR link, and a warning for an export uploaded before. The manual
command stays as the fallback. ft-handrec --hub-dry-run.

session.py also saves device.json: cv.cad_from_cal and head from
/persist/device_config.json, the labeller's shape, nothing identifying;
export copies it. Session ids with a -N suffix are accepted everywhere.

hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays
30 s of a capture into a ring, session.py records a short test script
with ft-handpanel --no-vr and a tracker, then export, validate and a
dry-run upload (--repo ID uploads for real). It runs in one frame-job
scope, deletes its data and stops its processes, also on Ctrl+C.

hands/rec/tests/test_validate.py covers good and broken exports and hub.py
without the network. The dev container gains python3-huggingface-hub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* README: Frametop doesn't work on the SteamOS beta yet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 072a294941)

* Hands: step mode and pose pictures for the hand recorder

The first real session moved on every 5 s with text only, too fast to follow.
Each step now waits for Next (Space or the window's button), counts down 3-2-1
while recording, then holds. P pauses, R redoes a step, S skips a section;
"Advance by itself" (--auto) keeps the old timed flow. Nothing records while
a step waits: each step is its own recording part.

The panel and the window show a picture of each pose (hands/rec/poses,
generated by make_poses.py from a parametric hand, MIT) and a diagram of where
to hold the hands and how far out. prompts.jsonl gains ready, wait and redo
events; session.json gains mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the headset button as Next, clearer push steps

The headset's right-side click button (KEY_SELECT on gpio-keys, read without
a grab) now works the session: Next while a step waits, pause during a hold,
resume while paused. With no mouse connected the hints lead with it.

The push sections say plainly to push straight out from the headset and pull
back, with a side-view picture of the head, the headset and the arrow, and
the bar's ends read "At your chest" and "Arm out". The bar labels are sent
as one field, so labels with spaces no longer split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-floatd: a launch for a missing app no longer kills the control socket

Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.

Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-floatd: profiles keep and relaunch Flatpak apps whose window names another app id

An X11 window in a Flatpak can give KWin an app id with no desktop file:
RustDesk's says com.carriez.flutter_hbb (its GTK application id), and the
Flatpak's desktop file is com.rustdesk.RustDesk. A profile recorded that
id, so it couldn't relaunch the app (PR #16 keeps that from killing
ft-floatd's socket). And the window's pid is the sandbox's own, so a
launched window matched neither by process nor by app id, and didn't
float.

desktop_name() finds the desktop file whose StartupWMClass names the
window's class (or app id) when the app id has none. Capture records
that name, a profile claims open windows by it, and a launch's window
matches by it.

Profiles saved before this keep the old id; save them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: check the tracking cameras before recording, and watch for losing them

After the headset wakes, XRService sometimes fails to load the colour module's
VCINT FPGA image; then only the two side cameras run, without the IR light,
and the tracker finds no hands. hands/camcheck.py reads XRService's log, the
video nodes it holds and ft-camd's ring, and says ok, degraded or unknown.

The recorder won't start while degraded (--ignore-cameras overrides it), offers
a confirmed SteamVR restart, and stops the first hand-size step when the
tracker sees no hand at all. ft-camwatch (unit file only, not enabled) follows
the log, notifies, and with CAMWATCH_AUTO_RESTART=1 restarts SteamVR when the
headset isn't worn and nothing else uses VR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: shorter recording sessions with pose sweeps

The second real session took 16 minutes, half of it 36 still poses. Labels
come from the auto-labeller, so what matters is variety, not clean holds.
A sweep step shows a strip of pose pictures and lights one every 4 s while
the hands move slowly near and far; each cue is a prompt event with
"cue": true. The core session is now 15 steps, about 5 minutes recorded.

The pose groups, the one-hand sweeps' groups and the cue order are shuffled
per session, seeded from its id and saved in session.json. A quick round
(--quick, or the checklist's choice) is about 2 minutes for extra lighting.
Touch the dot has 6 dots, the push sections two heights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay test: let the fake devices past the udev permission check

Since the relay leaves a node it can't read yet for the next scan (12f2e84, PR #12), it
checks os.access first, and the test's fake /dev/input paths don't exist, so the relay never
opened them and every key check failed. The fake os now says they're readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pause Frametop for VR games

Frametop kept using the headset during games: with gaze mode off, our eye tracker still took
about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden
screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and
resuming brings back only what pausing stopped. It's also a way to keep the gaze service and
our eye tracker off during games, which PR #13 asked for.

Paused (input/game_pause.py, run by the input relay):
- frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop
  stop if they run
- the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a
  frame callback once a second; or, with pause_desktop "close", the desktop closes and starts
  again on resume
- the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key
  combinations do only pause_toggle, steam_menu and commands

Toggled by both thumbsticks clicked together twice (configurable), read passively from
vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the
new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game
starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input
Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a
paused gaze service as failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: ft-hands works out which side camera is which

ft-camd tells the side cameras apart by XRService's buffer allocation order,
which some XRService starts reverse; both of 2026-10-02's starts did, so the
cutouts missed the hands. HANDS_SWAP_SIDES=auto (the default) has ft-hands
vote from hands seen in both side cameras: the landmark rays meet in front
of both cameras only under the right naming. While undecided it probes the
exchanged naming with the landmark model. It decides in about 2 s of hands
(right on all 7 recordings replayed), swaps the views in place, and publishes
sides.json. 0 and 1 still force it, with a warning when the hands disagree.

Recordings carry each part's naming and the session's decision; review,
export, validate and ft-handreplay put the names right, and takes.py sides
records a decision by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: idle while the gaze isn't used

The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes
still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze
and our tracker run only while gaze mode is on and someone wears the headset, while a check or
the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop
Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber
idles with our tracker.

- The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the
  headset); an older helper answers it as before, and the service then goes by gaze mode alone.
- A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once
  it sends; the automatic calibration waits quietly while it starts.
- Status has "awake" and "idle" (why), and the Gaze page shows it.
- gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* update-check: a still controller isn't a broken web socket

vrserver sends a controller's state only when something on it changes, and one lying still
or asleep may not even send its first one. The check subscribed to one controller and failed
after 3 s of silence. It now subscribes to all, and silence after a good handshake is a skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder measures the light itself

The checklist page measures the light when it opens, starting ft-camd if
nothing runs it (and stopping it on quit), instead of saying the cameras
aren't running. The round's lighting defaults to what the cameras measure:
daylight or indoor, from the mono cameras' ambient infrared. Lamps give off
little infrared, so dim and normal rooms read alike; picking dim, room or
daylight still overrides it. session.json gets source, measured and
ambient_ir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: the recorder's host commands run from the home folder

host-spawn starts a host command in the caller's folder. Started from /tmp,
the app's folder in the container is /run/host/tmp, which the host doesn't
have, so starting ft-camd (and every other host command) exited 127.
host_command now runs from home (env -C), and the launcher cds there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input Settings: the Games page is Game optimization

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: push steps say to follow the hollow circle, not the blue dot

The dot is the current tracker's distance guess, often wrong; the ring is
where the hands should be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: one thread for OpenCV and numpy

Nothing called cv2.setNumThreads, so OpenCV kept a pool of one worker per core for
pupil windows of 140 to 240 px. Live, its idle workers spun and yielded about 14,000
times a second each, about a quarter of a core, next to SteamVR's compositor. numpy's
OpenBLAS also started 8 threads that never had work.

eyes_pupil.py now sets OpenCV to one thread, and ft-eyes sets OPENBLAS_NUM_THREADS and
OMP_NUM_THREADS to 1 before numpy loads (a value already in the environment wins).

Replaying fit1 into a scratch share (ft-eyes-replay, 14 s measured, capped at one core
with the replay): threads 13 -> 1, involuntary context switches 3,812/s -> 430/s,
system time 6.9% -> 1.6% of a core. Under that cap the frames it kept up with went from
21-36 to 57-69 a second per eye.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: read the overlay list every 20 s, not every second

The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze
mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about
26 to 30 ms of CPU, so about 3% of a core all the time.

The list is now read every 20 seconds, and at once (at most once a second) when it may have
changed: the pointer waking, the dashboard opening or closing or creating an overlay, the
scene app changing, an "overlays" request, and a left click that hit nothing, which may be
on a panel that came up since. The thread waits on a condition variable instead of waking
every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a
new list is in, rather than at its next 1 s tick. Overlays already on the list still show
and hide within 50 ms, from the IsOverlayVisible poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-eyes and ft-gaze below SteamVR's priority

ft-eyes and ft-gaze run in the dev container through distrobox, so they live in podman's
libpod scope: frametop-gaze.service's limits never reach them, and they ran at nice 0
next to vrcompositor and vrserver, also at nice 0.

- ft-eyes sets itself to nice 10 and SCHED_BATCH at start, before its threads. Batch
  turns off wakeup preemption, so a frame ft-eyes wakes up for can wait out a running
  compositor's turn; a few ms late costs the gaze little.
- ft-gaze sets nice 5 before its threads start, but stays SCHED_OTHER: each sample goes
  on to the pointer, and batch would add the same wait to every one of them.
- Both only ever lower their priority (a higher nice already set wins), and a failure
  is logged and ignored. Checked in the dev container: nice 0 -> 10, policy 0 -> 3
  (SCHED_BATCH) without any capability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: sleep on the command socket while the pointer is off

The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake
or not, and every second it looked up every overlay's handle and read a string property from
all 64 device slots to find its own device.

With the pointer off and hand gestures off, the loop now waits in poll() on its command
socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read
(SteamVR input has no event to wait for). A mouse command ends the wait at once. The
headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep
going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the
pointer is awake, and waking forces both. The device index is looked for only while it's
unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is
kept only with hand gestures on, its one user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote desktop: connect FreeRDP only while a VNC viewer is connected

vnc-bridge.sh kept FreeRDP connected to krdpserver from the moment remote desktop
started, so krdp captured and H.264-encoded every KWin redraw in software (openh264)
with nobody watching: krdpserver 55-78% of a core, xfreerdp 16-27%, Xvnc 6-11%, with 0
clients on :5900. krdp 6.7 creates its screencast session per RDP connection and drops
it when the connection closes, so krdpserver itself idles without one and stays up.

The bridge now counts established connections to Xvnc's port with ss, starts FreeRDP
when a viewer appears (the desktop shows about 3 s later; the VNC screen is black until
then) and stops it 45 s after the last one leaves (VNC_IDLE_SEC). Xvnc has no client
hook, so its log output, which it writes for every connection, wakes the bridge early;
otherwise it looks every 5 s while idle (0.1% of a core measured, against 0.9% for ss
once a second) and every second while FreeRDP runs. The layout check runs only while
FreeRDP runs.

While a viewer is connected the bridge sends "watch 15" to ft-screens (@ft_screens) at
once and every 5 s, so screens at a reduced frame rate (out of view, headset on a
stand) stream at full rate; it lapses by itself if the bridge dies, and an ft-screens
without the command just answers an error. The window search after starting FreeRDP
now ends when FreeRDP exits instead of polling for 30 s.

krdp on 127.0.0.1 with a fresh password, VNC on the tailnet address with VncAuth, and
remote-ctl.sh start/stop (pause and resume) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote desktop: read the layout only after it changes

While FreeRDP ran, vnc-bridge.sh called ft-layout remote-view every 5 s, which scans
all of /proc for plasmashell and runs kscreen-doctor -j: about 4.4% of a core for a
layout that rarely changes.

It now stats the two files the answer depends on, the nested KWin's
~/.config/frametop/kwinoutputconfig.json (positions, scales, primary) and
~/.config/frametop-layout.json (screen sizes), once a second while FreeRDP runs. After
either changes it reads the layout every 2 s for 10 s, since KWin's outputs follow the
file a few seconds later; otherwise once a minute, in case a change touched neither.
With no VNC viewer connected nothing runs (previous commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: ft-eyes sleeps until the next frame is due

ft-eyes looked for new frames about 1,000 times a second: each pass of its loop asked the
control socket with a non-blocking recvfrom (a BlockingIOError nearly every time), read
both cameras' counters, and slept 1 ms. The frames come every 11.1 ms per camera, and only
as counters in ft-eyegrab's shared memory, so there's no fd to wait on.

Now each pass ends in select() on the control socket, with a timeout until 2 ms before the
next frame of either camera is due (from when its last one was seen), then every 1 ms until
it comes. A command wakes it at once. A camera with no frame for 0.1 s (headset off,
grabber idle) isn't waited for, and with both stopped it looks every 20 ms. Waiting for the
frame grabber's file uses the same select, 0.2 s at a time, instead of sleeping through
commands.

A new frame is still seen within about 1 ms of when it lands. On a synthetic share at 90 Hz
per camera, on a heavily loaded headset (load average 23, so ft-eyes rarely sat idle), its
waits went from 178 to 81 a second; unloaded, the old loop's 1 ms sleeps add up to about
1,000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: frame rates by attention, ticks in step with the display

ft-screens gave KWin a frame callback for every committed screen on each tick, and the tick
was an 11 ms timer set again after each run, so it slid through the display's frame and came
about 85 times a second at 90 Hz: the desktop repeated a frame several times a second (judder
in scrolling and video), and KWin drew every screen in one burst at a random point of
vrcompositor's frame. Hidden screens got the same 90 Hz unless Frametop was paused for a game.

- Ticks run on a timerfd at absolute times, once per display frame, 1 ms after the vsync
  (IVRSystem::GetTimeSinceLastVsync and the HMD's display frequency, read once a second), so
  KWin gets its callbacks early in the frame. Measured with --no-vr: 91 wakeups a second
  instead of about 85. On the Frame the vsync times SteamVR reports lie on a 90 Hz grid.
- Each screen's callbacks come at a rate for how much of it you see (vr.cpp,
  UpdateAttention): every frame while focused (within 12 degrees of where your head points,
  a laser or the mouse on it in the last 1.5 s, carried, or typed on), 15 a second for the
  rest of what you see (within 60 degrees), and 1 a second when hidden, behind you, or
  paused. Levels rise at once and fall after 1.5 s (focused) or 0.5 s (in view). KWin draws
  a screen only after its callback and its apps wait for theirs, so this throttles the apps
  too. A screen where nothing changes costs nothing at any rate, as before.
- A video in view keeps every frame: 8 commits in a row that each redraw 6% or more of the
  screen, at 10 a second or more, count as one (from the surface's buffer damage).
- "rates F V H" / --rates set the three rates (default 0 15 1, 0 = every frame), "rates?"
  shows them and each screen's level, "watch S" gives everything full rate for S seconds for
  a remote viewer (vnc-bridge.sh renews it), and "phase MS" moves the ticks for tuning.
- ft-screens' main thread runs at nice -5 after the session starts: SteamOS allows down to
  -8 once the soft RLIMIT_NICE is raised, and KWin waits on these ticks. It had spent nearly
  3 times as long waiting to run as running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: skip unchanged work while the pointer is awake

Every frame (about 116 a second) the helper tested the cursor ray against every visible
overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and
visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the
mouse and the head still.

Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the
eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run
at least every 100 ms, since overlays move on their own (a floating window's controls follow
it), and always while dragging. The dots' setters go to SteamVR only when their value changes:
the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well
under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the
laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and
at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's
pose, a placement, or waking sends the next one regardless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Session: blur, background contrast and animations off by default

The nested kwinrc had no [Plugins] group, so KWin ran its default blur and background
contrast effects, and kdeglobals had no AnimationDurationFactor, so animations ran at
full length. KWin renders through zink on Turnip, on the GPU vrcompositor needs, and
blur re-renders what's behind every translucent panel and menu; each animation frame is
another frame for KWin and ft-screens.

Before KWin starts, the session script now writes [Plugins] blurEnabled=false and
contrastEnabled=false to $XDG_CONFIG_HOME/kwinrc and [KDE] AnimationDurationFactor=0 to
its kdeglobals, each only if the desktop's own file has no value for it. It does this
once and records that in $XDG_CONFIG_HOME/frametoprc ([Defaults] effects=1), because
System Settings deletes a key put back to its default: without the marker, turning blur
back on wouldn't survive a restart. The ids blur and contrast are the built-in effects
of KWin 6.2.5 on SteamOS (both enabled by default in its plugin metadata). Tested
against a temporary XDG_CONFIG_HOME: fresh config, an existing blurEnabled=true kept,
and a deleted key not rewritten.

README and docs/reference.md say how to turn them back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Session: don't autostart Discover's notifier or IBus in the desktop

The nested Plasma session runs the system's XDG autostart entries. Discover's update
notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) started
plasma-discover --mode update inside it, 520-620 MB resident and about 9% of a core,
with flatpak-system-helper and AppStream downloads behind it. IBus started a nested
ibus-daemon with kimpanel and ibus-extension-gtk3, which no app in the desktop can use:
KWin's input method is ft-textinput (zwp_input_method_v1, focus reports only; the VR
keyboard types through ft-screens' seat), and the session already drops QT_IM_MODULE,
GTK_IM_MODULE and XMODIFIERS. Nothing in Frametop talks to IBus.

Before Plasma starts, the session script copies both entries into
$XDG_CONFIG_HOME/autostart with Hidden=true, which plasma-session honours for that
desktop only. It does this once ([Defaults] autostart=1 in frametoprc) and skips a name
the user already has a file for, so deleting the copy brings the program back. The
geoclue demo agent stays (it answers apps' location requests outside GNOME and idles at
0%), and orca's entry is OnlyShowIn GNOME-family desktops, so it never ran. Tested
against a temporary XDG_CONFIG_HOME, including an existing user ibus.desktop left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: never block on the pointer helper's socket

The relay sent to @ft_pointer_helper on a blocking socket. When the helper stalled, a
layout placement or grabprobe holds it for seconds while ft-gazed keeps filling its socket at
90 Hz, the relay's one loop blocked with it: keyboards, the volume keys (which must never
reach gamescope), and pausing all stopped until the helper read again.

The socket is non-blocking now. A command the helper doesn't take (EAGAIN) waits in a queue,
and everything after it queues behind it so the order holds; tick() sends what it can on each
loop, and the select timeout drops to 20 ms while anything waits. Mouse moves add up into one
queued move. A scroll notch is dropped rather than queued, since scrolling seconds late is no
use; its release still goes. Presses, releases, show, hide, and the rest are kept, so no
button stays down. The queue holds at most 512 commands. While paused, the configured
pointer's queue still drains, so the releases and "hide" from standing down arrive. A
"vrbind" that hits a full socket is sent again on the next loop instead of being lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer driver: parse outside the lock, report only changes

Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and
RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands
(about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands
are now parsed into locals first, and the lock is held only to store the result.

RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every
frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to
SteamVR only when they change (all of them on the first frame). The pose still goes out every
frame while the device is connected, as a tracked device's should; the disconnected pose goes
out once. The helper now sends a pose only when it changes, so the comment says the driver
keeps the last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-powerd: ask SteamVR every 100 ms, not on every input event

The loop polled the input devices with a 100 ms timeout and then, on every wake, did
SteamVR's part: PollNextEvent, the headset's activity level and every device's pose, all
IPC calls to vrserver. Input wakes it at once so the displays come on with the first
key or motion, but a moving mouse sends hundreds of events a second, so moving the mouse
meant hundreds of rounds of IPC a second instead of 10.

Every wake still drains the input devices and the control socket and counts input as
use straight away; SteamVR's part, and the backlight read that goes with it, now run
only when 100 ms have passed since the last time, and poll sleeps until then. Built in
the dev container (power/build.sh, no warnings); not run, since the live ft-powerd holds
@ft_powerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Eye tracker: ft-eyegrab checks only the slot each camera writes next

While copying, ft-eyegrab woke every 300 us (about 1,500 to 3,000 times a second) and
fingerprinted all eight slots each time: 8 x 256 strided reads from DMA-BUF memory.

- Each look now checks only the slot each camera writes next. The order is known (camera 0
  3,0,1,2; camera 1 7,5,4,6,5,7,6,4), and the next slot follows from the last two; the table
  starts from those orders and learns from every frame, so a SteamVR update that changes
  them costs a few seconds of full scans, not frames.
- A camera with nothing in its expected slot 1.5 frames after its last one, or with no
  order yet, gets all four slots checked, as before. A frame that turns up in an unexpected
  slot means full scans for that camera for 2 s.
- A slot's fingerprint is taken again when it stops being one of the two in use, so a later
  check sees only a new frame. A frame is still passed on when its camera starts the frame
  after next.
- Between frames it sleeps until 2.5 ms before the next is due, then looks every 1 ms, with
  0.5 ms of timer slack (PR_SET_TIMERSLACK, --share only). With no frames from either
  camera for 0.5 s (headset off) it looks every 4 ms.
- --rec keeps its 0.3 ms polls (and the expected-slot checks), for its timestamps.

Tested offline by building poll_frames against simulated cameras that write each frame in
four bursts, the last after the next frame starts (6 s, both cameras): 1,076 frames passed
on, none torn or skipped, with the known orders and with camera 1 in a different order.
Wakeups 1,486/s -> 207/s, the poller's CPU 3.6% -> 0.8% of a core (in plain memory; the real
DMA-BUF reads cost more), and a frame's start is seen 1.35 ms after it begins on average
instead of 0.76. With a camera stalling 15 ms every 2 s, the old poller passed on 22 torn
frames and the new one 8 or fewer.

Built (glibc 2.38 symbols at most, the host has 2.39), not installed: it runs as root from
/etc/frametop, so it takes effect only after gaze/tracker/install.sh (sudo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: upload from the headset, then plug in and leave it

Export and upload are done in the headset now: the export page only notes
that VR may stutter a little. Upload opens the pull request first (a
draft) and shows its link, telling the person to plug in the headset and
leave it until it says Uploaded; the files then go to refs/pr/N, and the
pull request is marked open at the end. A retry of the same export goes on
in the same pull request. While an export or upload runs, a host unit holds
a logind sleep inhibitor so the Frame stays awake with the headset off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remote Access: check the status every 5 s instead of every 2 s

While its window was open, Frametop Remote Access ran remote-ctl.sh status every 2 s,
and each run spawns bash, curl (the tailnet name from tailscaled) and python3 to parse
it. It now checks every 5 s, plus when the window comes to the front and once more 2 s
after turning remote access on or off or changing the password, so a change still
shows within a couple of seconds. A check doesn't start while one is still running.
Doing the check in-process would duplicate remote-ctl.sh's idea of "running", which
the session and the pause code share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* update-check: KWin's blur and contrast effect ids, retest hints

The session now turns KWin's blur and contrast effects off by id (blurEnabled and
contrastEnabled in the desktop's kwinrc), and a KWin that renamed them would quietly
leave them on. The check looks for their built-in factories (KWin::blur_factory,
KWin::contrast_factory) in kwin_wayland, which it already reads for --output-count,
and warns if one is gone. The kwin and plasma-workspace retest hints gain the blur and
the hidden autostart entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pause gesture: look the controllers up every 30 s, not every 3 s

The gesture reader fetched vrserver's /input/getstate.json over HTTP every 3 seconds, the
whole time the relay runs, to notice a controller's root path changing when the 3D mouse
takes or gives back its hand role.

It now looks them up when it connects, when a message comes from a device path it doesn't
know (at most every 3 s; the device is read from the message with two string searches, not
a JSON parse of all 160 a second), 1.5 s after the relay's 3D mouse connects or lets go (the
relay tells it through GamePause.controllers_changed), and otherwise every 30 s. The keys
test's pause stub gets the new method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Input relay: send mouse motion at most every 4 ms

The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams
a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf
and strncmp tests in the helper before reaching the move handler. In a 200 ms test at
1000 Hz, 149 reports now make 45 moves with the same total.

flush() on a report now sends only once 4 ms have passed since the last move; tick() sends
the rest when due, and the select timeout shrinks to match. Buttons and the gaze
keys still flush first, unconditionally, so a click lands where the pointer was. In the
helper, "move" is now tested first in the command dispatch, and its handling is one lambda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-gaze prints and reads only the sources in use

ft-gaze computed and printed all six sources for every sample: about 1.3 KB of JSON a line
with our tracker (practice2), 120 KB a second through podman's stdio relay for ft-gazed to
json.loads 90 times a second. It also read SteamVR's gaze action for every sample outside
games (UpdateActionState and GetEyeTrackingDataRelativeToNow, two calls into vrserver, 180 a
second), though with our tracker ft-gazed only uses own and mmap1.

- ft-gaze takes --sources LIST (action, mmap1, mmap2, left, right, own, and eye for the EYE
  object; all by default, so the probe and ft-eyes-session are unchanged), and with
  --watch-stdin a line "sources LIST" on stdin switches them. A source left out isn't read
  and prints as {"ok":0} ("eye" as null), so every line keeps the same keys. An older
  ft-gaze ignores both, and prints everything as before.
- ft-gazed asks for what it reads: own,mmap1 with our tracker; left,right,mmap1 with
  SteamVR's eyes; the source plus mmap1 and mmap2 on the older one-source path. While a
  check or the calibration runs or waits to open, all of them, since checks record every
  source (the calibration fits the action's correction too) and the fit check reads "eye".
  It switches as soon as that changes, well inside the check's 0.45 s settle.
- So the action is read only during checks, or with --source action.

On recorded samples, a line with own and mmap1 is about 700 bytes instead of 1,300
(practice2), and one with left, right and mmap1 about 550 instead of 940 (test1).
gaze/test/idle-test.py now checks that ft-gaze starts with every source for a check and is
then switched to those in use, without the action or own; all its checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: don't put a vanished panel back in the visibility map

The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: ft-gaze's loop runs every 4 ms instead of 2

ft-gaze's loop slept 2 ms, so 500 times a second it read the head pose
(GetDeviceToAbsoluteTrackingPose), checked the eye tracker's counter, and drained SteamVR's
events, for samples that come 90 times a second.

It now sleeps 4 ms. A new sample is printed within 4 ms of appearing, 2 on average (was 1),
and the pose history still has a pose within 2 ms of any sample's time, which keeps the
head-pose error under 0.2 degrees for a head turning 100 degrees a second. Sleeping until
the next sample is due would have thinned the pose history to 11 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Gaze: the hidden panel waits for a command instead of waking every 50 ms

The calibration panel runs for as long as the gaze service does, hidden nearly all the time,
and it woke 20 to 30 times a second to look at its socket and SteamVR's events: about 0.9% of
a core, the main cost left with gaze idle.

It now waits in poll() on its command socket: up to a second while hidden, and up to 10 ms
while shown, as before (it still drains SteamVR's events each pass, so a quit is acknowledged
within a second while hidden). A command wakes it at once, so "show" draws sooner than
before. With --watch-stdin, its stdin closing wakes it as well, so stopping it doesn't wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pointer: ft-screens announces new panels to the helper

The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel
made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard
the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens
now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds
it to its list at once (only frametop.* keys). An older helper ignores it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hands: fix Export doing nothing, and show it's busy at once

af2ea7c put _stay_awake between exportSession and its @Slot, so the
window's Export button called a method QML couldn't see. A new test checks
every backend call in main.qml against Backend's slots and properties.
Export and Upload now say Exporting…/Uploading… with a spinner the moment
they're pressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Lazy susan: Meta+Alt+Tab spins the panels around you

- ft-screens "spin next|prev|<degrees>": every unpinned screen and
  floating window turns together about a vertical axis through your
  head (0.3 s, eased), so the next panel on the right or left comes to
  straight ahead; the arrangement stays as it is. Taps during a spin
  add to it, from where the panels are headed; grabbing a panel or
  placing it (ft-layout, ft-floatd) takes it out of the spin
- when a spin settles, the panel in front gets the pointer (recenter),
  typing (as after a click), and KWin's active window: its floating
  window, or the top window on a screen (ft-floatd "front N", the KWin
  script's activate-output). KWin's outputs follow the screens'
  new places (ft-layout scale), as after a move
- the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and
  Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them.
  Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote
  desktop like RustDesk, and the relay would take the Mac's app
  switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS,
  Windows, and KDE

Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and
three or four floating windows, through RustDesk to a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: login command works from Frametop's Konsole

Frametop's Konsole sets XDG_RUNTIME_DIR=/run/user/UID/frametop, where podman finds
no container state, so 'distrobox enter dev -- hf auth login' failed with a crun
error. The command the Upload page shows now sets the real runtime folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: log in from the Upload page, three-step page, no terminal

The Upload page is now three numbered steps: choose the export, log in to
Hugging Face, upload. Log in runs hub.py login, huggingface_hub's browser
login (OAuth device code, as hf auth login does): the link opens in the
browser and the page shows the code to enter, with Copy code and Cancel.
hub.py saves the token; the window never sees one, and nobody pastes one.

The terminal upload and the login command are gone from the page, and
UPLOAD.md is now a short 'About uploading' under the steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: take.json keeps camera clock samples

sets.bin's capture_ns is CLOCK_MONOTONIC_RAW; poses.jsonl and prompts.jsonl are
CLOCK_MONOTONIC. On 2026-10-03 the two were 0.80 s apart during a session and
1.11 s apart five hours later, so images can't be paired with poses by
capture_ns. take.json now samples RAW minus MONOTONIC as each recording part
starts and stops (as ft-hands' raw_minus_mono_ns), so readers can put each
exposure on the poses' clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder export: no controller poses without controllers, nothing in deleted ranges

When the checklist says no controllers, exported poses.jsonl has left and right
null and feedback lines carry no controller state: controllers left switched on
still get tracked (one wandered 2 m in a real session) and would read as the
hands' ground truth. Poses and live-tracker feedback inside deleted ranges are
left out too, as the images there are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* hand recorder: final consent text (2026-10-03), residency check, installer

Consent 2026-10-03, after a non-lawyer review: who runs this and how to reach
them, the dataset is public (Hugging Face, possibly abroad), the Hugging Face
username shows next to the contributor id, purposes (no identification), safety,
the maintainer grant passes to whoever maintains Frametop next, withdrawal
before and after merge, rights such as the GDPR's, and what a new version means.
Residents of Illinois, Texas and Washington can't take part for now (biometric
privacy laws): a third checkbox, profile consent.region_ok, checked by
validate.py from this consent version on. The DRAFT banners are gone, so uploads
no longer need FT_HANDREC_ALLOW_UPLOAD.

hands/rec/install.sh installs the recorder on a Frame with Frametop: container
packages, hand tracking and panel builds, ft-camd's capabilities, menu entry.

test_qml_backend also checks each call's argument count against the slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: a reset button next to the grab bar, clickable in VR games

Each desktop screen gets a reset button left of its bar (a reticle). It
puts every screen back in its layout around where you are now, like
Meta+Shift+R (ft-layout apply).

In a VR game the screens leave the controllers to the game (the
outside_games and dashboard modes), so a controller couldn't click any
of their controls. Aiming a hand controller at the reset button now sets
MakeOverlaysInteractiveIfVisible on that button's overlay alone, so the
trigger clicks it; the flag clears half a second after the aim leaves a
zone twice as wide, and the game gets the controllers back. The aim
comes from the laser poses ft-screens already reads to show the controls.

The ft-layout spawn is now RunLayout(cmd), shared with the arrange.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Click stability: 32 logical pixels by default, not 8

8 is about 0.2 degrees on a 3.4 m wide 3440-pixel screen 2 m away, so a
trigger press turned into a drag unless the hand was very still. 32
(about 0.9 degrees) felt much better in the headset (2026-10-03).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: in games, pointing a controller at a panel turns its laser on

SteamVR's own floating windows take the laser while a controller points
at them in a game and give it back when it points away. Frametop's
panels didn't: with the controllers left to the game (outside_games, the
default, or dashboard), they couldn't be clicked without the dashboard.

ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or
floating window while a hand controller's laser pose meets it, its
controls, or its popups (UpdateAim; curved screens hit on their
cylinder), and clears it 0.3 s after the aim leaves a wider margin. A
drag or a held button keeps it on. The keyboard, one overlay, uses
ComputeOverlayIntersection and now follows the mode when a game starts
or ends while it's open. This replaces the reset button's own aim zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: find the controllers' laser tip during VR games too

GetComponentStateForDevicePath with no input source handle fails for
every render model component while a VR game runs (checked 2026-10-03
with a game up: all 21 components of frame_controller_right). TipOffset
then fell back to the controller's pose, which aims 40 degrees above the
Frame controller's laser. In games, pointing at a screen's middle missed
it and pointing below it hit, so the new aim-to-laser only worked from
the bottom; the controls' reveal and pin/roll aim were off the same way.
GetComponentState still answers then, with the same tip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* input-relay: Add mute key as volume key

Add KEY_MUTE as volume key. It will be mapped to KEY_MACRO28 and
use wpctl to toggle the mute of the default audio sink. The toggle of
the mute state will be done once when the key is pressed instead of
continuously toggling it when it is held down.
This allows the mute button on keyboards to work properly.

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>

* Session: bring back a taskbar saved on a screen the desktop doesn't have

Plasma 6.2.5 keeps a panel on a screen number and never moves one whose
number is past the screen count, so a taskbar saved on a spare output
(#18, lastScreen=8 with three screens) or on a screen a smaller layout
dropped stayed hidden. Before Plasma starts, session/fix-panels.py moves
such a panel and its tray's containment to screen 0 (the primary),
keeping its widgets, unless screen 0 already has a panel on that edge.
doctor.sh checks the panels' screens, and report.sh lists them with the
live outputs and panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* keys-test: the spin bindings (Meta+Alt+Tab, Meta+Alt+Shift+Tab), not while paused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Menu entries: own programs for Reset Screen Layout and Hide/Show Screens

Reset Screen Layout and Hide/Show Screens both ran ft-layout. Steam lists
entries by program, so Hide/Show launched Reset. Each gets a wrapper.

From PR #17 (only this part of 9618be8; its host_command change is for the
Nix packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Screens: release a held button that can't come up on a screen

Pausing, or hiding the screen a button went down on, took the laser off
it mid-click; the pause gesture's second thumbstick click does that.
SteamVR's laser mouse then forgot the button ("Mouse down count is 1 but
states are all false"), no release came, and the catcher kept showing
whenever the pressing laser was off the panels, even while paused. Being
interactive, it kept the VR game's controllers from it until the
desktop restarted (2026-10-04, Beat Saber).

ft-screens now releases a held button when it's paused, when the screen
it went down on is hidden, or, during VR games, when the pressing hand
controller has held nothing for a second. GetControllerState answers
overlay apps only while a game runs; outside games a hold is never cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <codex@localhost>
Co-authored-by: CuriousJ <curious.j.tuber@gmail.com>
Co-authored-by: Patrick McDavid <fusionjunky@gmail.com>
Co-authored-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
Co-authored-by: John Murray <5672686+JRMurr@users.noreply.github.com>
2026-10-04 19:44:30 -06:00
JakeGreen2145 5e3323c188 [verified] merge experimental into nested AT-SPI fix 2026-10-04 17:38:33 -04:00
JakeGreen2145 25504e0ed4 [verified] fix(session): start nested AT-SPI registry 2026-10-04 16:13:03 -04:00
DeeJanuzandClaude Opus 5.5 ec870d2d7a Merge branch laser-release (release a held button that can't come up on a screen: the game kept losing its controllers) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:37:46 -06:00
DeeJanuzandClaude Opus 5.5 80797c98b9 Screens: release a held button that can't come up on a screen
Pausing, or hiding the screen a button went down on, took the laser off
it mid-click; the pause gesture's second thumbstick click does that.
SteamVR's laser mouse then forgot the button ("Mouse down count is 1 but
states are all false"), no release came, and the catcher kept showing
whenever the pressing laser was off the panels, even while paused. Being
interactive, it kept the VR game's controllers from it until the
desktop restarted (2026-10-04, Beat Saber).

ft-screens now releases a held button when it's paused, when the screen
it went down on is hidden, or, during VR games, when the pressing hand
controller has held nothing for a second. GetControllerState answers
overlay apps only while a game runs; outside games a hold is never cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:31:27 -06:00
DeeJanuzandClaude Opus 5.5 2fadbbe148 Merge branch orphan-panel (bring back a taskbar saved on a screen the desktop doesn't have, #18) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:42:12 -06:00
DeeJanuzandClaude Opus 5.5 5ee07f99a8 Merge branch menu-entry-programs (Hide/Show Screens no longer launches Reset Screen Layout) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:39:44 -06:00
John MurrayandClaude Opus 5.5 f0b93b79d3 Menu entries: own programs for Reset Screen Layout and Hide/Show Screens
Reset Screen Layout and Hide/Show Screens both ran ft-layout. Steam lists
entries by program, so Hide/Show launched Reset. Each gets a wrapper.

From PR #17 (only this part of 9618be8; its host_command change is for the
Nix packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:39:40 -06:00
DeeJanuzandClaude Opus 5.5 13a530d829 Merge branch mute-key (PR #24: the mute key toggles the default output's mute) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:44 -06:00
DeeJanuzandClaude Opus 5.5 f260a6a9e4 Merge branch lazy-susan (PR #22: Meta+Alt+Tab spins the panels around you) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:44 -06:00
DeeJanuzandClaude Opus 5.5 2e09cf9efe Merge PR #24 (SuperTuxii: the mute key toggles the default output's mute) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 9a4b66ff50 keys-test: the spin bindings (Meta+Alt+Tab, Meta+Alt+Shift+Tab), not while paused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 17269134ce Merge PR #22 (ehippy: lazy susan, Meta+Alt+Tab spins the panels around you) into experimental
Conflicts with experimental's pause_toggle, steam_menu and command: actions and its
AnnounceOverlay: both kept. The spin bindings join the Meta tap in the defaults, spinning
doesn't need pointer mode, and like other actions it does nothing while Frametop is paused.
AnnounceOverlay now sends through the PR's SendPointer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:36:33 -06:00
DeeJanuzandClaude Opus 5.5 d995b815f8 Session: bring back a taskbar saved on a screen the desktop doesn't have
Plasma 6.2.5 keeps a panel on a screen number and never moves one whose
number is past the screen count, so a taskbar saved on a spare output
(#18, lastScreen=8 with three screens) or on a screen a smaller layout
dropped stayed hidden. Before Plasma starts, session/fix-panels.py moves
such a panel and its tray's containment to screen 0 (the primary),
keeping its widgets, unless screen 0 already has a panel on that edge.
doctor.sh checks the panels' screens, and report.sh lists them with the
live outputs and panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:32:04 -06:00
SuperTuxii f18918f781 input-relay: Add mute key as volume key
Add KEY_MUTE as volume key. It will be mapped to KEY_MACRO28 and
use wpctl to toggle the mute of the default audio sink. The toggle of
the mute state will be done once when the key is pressed instead of
continuously toggling it when it is held down.
This allows the mute button on keyboards to work properly.

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
2026-10-04 16:06:37 +02:00
DeeJanuzandClaude Opus 5.5 e0208687b6 Merge branch tip-in-games (controller laser tip found during VR games) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:33:49 -06:00
DeeJanuzandClaude Opus 5.5 c699e13104 Screens: find the controllers' laser tip during VR games too
GetComponentStateForDevicePath with no input source handle fails for
every render model component while a VR game runs (checked 2026-10-03
with a game up: all 21 components of frame_controller_right). TipOffset
then fell back to the controller's pose, which aims 40 degrees above the
Frame controller's laser. In games, pointing at a screen's middle missed
it and pointing below it hit, so the new aim-to-laser only worked from
the bottom; the controls' reveal and pin/roll aim were off the same way.
GetComponentState still answers then, with the same tip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:33:49 -06:00
DeeJanuzandClaude Opus 5.5 9b12b7d74e Merge branch aim-lasers (in games, pointing a controller at a Frametop panel turns its laser on) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:27:49 -06:00
DeeJanuzandClaude Opus 5.5 c7c7c1f772 Screens: in games, pointing a controller at a panel turns its laser on
SteamVR's own floating windows take the laser while a controller points
at them in a game and give it back when it points away. Frametop's
panels didn't: with the controllers left to the game (outside_games, the
default, or dashboard), they couldn't be clicked without the dashboard.

ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or
floating window while a hand controller's laser pose meets it, its
controls, or its popups (UpdateAim; curved screens hit on their
cylinder), and clears it 0.3 s after the aim leaves a wider margin. A
drag or a held button keeps it on. The keyboard, one overlay, uses
ComputeOverlayIntersection and now follows the mode when a game starts
or ends while it's open. This replaces the reset button's own aim zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:27:49 -06:00
DeeJanuzandClaude Opus 5.5 2486a601e3 Merge branch click-threshold (controller click zone 32 px by default) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:49 -06:00
Patrick McDavidandClaude Opus 5.5 6fb168a8b8 Lazy susan: Meta+Alt+Tab spins the panels around you
- ft-screens "spin next|prev|<degrees>": every unpinned screen and
  floating window turns together about a vertical axis through your
  head (0.3 s, eased), so the next panel on the right or left comes to
  straight ahead; the arrangement stays as it is. Taps during a spin
  add to it, from where the panels are headed; grabbing a panel or
  placing it (ft-layout, ft-floatd) takes it out of the spin
- when a spin settles, the panel in front gets the pointer (recenter),
  typing (as after a click), and KWin's active window: its floating
  window, or the top window on a screen (ft-floatd "front N", the KWin
  script's activate-output). KWin's outputs follow the screens'
  new places (ft-layout scale), as after a move
- the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and
  Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them.
  Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote
  desktop like RustDesk, and the relay would take the Mac's app
  switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS,
  Windows, and KDE

Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and
three or four floating windows, through RustDesk to a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 10:17:22 -06:00
Patrick McDavidandClaude Opus 5.5 1ebf3692fb ft-floatd: a launch for a missing app no longer kills the control socket (#16)
Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.

Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:12:49 -06:00
DeeJanuzandClaude Opus 5.5 072a294941 README: Frametop doesn't work on the SteamOS beta yet
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 16:12:52 -06:00
DeeJanuzandClaude Opus 5.5 7816633353 README: link the Frametop Discord
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 21:18:09 -06:00
51 changed files with 3024 additions and 142 deletions

No files matched your search

+1 -1
View File
@@ -26,7 +26,7 @@ A Steam Frame is someone's personal headset, and they may be wearing it while yo
- Don't kill or restart `gamescope`, `steam`, `vrserver`, `vrcompositor`, the gamescope session, or the Frametop desktop without asking. Each one ends or disrupts whatever is happening in VR.
- Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Three installers need host `sudo`, and they ask for it: the Bluetooth fixes (`setup/bluetooth/install.sh`), hand tracking (`hands/run.sh install` and `caps`, which set ft-camd's file capabilities with `setcap`), and our own eye tracker's frame grabber (`gaze/tracker/install.sh`).
- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`).
- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`). When an installer starts writing something new outside the repo, add it to `uninstall.sh` too: users uninstall with that script, not with each installer's `uninstall`.
- Never copy `.netrc`, SSH keys, or Steam config off the Frame or into this repo.
## SteamOS updates
+15 -14
View File
@@ -57,6 +57,7 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
| While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off |
| Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
| Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
| Meta+Alt+Tab, or Meta+Alt+Shift+Tab | Spins every screen and floating window around you together, like a lazy susan, so the next one on your right (or left) glides to straight ahead, with the pointer and typing going to it. Their arrangement stays the same: the room turns instead of you. Tap again to keep going; Meta+Shift+R puts the screens back. Pinned screens stay where they are |
| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
| Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
@@ -170,23 +171,22 @@ Or by hand: `cd ~/frametop && git pull && ./install.sh`.
## Uninstall
In a terminal on the headset, run:
```
./desktops.sh uninstall # the launcher's Desktop entry goes back to the stock desktop
./desktops.sh relay uninstall
pointer/helper/run.sh uninstall
power/run.sh uninstall
pointer/driver/install.sh uninstall # then restart SteamVR
input-settings/install.sh uninstall
display-settings/install.sh uninstall
remote/install.sh uninstall
setup/bluetooth/install.sh uninstall # if you installed the Bluetooth fixes
hands/run.sh uninstall # if you installed hand tracking by hand
gaze/run.sh uninstall # if you installed the gaze service
gaze/tracker/install.sh uninstall # if you installed our own eye tracker's frame grabber
gaze/probe/install.sh uninstall # if you installed the gaze probe
curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
```
Your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, and the gaze calibration in `~/.local/state/frametop/gaze`. So does the desktop's own Plasma setup, in `~/.config/frametop`. Delete them too for a clean slate.
It works in two steps, so it never takes away the keyboard, mouse, or desktop you're using while it runs:
1. It stops Frametop from starting. Launch a program → Desktop opens the stock desktop again, and Frametop's services, its SteamVR driver, its menu entries, and the system files of our eye tracker and the Bluetooth fixes are removed (those need your `sudo` password). Everything running now keeps running until you restart the headset, and it offers to restart it for you.
2. After the restart, run the same command again. It deletes the code in `~/frametop`, and asks whether to delete your settings, any eye or hand recordings, and the build container (1–2 GB) too.
To see what it would do without changing anything, add `-s -- --dry-run` after `bash`. If the code isn't in `~/frametop`, add `-s -- --dir <folder>`. From the repo, the same script is `./uninstall.sh`.
Don't delete `~/frametop` by hand before you uninstall and restart: the desktop and the input relay run from it, and without it Launch a program → Desktop no longer opens anything. If you've already deleted it, the command above still works, since it doesn't need the repo.
Unless you ask for them to go, your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, the gaze calibration in `~/.local/state/frametop`, and the desktop's own Plasma setup in `~/.config/frametop`. A later install picks them up again.
## How it works
@@ -196,6 +196,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
| --- | --- |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
| `install.sh` | The one-step installer. Safe to re-run. |
| `uninstall.sh` | The uninstaller: run it, restart the headset, and run it again. It doesn't need the rest of the repo. |
| `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
| `screens/` | ft-screens, the compositor (wlroots and OpenVR). |
| `session/` | The desktop session script and its config example. |
+1 -1
View File
@@ -3,7 +3,7 @@ Type=Application
Name=Reset Screen Layout
GenericName=Put the VR desktop's screens back in their layout
Comment=Float the screens and arrange them in the layout from Frametop Display Settings
Exec=@REPO@/layout/ft-layout apply
Exec=@REPO@/layout/ft-layout-reset
Icon=view-restore
Categories=Settings;
Keywords=display;screen;layout;arrange;reset;steamvr;frametop;
+1 -1
View File
@@ -3,7 +3,7 @@ Type=Application
Name=Hide/Show Screens
GenericName=Hide or show the VR desktop's screens
Comment=Hide the screens (and SteamVR's laser) for a VR game; press again to bring them back
Exec=@REPO@/layout/ft-layout toggle
Exec=@REPO@/layout/ft-hide-show
Icon=view-visible
Categories=Settings;
Keywords=display;screen;hide;show;steamvr;frametop;
+18 -5
View File
@@ -38,7 +38,7 @@ The curved layout chains screens edge to edge, like monitors on a desk: the midd
A resize handle has to be able to shrink a screen from any direction, so the dragged corner follows the laser along the screen's diagonal rather than taking the larger of its horizontal and vertical reach. Pushing and pulling a carried screen moves it along the line from your head, because the 3D mouse's virtual controller sits just in front of the bar, below the screen's centre, so the line from the device points mostly upward.
Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose.
Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose. ft-screens reads the tip with `GetComponentState`: `GetComponentStateForDevicePath` without an input source handle fails for every component while a VR game runs, so in games the rays came from the pose, 40° too high.
`ComputeOverlayIntersection` ignores `SetOverlayIntersectionMask`, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner.
@@ -62,7 +62,7 @@ A profile's screen part is the custom arrangement under a name: each screen's po
`IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting.
The reset button needs to work in a game, where the screens have the flag off. So ft-screens turns the flag on for that button's overlay alone while a hand controller aims within about one button's width of it, and off half a second after the aim leaves a zone twice as wide. ft-screens finds the aim from the controllers' laser poses, which it reads anyway to show the controls, so it doesn't need SteamVR's laser to be on first. The game loses the controllers only while you aim at the button.
In a game, Frametop's panels work like SteamVR's own floating windows: point a controller at one and its laser comes on, point away and the game has the controllers again. ft-screens turns the flag on for a panel while a hand controller's laser pose meets it, its controls, or a floating window's popups. It finds that from the poses it already reads to show the controls, so SteamVR's laser doesn't have to be on first. Leaving takes a margin two control-sizes wide and 0.3 s, a drag or a held button keeps the flag on, and the keyboard, a single overlay, uses SteamVR's `ComputeOverlayIntersection`. The 3D mouse doesn't need any of this: it has its own laser mode.
## Floating windows
@@ -133,7 +133,7 @@ A few overlays need special handling:
Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a right click or MetLine truncated
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: the session now starts an AT-SPI registry, but apps still need to expose useful accessibility trees (and may need restarting), and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a rightLine truncated
Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
@@ -169,10 +169,22 @@ The Frame controllers can be mapped like mouse buttons, but they aren't input de
## The desktop session
The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly.
The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly. Its own config folder also hides SteamVR's path registry (`~/.config/openvr/openvrpaths.vrpath`) from everything started in it: OpenVR programs there fail with `VRInitError_Init_PathRegistryNotFound`, and `vrpathreg adddriver` writes a new registry under `~/.config/frametop/openvr` that has no SteamVR in it and that SteamVR never reads. So Frametop's scripts run SteamVR's tools with `XDG_CONFIG_HOME=~/.config`.
The VR launcher starts the session from the Steam client, and the client's environment came along: `LD_LIBRARY_PATH` pointing at Steam's own runtime, whose `libavcodec` has no H.264 decoder, so VLC in the desktop couldn't play most videos, plus the client's overlay and launch settings. The session script drops the client's variables before it starts anything. SteamOS's global Mesa settings (`/usr/share/deckard/mesavars.sh`) stay, and the gamescope session's Vulkan layer (`ENABLE_GAMESCOPE_WSI`) is only kept for the gamescope backend.
### Nested accessibility
The session drops an inherited `AT_SPI_BUS_ADDRESS`, so apps cannot accidentally use the host desktop's registry. It autostarts `session/ft-atspi` in Plasma phase 2, after KWin has set the nested display environment. The helper gets the live accessibility address from `org.a11y.Bus` on the private session bus, preserves any existing registry owner, updates the accessibility bus's activation environment, and tries `StartServiceByName` first.
On SteamOS 0.3.0 with at-spi2-core 2.52.0, the native launcher can choose dbus-broker because its process belongs to a systemd user unit. Registry activation then fails: this desktop's private session bus does not have a systemd activation manager. In that case the helper starts only `at-spi2-registryd` on the already-existing accessibility bus. The registry refuses duplicate ownership. Unlike native activation's `--use-gnome-session`, the fallback does not try to register with GNOME's session manager; that flag did not explain the observed native activation failure.
The fallback registry does not exit merely when its bus disconnects in the isolated SteamOS test. Its small watcher checks both private buses every 5 seconds, and terminates and reaps only the child it started when either bus disappears or the watcher is stopped. Each check runs `gdbus` twice; once a second, that cost about 1% of a core. `keep-apps.sh` keeps the watcher in the desktop unit when `desktops.sh start` runs the desktop as `frametop-desktop`. Started from the VR launcher, the desktop runs in steam.service, which doesn't stop with it, so there the watcher is the only thing that stops the registry. There is no second accessibility bus, global systemd environment update, process-name kill, or host registry replacement. Missing accessibility files or bus errors are nonfatal; the desktop still starts. Toolkit-specific accessibility opt-ins and pointer snapping are separate work.
Run the isolated checks on the host with `/usr/bin/python3 session/test/test_accessibility.py`. They use private D-Bus buses, Xvfb and a GTK3 app, never the production display or input. Native activation uses a small `org.a11y.Bus` test provider pointing to a real private dbus-daemon with the installed registry service; the SteamOS fallback uses the installed bus launcher and broker. The tests check real app-tree discovery, existing owners, concurrent starts, session stop/restart, and teardown. They require test-only PyGObject (Gio and GTK3), Xvfb, and at-spi2-core; the runtime helper uses Python's standard library and the existing host `gdbus`. Actual Plasma autostart and VR desktop restart still require an approved hardware test.
### Other session behavior
Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.
Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens Discover instead of launching them, so the session sources `/etc/profile.d/flatpak.sh`.
@@ -185,6 +197,8 @@ KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompos
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
Plasma 6.2.5 keeps each panel on a screen number (`lastScreen` in `plasma-org.kde.plasma.desktop-appletsrc`), and the numbers rank the enabled outputs by priority, so 0 is the primary screen. A panel whose number is past the screen count gets no view, and Plasma never moves it: the remap it runs at every start only moves a panel whose number has no desktop, and this desktop keeps a desktop for every output it has seen, spares included. So the taskbar was lost when the number of screens went down, and once it was found saved on a spare output, number 8 of a desktop with three screens ([#18](https://github.com/DeeJanuz/frametop/issues/18)). Before Plasma starts, the session runs `session/fix-panels.py`, which moves any panel numbered past the screen count, with its system tray's containment, to screen 0, keeping its widgets and settings. A panel stays put when screen 0 already has one on that edge, and comes back by itself if the screens do. The file is backed up to `<file>.ft-bak` first. Plasma's scripting can't do this while it runs (`panel.screen` is read-only in 6.2.5), so a lost taskbar comes back at the desktop's next start. `scripts/doctor.sh` and `scripts/report.sh` list the panels and their screens.
Remote desktop is a chain (krdp, then FreeRDP inside Xvnc) because nothing on SteamOS serves KWin over VNC directly. Kept connected all the time, it cost about a core with nobody watching: krdpserver 55 to 78% (it encodes H.264 in software with openh264: VA-API finds no driver for the Frame's GPU in the container), FreeRDP 16 to 27%, Xvnc 6 to 11%, and the bridge's layout check every 5 seconds another 4%. krdp creates its screencast session per RDP connection and drops it when the connection closes (`SessionController::onNewConnection` in krdp 6.7), so an idle krdpserver costs nothing and can stay up; only the RDP connection has to go. The bridge connects FreeRDP when a VNC client appears and disconnects 45 seconds after the last one leaves. Xvnc has no hook for its clients, so the bridge counts established connections to its port with `ss`, woken early by Xvnc's log output; looking with `ss` once a second cost about 0.9% of a core in bash, against about 0.1% this way. `Xvnc -inetd` from a systemd socket would start a server per connection and lose sharing between viewers. The layout check (`ft-layout remote-view`, which scans `/proc` for plasmashell and runs `kscreen-doctor -j`) now runs only while FreeRDP runs, and then only after `kwinoutputconfig.json` or `frametop-layout.json` changes, with one check a minute in case a change touched neither.
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
@@ -224,6 +238,5 @@ On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-stea
- A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens.
- Drawing KWin's cursor on the screens.
- Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back.
- Frame pacing and GPU cost with several busy screens haven't been measured.
- Real standby on a stand, with rendering and tracking paused, not just the backlight off. SteamVR has no call for it, and its activity level follows the proximity sensor.
+6 -4
View File
@@ -18,6 +18,8 @@ desktops.sh start | stop | restart | status | log [lines]
When the VR launcher starts the desktop, it inherits the Steam client's environment. The session script drops the client's runtime from it (`LD_LIBRARY_PATH`, the `STEAM_*` settings, and the Steam overlay's Vulkan layer), so apps in the desktop use the system's libraries, including its video codecs, just as they would after a normal login.
The nested session also starts an AT-SPI accessibility registry through `session/ft-atspi` in Plasma's autostart. It discovers the bus from this session, ignores an inherited host accessibility address, and leaves an existing registry alone. Accessibility errors do not stop the desktop. This supplies the registry infrastructure for apps that expose AT-SPI trees; it does not enable gaze snapping or force Chromium/Electron accessibility. After an approved desktop restart, an AT-SPI-aware app should be visible on the nested bus. See [design.md](design.md#nested-accessibility) for native activation, fallback lifecycle, and the isolated test command.
KWin's blur and background contrast effects and its animations are off in this desktop, because KWin draws on the headset's GPU, which SteamVR needs. The session script turns them off once, the first time it starts (it leaves a setting you already have alone, and marks it done in `~/.config/frametop/frametoprc`), so turning them back on sticks. In the Frametop desktop, System Settings → Window Management → Desktop Effects has Blur and Background Contrast, and General Behavior has Animation speed. Or from a terminal, then restart the desktop:
```
@@ -44,7 +46,7 @@ Every screen is an overlay named `frametop.screen.N` with five controls:
- `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
- `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
- `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.
- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`). In a VR game, where the screens leave the controllers to the game, aiming a controller at it turns SteamVR's laser on for that button alone, so the trigger clicks it; the game gets the controllers back half a second after you aim away.
- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`).
The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.
@@ -62,7 +64,7 @@ The Visibility & pins tab of Frametop Display Settings decides when the screens
In the last three modes the hotkey shows the screens anyway. A screen can also be hidden on its own (Screens shown on the same tab, or `ft-layout hide N`): it stays hidden whatever the mode or the hotkey says, until it's shown again there. Windows on it stay put, and a new window that would open on it floats instead (ft-floatd). Profiles use this to show only some screens. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:
- During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. Pointing a controller at a screen, a floating window, or the keyboard still turns its laser on, like SteamVR's own floating windows, and pointing away gives the game the controllers back. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.
@@ -134,7 +136,7 @@ A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|<degrees>`); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
@@ -261,7 +263,7 @@ Your hands show over the screens: where a tracked hand is between an eye and a s
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`.
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
- The install leaves both off, and they don't start with SteamVR. `ft-handsctl on` starts them while SteamVR runs, and `ft-handsctl off` stops them; they also stop with SteamVR. The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
+11
View File
@@ -378,6 +378,17 @@ function run(c) {
case "activate":
if (w) workspace.activeWindow = w;
break;
case "activate-output": { // the top window on that output (a spin brought it to the front)
const order = workspace.stackingOrder;
for (let i = order.length - 1; i >= 0; --i) {
const o = order[i];
if (o.deleted || o.minimized || o.hidden || !o.managed || !o.output) continue;
if (o.output.name !== c.output || !o.normalWindow || o.popupWindow) continue;
workspace.activeWindow = o;
break;
}
break;
}
case "minimize":
if (w) w.minimized = c.on;
break;
+9
View File
@@ -18,6 +18,8 @@ command-line side). Replies go to the sender:
("float pointer" is the float key: the window under the pointer, else the active one,
floated or docked; the input relay sends it for float_toggle, and "dock all" for dock_all)
dock N | close N | resize N W H | scale N STEPS (ft-screens, N = its screen)
front N (ft-screens: a spin brought panel N to the front: make its window, or the top one
on a screen, KWin's active window)
Spare outputs are WL-<screens> .. WL-<screens + slots - 1>. A floating window's output is its
frame plus a margin on each side (FLOAT_MARGIN pixels), so menus have room; the panel shows
@@ -1063,6 +1065,13 @@ class Daemon:
for f in list(self.floats.values()):
self.dock(f)
return "ok"
if cmd == "front" and len(rest) == 1 and rest[0].isdigit():
f = self.by_panel(int(rest[0]))
if f:
self.command(cmd="activate", id=f.id)
else: # a screen: its output is WL-<N - 1>
self.command(cmd="activate-output", output=f"WL-{int(rest[0]) - 1}")
return "ok"
if cmd in ("dock", "close", "resize", "scale") and rest and rest[0].isdigit():
f = self.by_panel(int(rest[0]))
if not f:
+1 -1
View File
@@ -33,7 +33,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
+35 -5
View File
@@ -48,6 +48,13 @@ click with nothing taken after ACCEPT_WAIT, and a failed calibration names its m
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
renewed every second; the helper shows it again by itself when that stops).
Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (it maps pupils
to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs
anyway ("blind"): someone in the headset (SteamVR's tracker sees an eye) and ft-eyes answering
are enough to start it, each dot stands in for the gaze, and a click takes the CHECK_WINDOW up
to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point)
and says why not. Without this, a fresh install could never calibrate our tracker.
What a capture teaches:
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
calib-start, a calib-point for each dot, calib-fit (its calibration)
@@ -336,9 +343,17 @@ class Checks:
return time.monotonic() - self.seen_at < within
def can_run(self):
"""Someone's in the headset and the tracker is sending."""
"""Someone's in the headset and the tracker is sending. Our tracker sends no gaze before
its first calibration: then ft-eyes answering (calibrated() is False only once it has)
is enough, since the calibration is what it needs (see "first calibration" at the top)."""
if self.blind():
return self.eyes_seen()
return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
def blind(self):
"""Our tracker is in use, running, and not calibrated yet: it has no gaze to send."""
return self.svc.kind == "own" and self.calibrated() is False
def on_gaze_on(self):
self.full_armed = True
self.need_full("gaze mode came on without a calibration")
@@ -356,7 +371,7 @@ class Checks:
if not self.can_run() and self.svc.waking():
return # the tracker is still starting (the service idled)
if not self.can_run():
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 and not self.blind()
else "no eyes seen (is the headset on?)")
elif now < self.full_retry_at:
return
@@ -408,6 +423,9 @@ class Checks:
if not self.can_run():
return "error the headset is off or the tracker isn't sending"
own = svc.kind == "own"
blind = self.blind()
if blind and kind != "full":
return "error our tracker isn't calibrated yet: use Calibrate"
if kind == "full" and own:
reply = ask(EYES, "calib-start", 3.0)
if not reply.startswith("ok"):
@@ -416,8 +434,10 @@ class Checks:
now = time.monotonic()
self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
"started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
log(f"{kind} check: {reason}")
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": "",
"blind": blind}
log(f"{kind} check: {reason}" + (" (our tracker's first: no gaze yet, so each click takes the look "
"up to it)" if blind else ""))
if kind == "full":
st = ask(SCREENS, "state", 0.5).split()
if len(st) >= 3 and st[0] == "ok":
@@ -519,6 +539,12 @@ class Checks:
return
if c["own"]:
src = s["src"].get("own") or {}
if c["blind"]:
# Our tracker's first calibration: no gaze yet, so the dot stands in for it (the
# gaze can't seem to move) and a click takes the look up to it. ft-eyes checks
# the pupils held still (see the top).
yaw, pitch, _ = c["dots"][c["i"]]
src = {"hy": yaw, "hp": pitch}
else:
src = s["src"].get("mmap1") or {}
if "hy" not in src:
@@ -868,7 +894,11 @@ class Checks:
self.back_since = None # gone again before DON_DELAY
elif self.back_since is not None and now - self.back_since >= DON_DELAY:
self.away, self.back_since = False, None
self.full_armed = True # a calibration that closed unfinished opens again
if not self.check:
# A calibration that closed unfinished opens again. Not one still open: gaze mode
# coming on wakes our tracker, so its eyes come back just as the calibration
# opens, and re-arming then opened a second one when the first ended.
self.full_armed = True
self.auto_quick("the headset went on")
if svc.kind == "own" and now - svc.own_at < 5:
reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""Offline test of our own eye tracker's first calibration (gaze/gazecheck.py, "blind"): before
it has a calibration, ft-eyes publishes no gaze, and the calibration must still open and take
its dots. Until 2026-10-05 it couldn't: a fresh install that chose our tracker never calibrated.
Runs ft-gazed's Service with its sockets renamed and HOME in a temp folder (state and settings
go there), a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR sees both
eyes; "own" is {"ok":0}, as with an uncalibrated ft-eyes), a fake ft-eyes control socket, and no
panel (a stand-in process). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
or SteamVR, so it's safe next to them.
gaze/test/first-calibration-test.py
"""
import os
import tempfile
HOME = tempfile.mkdtemp(prefix="ft-gaze-first-cal-test-")
os.environ["HOME"] = HOME # before gazecal: its STATE, and frametop.conf, follow HOME
import importlib.machinery # noqa: E402
import importlib.util # noqa: E402
import json # noqa: E402
import selectors # noqa: E402
import shutil # noqa: E402
import socket # noqa: E402
import subprocess # noqa: E402
import sys # noqa: E402
import threading # noqa: E402
import time # noqa: E402
HERE = os.path.dirname(os.path.abspath(__file__))
GAZE = os.path.join(HERE, "..")
sys.path.insert(0, GAZE)
loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
loader.exec_module(gazed)
import gazecheck # noqa: E402 (the module ft-gazed imported)
tag = f"ft_gaze_first_cal_test_{os.getpid()}"
gazed.ME = f"\0{tag}_gazed"
gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL = f"\0{tag}_panel"
gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes"
gazed.read_settings = lambda: ("own", "auto", "auto", 55.0)
DOTS = 3
real_dots = gazecheck.check_dots
gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS] # a short calibration
logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg)
# The fake ft-gaze: 90 samples a second, SteamVR sees both eyes, no gaze from ours.
FAKE = os.path.join(HOME, "ft-gaze")
with open(FAKE, "w") as f:
f.write('''import json, os, select, sys, time
while True:
if select.select([sys.stdin], [], [], 1 / 90)[0]:
if not os.read(0, 4096):
break
print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
"open": [0.8, 0.8]}, "own": {"ok": 0}}}), flush=True)
''')
SLEEPER = [sys.executable, "-c", "import sys; sys.stdin.read()"] # quits when its stdin closes
def start_helper(self):
"""ft-gaze, straight from here instead of the dev container."""
self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
self.proc_sources = self.wanted_sources()
os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
self.buf = b""
def start_eyes(self):
"""ft-eyes' process: a stand-in. Its control socket is the fake below."""
self.eyes_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
os.set_blocking(self.eyes_proc.stderr.fileno(), False)
self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
def start_panel(self):
self.panel_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
os.set_blocking(self.panel_proc.stderr.fileno(), False)
self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
gazed.Service.start_helper = start_helper
gazed.Service.start_eyes = start_eyes
gazecheck.Checks.start_panel = start_panel
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that.
eyes_state = {"cal": None, "points": [], "fail": None}
eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
eyes.bind(gazed.EYES_SOCKET)
eyes.settimeout(0.2)
def eyes_answer():
while True:
try:
data, addr = eyes.recvfrom(512)
except socket.timeout:
continue
except OSError:
return
w = data.decode().split()
if w[0] == "status":
reply = json.dumps({"calibration": eyes_state["cal"], "calibrating": False, "dots": 0,
"eyes": {"right": {"reseat": False}, "left": {"reseat": False}}})
elif w[0] == "calib-start":
eyes_state["points"] = []
reply = "ok"
elif w[0] == "calib-point":
eyes_state["points"].append(tuple(map(float, w[1:5])))
reply, eyes_state["fail"] = eyes_state["fail"] or "ok 50 50 1.00 1.00", None
elif w[0] == "calib-fit":
eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])}
reply = f"ok {len(eyes_state['points'])} dots"
else:
reply = f"fail unknown command {w[0]}"
if addr:
eyes.sendto(reply.encode(), addr)
threading.Thread(target=eyes_answer, daemon=True).start()
helper_state = {"reply": "ok off worn", "heard": []}
helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
helper.bind(gazed.POINTER)
helper.settimeout(0.2)
def helper_answer():
while True:
try:
data, addr = helper.recvfrom(512)
except socket.timeout:
continue
except OSError:
return
if data.startswith(b"gaze ?") and addr:
helper.sendto(helper_state["reply"].encode(), addr)
else:
helper_state["heard"].append(data.decode())
threading.Thread(target=helper_answer, daemon=True).start()
svc = gazed.Service(None, False, gazed.POINTER)
threading.Thread(target=svc.run, daemon=True).start()
ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
ctl.bind("")
ctl.settimeout(3)
def ask(cmd):
ctl.sendto(cmd.encode(), gazed.ME)
return ctl.recv(65536).decode()
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
def wait(cond, seconds):
end = time.monotonic() + seconds
while time.monotonic() < end:
if cond():
return True
time.sleep(0.05)
return cond()
def check_state():
return svc.checks.check or {}
def take_dot(i):
"""Wait for dot i to settle, then click: is it taken?"""
wait(lambda: check_state().get("i") == i and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
before = check_state().get("captured", 0)
ask("calaccept")
return wait(lambda: check_state().get("captured", 0) > before, 2)
check("gaze mode off, Gaze page open (wake): ours runs, uncalibrated", ask("wake 60"), "ok")
check("the service knows ours has no calibration", wait(lambda: svc.checks.calibrated() is False, 6), True)
check("a quick check is refused while ours has no calibration", svc.checks.start("quick", "test"),
"error our tracker isn't calibrated yet: use Calibrate")
helper_state["reply"] = "ok on worn"
check("gaze mode on: the calibration opens by itself", wait(lambda: check_state().get("kind") == "full", 6), True)
check("it runs blind (no gaze from ours yet)", check_state().get("blind"), True)
check("nothing says it can't open", any("can't open" in m for m in logs), False)
# Live 2026-10-05: turning gaze mode on woke our tracker, and the calibration opened before
# DON_DELAY of eyes had passed, so "the headset went on" came while it was open. That re-armed
# it, and a second calibration opened as soon as the first ended.
svc.checks.away, svc.checks.back_since = True, None
check("dot 1: a click takes it", take_dot(0), True)
check("the headset went on while it was open",
wait(lambda: not svc.checks.away, gazecheck.DON_DELAY + 2) and bool(svc.checks.check), True)
t0, t1, yaw, pitch = eyes_state["points"][0]
dot = gazecheck.check_dots("full", True)[0]
check("its window is the look up to the click (about CHECK_WINDOW)",
abs((t1 - t0) - gazecheck.CHECK_WINDOW) < 0.15, True)
check("ft-eyes got that dot's direction", (round(yaw, 3), round(pitch, 3)), (round(dot[0], 3), round(dot[1], 3)))
eyes_state["fail"] = "fail the left eye was seen in only 3 frames"
wait(lambda: check_state().get("i") == 1 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
ask("calaccept")
check("ft-eyes refusing a dot: its reason reaches the panel's note",
wait(lambda: "left eye in only 3 frames" in check_state().get("note", ""), 2), True)
check("dot 2, second try: taken", take_dot(1), True)
check("dot 3: taken", take_dot(2), True)
check("all dots: ours fits its calibration (calib-fit)",
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 4), True)
check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is True, 4), True)
check("and gaze mode stays on", "gaze off" in helper_state["heard"], False)
check("and no second calibration opens", wait(lambda: svc.checks.check is not None, 3), False)
check("one calibration in the log", sum(m.startswith("full check:") for m in logs), 1)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
svc.running = False
time.sleep(0.7)
shutil.rmtree(HOME, ignore_errors=True)
os._exit(1 if failures else 0)
+8 -1
View File
@@ -9,6 +9,7 @@
# Options (piped, they go after "bash -s --"):
# --stable the main branch: tested releases (the default for a new install)
# --experimental the experimental branch: the newest features, less tested
# --branch NAME another branch, such as a fix to test before it's released
# --dir DIR where the repo goes (default ~/frametop)
# --clone-only get or update the repo, but don't run install.sh
# --yes, --no-bluetooth passed to install.sh (--yes also answers this script's question:
@@ -17,7 +18,7 @@ set -euo pipefail
usage() {
cat <<'EOF'
usage: get.sh [--stable | --experimental] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
EOF
}
@@ -31,6 +32,7 @@ main() {
case $1 in
--stable) branch=main ;;
--experimental) branch=experimental ;;
--branch) branch=${2:?--branch needs a branch name}; shift ;;
--dir) dir=${2:?--dir needs a folder}; shift ;;
--clone-only) clone_only=1 ;;
--yes) yes=1; pass+=("$1") ;;
@@ -80,6 +82,11 @@ main() {
fi
fi
if ! git ls-remote --exit-code --heads "$repo" "$branch" >/dev/null; then
echo "Frametop has no branch called $branch (or GitHub can't be reached)." >&2
return 1
fi
if [ ! -e "$dir" ]; then
echo "Cloning Frametop ($branch) into $dir"
git clone --branch "$branch" "$repo" "$dir"
+68
View File
@@ -0,0 +1,68 @@
---
title: Install the Frametop Hand Recorder
---
# Install the Frametop Hand Recorder
The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face). These are the Konsole commands to install it.
> **Fixed in Frametop 0.2.1 (October 5, 2026):** the recorder now works on headsets without the Arcturus color passthrough module too. If you installed it earlier and the camera check stopped with "Not all of the headset's tracking cameras are running (ft-camd publishes only 2 of 4 mono cameras ...)", run the commands under [Update](#update).
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions and help with recording.
For now the recorder runs inside Frametop's desktop, so these steps install Frametop first. A standalone recorder that runs from the SteamVR dashboard without Frametop is planned.
## Before you start
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/DeeJanuz/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
## Open Konsole
1. In the launcher, choose Launch a program → Desktop.
2. In the application menu, open System → Konsole.
## 1. Install Frametop
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
```
This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
## 2. Install the Hand Recorder
After SteamVR restarts, choose Launch a program → Desktop again, open Konsole, and run:
```
~/frametop/hands/rec/install.sh
```
This builds the camera broker, the hand tracker and the headset panel (the first build takes a few minutes), asks for your `sudo` password once to let the camera broker read the cameras, and adds Frametop Hand Recorder to the application menu.
## 3. Record
Open Frametop Hand Recorder from the desktop's application menu. It walks you through the consent text, a short checklist, the recording, a review of what you recorded, and the upload to Hugging Face. Nothing leaves the headset until you press Upload.
## Update
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
~/frametop/hands/rec/install.sh
```
Run the second command after SteamVR has restarted, as in the install.
## Uninstall
```
~/frametop/hands/rec/install.sh uninstall
```
This removes the menu entry. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/DeeJanuz/frametop#uninstall) in the README.
## Help
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/DeeJanuz/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
+1 -1
View File
@@ -36,7 +36,7 @@ hands/run.sh uninstall
Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them), read when ft-camd and ft-hands start:
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks and logs a warning if the hands disagree.
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
+42 -5
View File
@@ -56,13 +56,19 @@ ANSI = re.compile(r"\x1b\[[0-9;]*m")
STAMP = re.compile(r"^\w{3} \w{3} \d{2} \d{4} (\d{2}:\d{2}:\d{2})\.\d+ (\w+): ?(.*)$")
# Lines worth reading; anything else is skipped before the regexes (the log grows by MBs a day).
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Closing tracking camera", "Streaming",
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService")
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService", "ISP ")
# XRService's numbering of its tracking cameras (the TrackingCameraInit index): ft-hands names
# them this way too (track/main.cpp, cameras_from_xrservice_log).
NAMES = ("slam_left", "slam_right", "upper_left", "upper_right")
RE_PASSTHRU = re.compile(r"Passthrough connected but FPGA is (\S+) - loading VCINT")
RE_INTERLEAVE = re.compile(r"Upper cameras FPGA interleaving support: (\d)")
RE_TASKS = re.compile(r"Created (\d+) tasks \((\d+) tracking, (\d+) passthrough\)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)")
RE_STREAM = re.compile(r"Streaming resumed \(FPGA: (\S+), VC interleaving: (\w+)\)")
RE_STATE = re.compile(r"FPGA state check: (\S+)")
# Without the colour module XRService runs the side cameras through the ISP, as NV12 on other
# capture pipes (vfe0 and vfe1), and the upper pair on vfe3 and vfe4.
RE_ISP = re.compile(r"ISP (enabled|disabled) for tracking cameras")
class LogState:
@@ -85,7 +91,7 @@ class LogState:
def _new_episode(self, t):
self.closed = False
self.episode = {"start": t, "fpga_before": "", "vcint": "", "interleave": None, "tasks": None,
"inits": {}, "stream": "", "failure": "", "evidence": []}
"inits": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
if self.closed_at:
self.episode["evidence"].append(self.closed_at)
@@ -146,6 +152,12 @@ class LogState:
ep["interleave"] = int(m.group(1))
ep["evidence"].append(short)
return
m = RE_ISP.search(text)
if m:
ep = self._ep(t)
ep["isp"] = m.group(1) == "enabled"
ep["evidence"].append(short)
return
m = RE_TASKS.search(text)
if m:
ep = self._ep(t)
@@ -184,6 +196,12 @@ class LogState:
got = tuple(self.nodes[i] for i in (2, 3) if i in self.nodes)
return got if len(got) == 2 else UPPER_NODES
def camera_map(self):
"""{calibration name: /dev/videoN's N} from the latest camera start's TrackingCameraInit
lines (the whole log's when that start has none yet)."""
inits = (self.episode or {}).get("inits") or self.nodes
return {NAMES[i]: node for i, node in sorted(inits.items()) if 0 <= i < len(NAMES)}
def tracking_nodes(self):
got = tuple(self.nodes[i] for i in range(TRACKING) if i in self.nodes)
return got if len(got) == TRACKING else SIDE_NODES + UPPER_NODES
@@ -353,7 +371,9 @@ def check(log=None, proc=True, ring=True, ring_path=None):
status, reason = "unknown", "no XRService log in %s" % LOG_DIR
out = {"log": path, "xrservice": None, "ring": None,
"episode": state.snapshot()["episode"] if state else {},
"failure": state.snapshot()["failure"] if state else ""}
"failure": state.snapshot()["failure"] if state else "",
"map": {name: {"node": n, "pipe": pipe_name(n)} for name, n in state.camera_map().items()} if state else {},
"ring_missing": []}
if proc:
if pid is None:
@@ -389,13 +409,30 @@ def check(log=None, proc=True, ring=True, ring_path=None):
evidence.append("ft-camd (pid %d, %s): %d mono cameras: %s" % (
r["writer_pid"], "running" if r["alive"] else "stale ring", len(r["mono"]), names))
if r["alive"] and len(r["mono"]) < TRACKING and status == "ok":
status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras (it started while "
"they were missing: restart it)" % (len(r["mono"]), TRACKING))
have = {c["node"] for c in r["mono"]}
want = state.tracking_nodes() if state else SIDE_NODES + UPPER_NODES
out["ring_missing"] = [n for n in want if n not in have]
status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras, missing %s" % (
len(r["mono"]), TRACKING, " ".join("video%d" % n for n in out["ring_missing"]) or "?"))
out.update(status=status, reason=reason, evidence=evidence,
summary="ok" if status == "ok" else "%s: %s" % (status, reason))
return out
def pipe_name(node):
"""The capture pipe behind /dev/videoN (msm_vfe3_video0 and so on), or ""."""
try:
with open("/sys/class/video4linux/video%d/name" % node) as f:
return f.read().strip()
except OSError:
return ""
def is_ring_short(result):
"""XRService runs all the tracking cameras, but ft-camd doesn't publish them all."""
return bool(result) and result.get("status") == "degraded" and bool(result.get("ring_missing"))
def is_vcint_failure(result):
return bool(result) and result.get("status") == "degraded" and result.get("reason") == VCINT_REASON
+25 -2
View File
@@ -379,9 +379,15 @@ static bool resolve_block(cam_t *c)
return true;
}
/*
* Through the ISP (no colour module), the near-black exposures come out all zeros,
* the same every time, so their dequeues change no buffer and get no votes. Such
* an index is left unmapped (slot -1): on_frame counts its frames as dark without
* reading them. Most of a camera's indices silent means it isn't streaming yet.
*/
static bool resolve_each(cam_t *c)
{
int depth = c->maxindex + 1;
int depth = c->maxindex + 1, silent = 0;
bool taken[MAX_SLOTS] = { false };
for (int i = 0; i < depth; i++) {
@@ -396,6 +402,12 @@ static bool resolve_each(cam_t *c)
}
}
if (c->nobs[i] >= 3 && v1 <= 0.2 * c->nobs[i]) {
c->slot_of[i] = -1;
silent++;
continue;
}
if (c->nobs[i] < 3 || best < 0 || taken[best] || v1 < 0.8 * c->nobs[i] || v2 > 0.3 * c->nobs[i])
return false;
@@ -403,9 +415,14 @@ static bool resolve_each(cam_t *c)
c->slot_of[i] = best;
}
if (silent * 2 > depth)
return false;
printf("%s: queue depth %d, buffers mapped one by one:", c->slug, depth);
for (int i = 0; i < depth; i++)
printf(" %d", c->slot_of[i]);
c->slot_of[i] < 0 ? printf(" -") : printf(" %d", c->slot_of[i]);
if (silent)
printf(" (- : %d indices whose frames are all zeros, skipped as dark)", silent);
printf("\n");
return true;
}
@@ -677,6 +694,12 @@ static void on_frame(cam_t *c, int64_t index, uint32_t seq, uint64_t ts, uint64_
int slot = c->slot_of[index];
if (slot < 0) { /* an index whose frames are all zeros (resolve_each): a dark one */
c->dark++;
c->rc->dropped++;
return;
}
/* color runs at 60 fps: skip frames early enough that the asked rate holds, before any sync */
if (c->color && color_fps > 0 && evtime - c->last_pub_ns < (uint64_t)(1e9 / color_fps) - 3000000) {
c->paced++;
+24 -7
View File
@@ -544,22 +544,26 @@ static void probe_cameras(xr_state_t *st)
/*
* qcom-camss can report bytesperline as the visible width while the VFE
* writes a larger aligned pitch. sizeimage is right, so derive the pitch.
* For NV12, plane 0 normally holds the chroma rows after the luma (the side
* cameras through the ISP: 1056 wide, 1152 bytes a row); if it's too small for
* that, it holds the luma alone.
*/
unsigned xr_camera_stride(const xr_camera_t *c)
{
if (!c->height || !c->planesize[0])
return c->bytesperline ? c->bytesperline : c->width;
double bpp = 1.0;
if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21)
bpp = 1.5;
unsigned s = (unsigned)((double)c->planesize[0] / ((double)c->height * bpp));
bool yuv = c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21;
unsigned s = (unsigned)((double)c->planesize[0] / ((double)c->height * (yuv ? 1.5 : 1.0)));
if (s >= c->width && s <= c->width * 4)
return s;
s = (unsigned)(c->planesize[0] / c->height);
if (yuv && s >= c->width && s <= c->width * 4)
return s;
return c->bytesperline ? c->bytesperline : c->width;
}
@@ -591,7 +595,20 @@ void xr_camera_layout(const xr_camera_t *c, xr_layout_t *l)
l->pitch = xr_camera_stride(c);
l->width = c->width < l->pitch ? c->width : l->pitch;
if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21) {
/*
* Without the colour module, XRService runs the side cameras through the
* ISP ("ISP enabled for tracking cameras (main VFE available)" in its log),
* and they come out NV12. They're mono sensors, so the luma is the image.
*/
bool yuv = c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21;
if (yuv && c->role && !strcmp(c->role, "tracking")) {
l->fmt = XR_FMT_GREY8;
l->rows = c->height;
return;
}
if (yuv) {
l->fmt = XR_FMT_NV12;
l->rows = c->height + c->height / 2;
} else {
+1 -1
View File
@@ -29,7 +29,7 @@ Every part runs in the dev container, as ft-hands and Input Settings do. The hos
- **A tracking ft-hands** gives feedback through the hands file: which hands are seen, the palm's distance, the index tip. If none is running, the session starts `ft-hands --no-gestures --status 0` (unit `frametop-handrec-hands.service`). An ft-hands already running is used as it is.
- **A recording ft-hands** runs once per recording part: `ft-hands --record-only --record DIR --record-for SECONDS --record-hz 10 --status 0 --sides auto|0|1` (below, "Side cameras"). It runs as a plain child process of the session, ended with SIGTERM when the part ends. SIGTERM ends ft-hands' loop, and `Recorder` writes out its queue when it's destroyed. In step mode (below) a part is one step's countdown and hold, so a take has one part per step (about 40 in the hand poses); in auto mode a take is one part, plus one more after each pause. `--record-for` is only a safety net.
- Why a process per part rather than one kept alive and paused: measured in the dev container with `ft-ringplay`'s ring (2026-10-02), `ft-hands --record-only` writes its first set 16-27 ms after it starts and ends 4-6 ms after SIGTERM, so a new part costs nothing the 3 s countdown doesn't cover. Every reader already takes parts in order (`takes.py`, `validate.py` through the export's single stream, the labeller's `fhl_io.py`, numbering `sets-10.bin` after `sets-9.bin`), ft-hands needs no new control, and nothing is written while a step waits. Before the hold starts the session checks that the part has written a set (`Recorder.has_data`, up to 3 s more), so the hold is recorded from its first frame.
- **Side cameras.** ft-camd can name the two side cameras the wrong way round (hands/README.md, "Which camera is which"). The tracking ft-hands decides from the hands within about 2 s of them being in view (`HANDS_SWAP_SIDES=auto`, hands/track/sides.h) and publishes that in `/run/user/UID/frametop-hands/sides.json`. The session reads it (`sides.py`, `read_live`) and stores it in session.json `"sides"`. Each later part is recorded named right (`--sides 1` or `0`). Parts recorded before the decision use ft-camd's names (`--sides auto`; a record-only ft-hands can't tell), and readers rename them (`sides.py`). Each part's `names_swapped` goes into take.json `"parts"`. Without a tracking ft-hands nothing decides: `"swapped": null`, the names stay as recorded, and the maintainer's check (`hub_review check`, check_sides on a few sets per take) tells. `takes.py sides SESSION --set swapped|named` records a decision by hand.
- **Side cameras.** ft-camd can name the two side cameras the wrong way round (hands/README.md, "Which camera is which"). The tracking ft-hands decides from the hands within about 2 s of them being in view (`HANDS_SWAP_SIDES=auto`, hands/track/sides.h) and publishes that in `/run/user/UID/frametop-hands/sides.json`. With `HANDS_SWAP_SIDES` forced to `0` or `1`, the hands still decide what's published once they disagree (state `"forced, disagrees"`; `read_live` also corrects an ft-hands built before that). The session reads it (`sides.py`, `read_live`) and stores it in session.json `"sides"`. Each later part is recorded named right (`--sides 1` or `0`). Parts recorded before the decision use ft-camd's names (`--sides auto`; a record-only ft-hands can't tell), and readers rename them (`sides.py`). Each part's `names_swapped` goes into take.json `"parts"`. Without a tracking ft-hands nothing decides: `"swapped": null`, the names stay as recorded, and the maintainer's check (`hub_review check`, check_sides on a few sets per take) tells. `takes.py sides SESSION --set swapped|named` records a decision by hand.
- **ft-handpanel** runs as a child process with `--watch-stdin`. It shows the panel and logs poses during each take.
- **The headset button's reader** is a thread of the session (`ButtonReader`, below), not a process.
+5 -4
View File
@@ -457,11 +457,12 @@ class Backend(QObject):
"""Start stays off: the check found the cameras degraded (unless --ignore-cameras)."""
return not self._ignore_cameras and self._camera.get("status") == "degraded"
def _check_now(self):
def _check_now(self, repair=False):
"""repair (off this thread only: it may restart ft-camd, up to 15 s): see session.camera_check."""
mod = self._runner()
if not mod or self._session_options.get("dry_run"):
return {"status": "unknown", "summary": "not checked (dry run)", "reason": "dry run", "evidence": []}
return mod.camera_check()
return mod.camera_check(repair=repair)
@Slot()
def checkCameras(self):
@@ -470,7 +471,7 @@ class Backend(QObject):
return
self._camera_busy = True
self.cameraChanged.emit()
self._thread(lambda: self._cameraArrived.emit(self._check_now()))
self._thread(lambda: self._cameraArrived.emit(self._check_now(repair=True)))
def _on_camera(self, result):
self._camera = result
@@ -499,7 +500,7 @@ class Backend(QObject):
end = time.monotonic() + RECHECK_FOR_S
time.sleep(RECHECK_S * 2)
while time.monotonic() < end:
result = self._check_now()
result = self._check_now(repair=True)
# Done once a new XRService (a new log) has opened its cameras, ok or not.
if result.get("status") in ("ok", "degraded") and result.get("log") != before:
break
+1
View File
@@ -23,6 +23,7 @@ case ${1:-install} in
"$root/hands/rec/build.sh"
echo "== 4/4 ft-camd's capabilities (asks for your password) and the menu entry"
"$root/hands/run.sh" caps
"$root/scripts/conf-migrate.sh" # HANDS_SWAP_SIDES=0, the old default, becomes auto
fill_template "$root/hands/rec/ft-handrec.desktop" |
on_frame "mkdir -p ~/.local/share/applications && cat > $entry"
echo
+37 -4
View File
@@ -1153,11 +1153,33 @@ def _steamvr_version():
return ""
def camera_check():
"""camcheck.check() (the XRService log, its open cameras where readable, ft-camd's ring);
never raises: a failure is "unknown"."""
_camd_restarted = set() # ft-camd pids camera_check(repair=True) has restarted: once each
def _unit_pid(unit):
r = subprocess.run(host_command("systemctl", "--user", "show", "-p", "MainPID", "--value", unit),
capture_output=True, text=True, timeout=30)
try:
return camcheck.check()
return int(r.stdout.strip() or 0)
except ValueError:
return 0
def camera_check(repair=False):
"""camcheck.check() (the XRService log, its open cameras where readable, ft-camd's ring);
never raises: a failure is "unknown". repair (the window, never during a session): when
XRService runs every tracking camera but the recorder's own ft-camd doesn't publish them all
(it started while some were missing), restart it, once per ft-camd, and check again."""
try:
r = camcheck.check()
pid = (r.get("ring") or {}).get("writer_pid")
if repair and camcheck.is_ring_short(r) and pid not in _camd_restarted and _unit_pid(CAMD_UNIT) == pid:
_camd_restarted.add(pid)
stop_unit(CAMD_UNIT)
start_camd()
r = camcheck.check()
r["evidence"].append("restarted ft-camd (pid %d) because it published only some of the cameras" % pid)
return r
except Exception as e:
return {"status": "unknown", "summary": "unknown: the camera check failed (%s)" % e,
"reason": str(e), "evidence": []}
@@ -1169,6 +1191,10 @@ def camera_text(result):
return ""
if camcheck.is_vcint_failure(result):
return camcheck.USER_TEXT
if camcheck.is_ring_short(result):
return ("The headset's tracking cameras are all running, but the recorder can't read some of them (%s). "
"Close the Hand Recorder, run ~/frametop/hands/rec/install.sh again, and open it again. If that "
"doesn't help, ask in the Frametop Discord." % result.get("reason", ""))
return ("Not all of the headset's tracking cameras are running (%s). Restart SteamVR, or restart the "
"headset if that doesn't fix it." % result.get("reason", ""))
@@ -1451,6 +1477,10 @@ class Session:
cam = self._status.get("camera")
if cam:
self._session_json["camera"] = {"status": cam.get("status"), "reason": cam.get("reason", "")}
# which device each calibrated camera was (XRService's numbering) and whether XRService
# ran the side cameras through the ISP (no colour module): to check the names later
self._session_json["device"]["camera_map"] = cam.get("map") or {}
self._session_json["device"]["isp"] = (cam.get("episode") or {}).get("isp")
if self.dry_run:
self._session_json["dry_run"] = True
if self.speed != 1:
@@ -1545,6 +1575,9 @@ class Session:
self._save_session()
self._log("side cameras: %s (%s, %s)" % ("SWAPPED" if swapped else "as named", new["decided_by"],
json.dumps(new["evidence"])))
if new["state"] == "forced, disagrees":
self._log("side cameras: HANDS_SWAP_SIDES in ~/.config/frametop.conf forces names the hands say are "
"backwards; the recording goes by the hands. Set HANDS_SWAP_SIDES=auto.")
def _sides_swapped(self):
"""session.json's decision: True, False, or None (not known yet)."""
+5
View File
@@ -125,4 +125,9 @@ def read_live(path=None, ring_path=None, now_ns=None):
return None
except OSError:
return None
if (s.get("state") == "forced, disagrees" and s.get("swapped") is not None
and bool(s["swapped"]) == bool(s.get("names_swapped"))):
# An ft-hands built before 2026-10-06 kept a forced HANDS_SWAP_SIDES as the truth even
# when the hands disagreed. The hands are right: the truth is the other way round.
s = dict(s, swapped=not s["swapped"], decided_by="auto")
return s
+39
View File
@@ -91,6 +91,29 @@ class RulesTest(unittest.TestCase):
finally:
shutil.rmtree(d)
def test_read_live_forced(self):
"""HANDS_SWAP_SIDES=0 forced and the hands disagree: the hands are the truth, whether
ft-hands published them (built after 2026-10-06) or kept the forced value (before)."""
d = tempfile.mkdtemp()
try:
path = os.path.join(d, "sides.json")
def live(**kw):
s = dict(pid=1, ring_ino=0, mode="0", names_swapped=False,
updated_ns=time.clock_gettime_ns(time.CLOCK_MONOTONIC), **kw)
with open(path, "w") as f:
json.dump(s, f)
return sides.read_live(path)
self.assertEqual(live(state="forced", swapped=False, decided_by="config")["swapped"], False)
self.assertEqual(live(state="forced, agrees", swapped=False, decided_by="config")["swapped"], False)
old = live(state="forced, disagrees", swapped=False, decided_by="config")
self.assertEqual((old["swapped"], old["decided_by"]), (True, "auto"))
new = live(state="forced, disagrees", swapped=True, decided_by="auto")
self.assertEqual((new["swapped"], new["decided_by"]), (True, "auto"))
finally:
shutil.rmtree(d)
class TakesTest(unittest.TestCase):
"""A swapped session: one take's parts recorded before the decision (ft-camd's names) and
@@ -226,6 +249,22 @@ class SessionSidesTest(unittest.TestCase):
self.assertTrue(s._session_json["sides"]["reversed_from"]["swapped"])
self.assertFalse(takes.read_json(os.path.join(self.tmp, "session.json"))["sides"]["swapped"])
def test_read_sides_forced(self):
"""PR #4 on the dataset: HANDS_SWAP_SIDES=0 from the old example config, and the hands
disagree. The session takes the hands' answer, not the forced one."""
s = session.Session(os.path.join(self.tmp, "base"), {}, {}, "room", self.script, dry_run=True,
hands_dir=self.tmp)
s.session_dir = self.tmp
s._session_json = {"sides": {"swapped": None}}
self.live(mode="0", state="forced", swapped=False, decided_by="config", names_swapped=False, evidence=None)
s._read_sides(force=True)
self.assertIs(s._sides_swapped(), False)
self.live(mode="0", state="forced, disagrees", swapped=False, decided_by="config", names_swapped=False)
s._read_sides(force=True)
self.assertIs(s._sides_swapped(), True)
self.assertEqual(s._session_json["sides"]["decided_by"], "auto")
self.assertEqual(s._session_json["sides"]["reversed_from"]["decided_by"], "config")
def test_recorder_parts(self):
calls = []
+48 -2
View File
@@ -66,6 +66,22 @@ RESUME_OK = [L("12:30:00", "[SystemdInhibitor] Received systemd resume notificat
for i, n in enumerate((9, 13, 6, 7))] + \
[L("12:30:02", "[DeckardCaptureSource] Streaming resumed (FPGA: VCINT, VC interleaving: enabled)")]
EXIT = [L("13:00:00", "XRService - main thread exiting"), L("13:00:00", "Exiting XRService")]
# The colour module unplugged while SteamVR runs (2026-10-05 12:42): XRService reopens the
# cameras with the side pair through the ISP on vfe0 and vfe1 (NV12); VCINT stays loaded, so the
# upper pair stays on vfe2.
UNPLUG = [L("12:42:25", "Received passthrough camera connection event (connected=0)"),
L("12:42:25", "[DeckardCaptureSource] Closing tracking camera interfaces camerasToUse: 1111"),
L("12:42:26", "FPGA state check: VCINT (register value: 0x00021211)"),
L("12:42:26", "Upper cameras FPGA interleaving support: 1 (Driver features available = 1 | VCINT loaded = 1)"),
L("12:42:26", "[buildMediaCtlSetupTasks] ISP enabled for tracking cameras (main VFE available)"),
L("12:42:26", "[buildMediaCtlSetupTasks] Created 4 tasks (4 tracking, 0 passthrough)")] + \
[L("12:42:26", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
for i, n in enumerate((0, 3, 6, 7))]
# Started without the module (FrameEyeCameraFeed's layout): the upper pair on vfe3 and vfe4.
NO_MODULE = [L("10:00:02", "[buildMediaCtlSetupTasks] ISP enabled for tracking cameras (main VFE available)"),
L("10:00:02", "[buildMediaCtlSetupTasks] Created 4 tasks (4 tracking, 0 passthrough)")] + \
[L("10:00:03", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
for i, n in enumerate((0, 3, 9, 13))]
def state(lines):
@@ -128,6 +144,23 @@ class SyntheticLogs(unittest.TestCase):
self.assertEqual(status, "degraded")
self.assertEqual(reason, "only 2 of 4 tracking cameras running")
def test_camera_map_with_module(self):
st = state(START + GOOD_OPEN)
self.assertEqual(st.camera_map(), {"slam_left": 9, "slam_right": 13, "upper_left": 6, "upper_right": 7})
def test_module_unplugged(self):
st = state(START + GOOD_OPEN + UNPLUG)
self.assertEqual(st.verdict()[0], "ok")
self.assertIs(st.episode["isp"], True)
self.assertEqual(st.camera_map(), {"slam_left": 0, "slam_right": 3, "upper_left": 6, "upper_right": 7})
self.assertEqual(st.tracking_nodes(), (0, 3, 6, 7))
def test_started_without_module(self):
st = state(START + NO_MODULE)
self.assertEqual(st.verdict()[0], "ok")
self.assertEqual(st.camera_map(), {"slam_left": 0, "slam_right": 3, "upper_left": 9, "upper_right": 13})
self.assertEqual(st.upper_nodes(), (9, 13))
def test_exited_and_empty(self):
self.assertEqual(state(START + GOOD_OPEN + EXIT).verdict()[0], "unknown")
self.assertEqual(state([]).verdict()[0], "unknown")
@@ -223,9 +256,9 @@ class RealLog(unittest.TestCase):
self.assertEqual(out.getvalue().split("\n")[0], "ok")
def make_ring(path, mono_names, alive=True):
def make_ring(path, mono_names, alive=True, nodes=None):
"""A ring header as ft-camd writes it (camd/fhring.h), no frames."""
cams = [(b"og01a1b", n.encode(), 9 + i) for i, n in enumerate(mono_names)]
cams = [(b"og01a1b", n.encode(), nodes[i] if nodes else 9 + i) for i, n in enumerate(mono_names)]
hb = time.clock_gettime_ns(time.CLOCK_MONOTONIC) if alive else 1
data = bytearray(camcheck.RING_HDR.pack(b"FHRING01", 1, 0, len(cams), 0, 0, 4242, 0))
struct.pack_into("<Q", data, 40, hb)
@@ -252,6 +285,19 @@ class Ring(unittest.TestCase):
self.assertEqual(r["status"], "degraded")
self.assertIn("ft-camd publishes only 2 of 4", r["reason"])
def test_ring_missing_the_side_cameras(self):
# an ft-camd from before NV12 support, without the colour module: only the upper pair
with open(self.log, "w") as f:
f.write("\n".join(START + NO_MODULE) + "\n")
make_ring(self.ring, ["og0ve10_5-003e_video9", "og0ve10_5-0060_video13"], nodes=[9, 13])
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
self.assertEqual(r["status"], "degraded")
self.assertEqual(r["ring_missing"], [0, 3])
self.assertIn("missing video0 video3", r["reason"])
self.assertTrue(camcheck.is_ring_short(r))
self.assertFalse(camcheck.is_vcint_failure(r))
self.assertEqual(r["map"]["slam_left"]["node"], 0)
def test_four_cameras_in_ring(self):
make_ring(self.ring, ["a_video9", "b_video13", "c_video6", "d_video7", "a_video9_dk"])
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
+23 -2
View File
@@ -34,8 +34,28 @@ from tools.show_set import index, read_set # noqa: E402
from tools import calib # noqa: E402
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # as ft-hands maps them
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # with the colour module
'msm_vfe2_video0': 'upper_left', 'msm_vfe2_video1': 'upper_right'}
def ring_names():
"""{/dev/videoN's N: calibration name} as ft-hands names them: by XRService's log
(camcheck.py), else by capture pipe (PIPES)."""
import camcheck
try:
by_log = {node: name for name, node in camcheck.read_log(camcheck.newest_log()).camera_map().items()}
except (OSError, ValueError):
by_log = {}
if by_log:
return by_log
out = {}
for path in os.listdir('/sys/class/video4linux'):
if path.startswith('video'):
with open('/sys/class/video4linux/%s/name' % path) as f:
name = PIPES.get(f.read().strip())
if name:
out[int(path[5:])] = name
return out
PAIRS = {'side': ('slam_left', 'slam_right'), 'upper': ('upper_left', 'upper_right')}
@@ -100,8 +120,9 @@ def live_pairs(count, names=PAIRS['side']):
if not ring.alive():
sys.exit('ft-camd isn\'t running (no heartbeat)')
cams = {}
names_of = ring_names()
for c in ring.cams:
name = PIPES.get(open('/sys/class/video4linux/video%d/name' % c.node).read().strip())
name = names_of.get(c.node)
if name and not c.name.endswith('-dark'):
cams[name] = c
for k in range(count):
+70 -14
View File
@@ -27,7 +27,8 @@
// default) tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// cameras if they're backwards (the tracked views move with their images), and checks once
// more. 0 and 1 force the naming (1: exchanged; --swap-sides is --sides 1); it still checks,
// and warns if the hands disagree. The decision is published in /run/user/UID/frametop-hands/sides.json (see
// and if the hands disagree it warns and publishes what the hands say as the truth ("swapped"),
// so recordings are labelled right while tracking keeps the forced names. The decision is published in /run/user/UID/frametop-hands/sides.json (see
// write_sides below) and, for recordings, in DIR/sides.json. --record-only can't tell (it tracks
// nothing): under auto it records the ring's names as they are.
//
@@ -63,7 +64,34 @@ namespace {
volatile std::sig_atomic_t g_stop = 0, g_record = 0;
// which calibrated camera each capture pipe carries (XRService's fixed routing)
// Which calibrated camera each video device carries. XRService numbers its tracking cameras
// (index 0 to 3: slam_left, slam_right, upper_left, upper_right) and logs the device each one
// opened ("TrackingCameraInit: index: 0. video device: /dev/video9"). The devices depend on
// the colour module: with it, the side cameras are on vfe3 and vfe4 and the upper pair on
// vfe2; without it, XRService runs the side cameras through the ISP on vfe0 and vfe1, and the
// upper pair on vfe3 and vfe4. So the running XRService's log decides; when it can't be read,
// the capture pipes as they are with the module. {} if the log has no cameras.
std::map<int, std::string> cameras_from_xrservice_log() {
static const char *const names[] = {"slam_left", "slam_right", "upper_left", "upper_right"};
const char *home = std::getenv("HOME");
std::ifstream in(std::string(home ? home : "") + "/.local/share/Steam/logs/xrservice.txt");
const std::string key = "TrackingCameraInit: index: ";
std::map<int, int> node_of; // index -> N of /dev/videoN, from the latest camera start
std::string line;
while (std::getline(in, line)) {
if (line.find("XRService logging to") != std::string::npos) node_of.clear();
const auto at = line.find(key);
int index = -1, node = -1;
if (at != std::string::npos &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d", &index, &node) == 2 &&
index >= 0 && index < 4)
node_of[index] = node;
}
std::map<int, std::string> out;
for (auto &[index, node] : node_of) out[node] = names[index];
return out;
}
const char *camera_for_pipe(int node) {
char path[64], name[64] = "";
std::snprintf(path, sizeof path, "/sys/class/video4linux/video%d/name", node);
@@ -302,6 +330,7 @@ int main(int argc, char **argv) {
double decided_after_s = -1;
if (sides_mode != "auto") truth = names_swapped, decided_by = sides_from, sides_state = "forced";
std::string rec_dir;
std::string cams_json; // which device each calibrated camera is (set below), for the sides files
std::vector<std::pair<size_t, bool>> rec_names; // from which recorded set on, names_swapped was what
// DIR/sides.json beside a recording's sets.bin: how its side cameras are named. A set's
// names are right when its names_swapped equals swapped (null: not known when recorded).
@@ -312,7 +341,8 @@ int main(int argc, char **argv) {
write_file(rec_dir + "/sides.json",
"{\"swapped\": " + json_bool(truth) + ", \"decided_by\": " + json_str(truth ? decided_by : "") +
", \"names_swapped\": [" + runs + "]" +
(decision_evidence.empty() ? "" : ", \"evidence\": " + decision_evidence) + "}\n");
(decision_evidence.empty() ? "" : ", \"evidence\": " + decision_evidence) +
(cams_json.empty() ? "" : ", \"cameras\": " + cams_json) + "}\n");
};
auto start_recording = [&](const std::string &dir, std::string &e) {
rec = std::make_unique<Recorder>();
@@ -340,19 +370,42 @@ int main(int argc, char **argv) {
// (--with-color). Recorded names hold 15 characters, so "upper_right_dark" wouldn't fit.
std::map<std::string, int> dark;
std::map<std::string, Camera> used;
// ft-camd's cameras by XRService's numbering, else by capture pipe (see camera_for_pipe);
// ft-ringplay's (no device) by the name it gives
const std::map<int, std::string> by_log = cameras_from_xrservice_log();
const char *named_by = by_log.empty() ? "capture pipe" : "XRService's log";
for (int i = 0; i < ring.cameras(); ++i) {
if (ring.camera(i).flags & FH_CAM_COLOR) {
const std::string name = "color_video" + std::to_string(ring.camera(i).node);
const fh_ring_cam_t &rc = ring.camera(i);
if (rc.flags & FH_CAM_COLOR) {
const std::string name = "color_video" + std::to_string(rc.node);
dark[name] = i, color[name] = i;
continue;
}
// ft-camd's cameras by capture pipe; ft-ringplay's (no device) by the name it gives
const char *name = camera_for_pipe(ring.camera(i).node);
if (!name && ring.camera(i).node < 0) name = ring.camera(i).name;
if (!name || !calib.count(name)) continue;
if (ring.camera(i).flags & FH_CAM_DARK) dark[std::string(name) + "_dk"] = i;
else index[name] = i, used[name] = calib[name];
const auto it = by_log.find(rc.node);
const char *name = rc.node < 0 ? rc.name
: !by_log.empty() ? (it == by_log.end() ? nullptr : it->second.c_str())
: camera_for_pipe(rc.node);
if (!name || !calib.count(name)) {
if (!(rc.flags & FH_CAM_DARK)) std::fprintf(stderr, "video%d (%s): not one of the calibrated cameras, left out\n", rc.node, rc.name);
continue;
}
if (int(rc.width) != calib[name].width || int(rc.height) != calib[name].height) {
std::fprintf(stderr, "video%d (%s) is %ux%u, but %s is calibrated at %dx%d: left out\n", rc.node, rc.name,
rc.width, rc.height, name, calib[name].width, calib[name].height);
continue;
}
if (rc.flags & FH_CAM_DARK) {
dark[std::string(name) + "_dk"] = i;
} else if (index.count(name)) {
std::fprintf(stderr, "video%d (%s) would be %s too (video%d is): left out\n", rc.node, rc.name, name,
ring.camera(index[name]).node);
} else {
index[name] = i, used[name] = calib[name];
cams_json += std::string(cams_json.empty() ? "" : ", ") + json_str(name) + ": {\"node\": " +
std::to_string(rc.node) + ", \"ring\": " + json_str(rc.name) + "}";
}
}
cams_json = "{\"named_by\": " + json_str(named_by) + ", \"cameras\": {" + cams_json + "}}";
// ft-camd tells the side cameras' buffers apart by XRService's allocation order, which
// some XRService restarts reverse (see the top).
const bool have_sides = index.count("slam_left") && index.count("slam_right");
@@ -397,7 +450,7 @@ int main(int argc, char **argv) {
}
const bool switching = automatic && !color.empty();
Cams mode = switching || color.empty() ? Cams::Mono : fixed;
std::printf("cameras:");
std::printf("cameras (by %s):", named_by);
for (auto &[name, i] : index) std::printf(" %s=video%d", name.c_str(), ring.camera(i).node);
for (auto &[name, i] : color) std::printf(" %s", name.c_str());
std::printf(" tracking with %s%s models: %s%s, %d threads on CPUs", cams_name(mode),
@@ -446,6 +499,7 @@ int main(int argc, char **argv) {
", \"decided_after_s\": " + std::to_string(decided_after_s) +
", \"evidence\": " + (decision_evidence.empty() ? "null" : decision_evidence) +
", \"checking\": " + (checking ? side_check.json() : "null") +
", \"cameras\": " + (cams_json.empty() ? "null" : cams_json) +
", \"updated_ns\": " + std::to_string(mono_ns()) + "}\n");
};
write_sides();
@@ -454,14 +508,16 @@ int main(int argc, char **argv) {
const bool backwards = v == SideCheck::Swapped; // relative to the names as they are now
const double after = (now - start) / 1e9;
const std::string ev = side_check.json(), text = side_check.summary();
if (sides_mode != "auto") { // forced: only say so
if (sides_mode != "auto") { // forced: the names stay; if the hands disagree, they're the truth
const std::string what = (sides_from == "option" ? "--sides " : "HANDS_SWAP_SIDES=") + sides_mode;
if (backwards)
std::printf("side cameras: %s looks WRONG: the hands say the side cameras are the other way round (%s). "
"Use auto.\n", what.c_str(), text.c_str());
"Tracking keeps the forced names; recordings are labelled by the hands. Use auto.\n",
what.c_str(), text.c_str());
else
std::printf("side cameras: %s agrees with the hands (%s)\n", what.c_str(), text.c_str());
sides_state = backwards ? "forced, disagrees" : "forced, agrees";
if (backwards) truth = !names_swapped, decided_by = "auto", decided_after_s = after;
decision_evidence = ev;
checking = false;
} else if (side_round == 0 || backwards) {
+5 -2
View File
@@ -77,6 +77,8 @@ ACTION_LABELS = {
"screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
"float_toggle": "Float window in VR / put it back", "dock_all": "Put all floating windows back",
"pause_toggle": "Pause/resume Frametop (for VR games)",
"spin_next": "Spin the panels: next one on the right to the front",
"spin_prev": "Spin the panels: next one on the left to the front",
"key": "Pass through as key",
"none": "Do nothing",
}
@@ -163,9 +165,10 @@ GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, releas
MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window.
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window,
# Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
def key_bindings(rules):
+30 -12
View File
@@ -30,7 +30,9 @@ gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
window back (both to ft-floatd, @frametop_float), profile:NAME = switch to that profile (ft-layout
window back (both to ft-floatd, @frametop_float), spin_next and spin_prev = turn every panel in the
room about your head so the next one to the right or left comes to the front (ft-screens' "spin",
Meta+Alt+Tab and Meta+Alt+Shift+Tab by default), profile:NAME = switch to that profile (ft-layout
use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), pause_toggle =
pause Frametop for a VR game, or resume it (game_pause.py), command:CMD = run
@@ -152,10 +154,10 @@ REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C} # REL_HWHEEL, REL_WHEEL and their _HI_RES
BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
# Volume keys are remapped to KEY_MACRO29 and KEY_MACRO30: above 255, so X11 can't
# carry them, and bound to nothing in the default keymap.
VOLUME_STANDIN = {KEY_VOLUMEUP: 0x2AC, KEY_VOLUMEDOWN: 0x2AD}
KEY_MUTE, KEY_VOLUMEDOWN, KEY_VOLUMEUP = 113, 114, 115
# Volume keys are remapped to KEY_MACRO28, KEY_MACRO29 and KEY_MACRO30: above 255, so X11
# can't carry them, and bound to nothing in the default keymap.
VOLUME_STANDIN = {KEY_MUTE: 0x2AB, KEY_VOLUMEUP: 0x2AC, KEY_VOLUMEDOWN: 0x2AD}
VOLUME_ORIGINAL = {v: k for k, v in VOLUME_STANDIN.items()}
VOLUME_CODES = set(VOLUME_STANDIN) | set(VOLUME_ORIGINAL)
BUS_USB, BUS_BLUETOOTH, BUS_VIRTUAL = 0x03, 0x05, 0x06
@@ -209,14 +211,16 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
"gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
"dock_all", "steam_menu", "pause_toggle", "key", "none")
"dock_all", "spin_next", "spin_prev", "steam_menu", "pause_toggle", "key", "none")
# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
# Meta+Shift+F floats a window.
# Meta+Shift+F floats a window, and Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels around
# you (ft-screens' lazy susan); not Meta+Tab, which is Cmd+Tab on a Mac reached through a remote
# desktop like RustDesk.
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
"42+125+33": "float_toggle"}
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
KEY_F24 = 194 # sent to the desktop with a Meta combination (see the top)
# Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
@@ -232,6 +236,9 @@ GAZED = "\0ft_gazed"
FLOAT = "\0frametop_float" # ft-floatd, floating windows in the Frametop desktop
# Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
# Actions for ft-screens: spin_next and spin_prev turn every panel in the room about your head,
# so the next one to the right or left comes to the front. They don't need pointer mode either.
SCREENS_ACTIONS = {"spin_next": b"spin next", "spin_prev": b"spin prev"}
PROFILE = "profile:" # "profile:NAME": switch to that profile (doesn't need pointer mode either)
COMMAND = "command:" # "command:CMD": run CMD (nor does this)
@@ -242,7 +249,7 @@ def known_action(a):
def needs_pointer(a):
return a not in FLOAT_ACTIONS and a not in ("steam_menu", "pause_toggle") and not a.startswith((PROFILE, COMMAND))
return a not in FLOAT_ACTIONS and a not in SCREENS_ACTIONS and a not in ("steam_menu", "pause_toggle") and not a.startswith((PROFILE, COMMAND))
def works_paused(a):
@@ -422,9 +429,13 @@ class Volume:
def key(self, fd, code, value, now):
if value == 1:
self.held = (fd, code)
self.step(code)
self.next_at = now + self.DELAY
if VOLUME_ORIGINAL.get(code, code) == KEY_MUTE:
subprocess.Popen(["wpctl", "set-mute", "@DEFAULT_AUDIO_SINK@", "toggle"],
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
else:
self.held = (fd, code)
self.step(code)
self.next_at = now + self.DELAY
elif value == 0 and self.held == (fd, code):
self.release()
@@ -925,6 +936,13 @@ def main():
except OSError:
pass # the Frametop desktop isn't running
log(action)
elif action in SCREENS_ACTIONS:
if value == 1:
try:
screens_sock.sendto(SCREENS_ACTIONS[action], SCREENS)
except OSError:
pass # the Frametop desktop isn't running
log(action)
elif state["pointer"]:
state["pointer"].action(action, value, now, source)
+15 -1
View File
@@ -206,7 +206,7 @@ def check(label, got, want):
failures.append(label)
META, RMETA, SHIFT, CTRL, RCTRL, A, F, J, P = 125, 126, 42, 29, 97, 30, 33, 36, 25
META, RMETA, SHIFT, CTRL, RCTRL, ALT, A, F, J, P, TAB = 125, 126, 42, 29, 97, 56, 30, 33, 36, 25, 15
F24 = ["key 194 1", "key 194 0"]
@@ -265,6 +265,20 @@ def tests():
key(META, 1); key(J, 1); key(J, 0); key(META, 0)
check("resumed: Meta+J is a combination again", typed(), ["key 125 1"] + F24 + ["key 125 0"])
use(None) # the defaults: Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels (ft-screens)
key(META, 1); key(ALT, 1); key(TAB, 1); key(TAB, 0); key(ALT, 0); key(META, 0)
got = typed()
check("Meta+Alt+Tab (default): spin next", [m for m in got if m.startswith("spin")], ["spin next"])
check("Meta+Alt+Tab: Tab isn't typed", [m for m in got if m.startswith("key 15 ")], [])
key(META, 1); key(ALT, 1); key(SHIFT, 1); key(TAB, 1); key(TAB, 0); key(SHIFT, 0); key(ALT, 0); key(META, 0)
check("Meta+Alt+Shift+Tab (default): spin prev", [m for m in typed() if m.startswith("spin")], ["spin prev"])
pause("on")
key(META, 1); key(ALT, 1); key(TAB, 1); key(TAB, 0); key(ALT, 0); key(META, 0)
check("paused: Meta+Alt+Tab doesn't spin", [m for m in typed() if m.startswith("spin")], [])
pause("off")
check("spinning works without pointer mode", (relay.needs_pointer("spin_next"), relay.needs_pointer("spin_prev")),
(False, False))
check("an empty command isn't an action", relay.known_action("command: "), False)
check("steam_menu, pause_toggle and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
+2 -1
View File
@@ -75,7 +75,7 @@ step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, b
step "4/10 3D mouse: SteamVR driver"
"$root/pointer/driver/build.sh"
"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
"$root/pointer/driver/install.sh" install
step "5/10 3D mouse: pointer helper service"
"$root/pointer/helper/build.sh"
@@ -93,6 +93,7 @@ step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frame
"$root/remote/install.sh"
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
"$root/scripts/conf-migrate.sh"
step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
gaze=0
+3
View File
@@ -0,0 +1,3 @@
#!/bin/bash
# Hide/Show Screens' own program (see ft-layout-reset).
exec "$(dirname "$(readlink -f "$0")")/ft-layout" toggle "$@"
+4
View File
@@ -0,0 +1,4 @@
#!/bin/bash
# Reset Screen Layout's own program. Steam lists menu entries by program, so entries that
# shared ft-layout (with different arguments) launched as one.
exec "$(dirname "$(readlink -f "$0")")/ft-layout" apply "$@"
+2 -1
View File
@@ -534,7 +534,8 @@ def capture_screens():
# ---------------------------------------------------------------- gamescope (dashboard panels)
def vrcmd(*args, timeout=10):
env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD))
# SteamVR's config folder: in the desktop, XDG_CONFIG_HOME is its own (docs/design.md).
env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD), XDG_CONFIG_HOME=os.path.expanduser("~/.config"))
try:
return subprocess.run([VRCMD, *args], capture_output=True, text=True, timeout=timeout, env=env).stdout
except (OSError, subprocess.TimeoutExpired):
+21 -5
View File
@@ -14,6 +14,10 @@ frame="$root/scripts/frame.sh"
src=$FRAME_REPO/pointer/driver
dest='$HOME/.local/share/frametop/ft_pointer' # expanded on the Frame, in the commands below
reg='/opt/steamvr/bin/linuxarm64/vrpathreg'
# SteamVR's tools with SteamVR's config folder and libraries. A terminal in the Frametop desktop
# has XDG_CONFIG_HOME=~/.config/frametop (session/frametop-session.sh), where vrpathreg finds no
# registry, so it made one there that SteamVR never reads.
steamvr='env XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64' # expanded on the Frame
case ${1:-install} in
install)
@@ -21,6 +25,11 @@ case ${1:-install} in
# Replacing the files of a driver SteamVR has loaded leaves its input bindings in a bad
# state (the 3D mouse no longer gets the laser) until SteamVR restarts, so an unchanged
# driver is left alone.
# A registry an earlier install made in the desktop's config folder (see steamvr above) has
# no SteamVR in it, and hid SteamVR from OpenVR programs started in the desktop
# (VRInitError_Init_InstallationNotFound): it goes.
# vrpathreg runs 'xdg-open vrmonitor://driverinstalled' for SteamVR's desktop monitor, which
# the Frame doesn't have, so the desktop said "could not read file": a stand-in takes it.
"$frame" --host "set -e; test -f $src/build/driver_ft_pointer.so
rm -rf $dest.new; mkdir -p $dest.new/bin/linuxarm64
cp -r $src/ft_pointer/. $dest.new/
@@ -31,15 +40,22 @@ else
rm -rf $dest; mv $dest.new $dest
echo 'installed; restart SteamVR to load it'
fi
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg adddriver $dest
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg show | grep -A3 -i 'external'" ;;
stray=\$HOME/.config/frametop/openvr/openvrpaths.vrpath
if grep -qs '\"jsonid\" : \"vrpathreg\"' \$stray && grep -qsE '\"runtime\"[[:space:]]*:[[:space:]]*null' \$stray; then
rm -f \$stray; rmdir \$HOME/.config/frametop/openvr 2>/dev/null || true
echo 'removed a SteamVR path registry an earlier install left in ~/.config/frametop/openvr'
fi
noopen=\$(mktemp -d); trap 'rm -rf \$noopen' EXIT
printf '#!/bin/sh\nexit 0\n' > \$noopen/xdg-open; chmod +x \$noopen/xdg-open
PATH=\$noopen:\$PATH $steamvr $reg adddriver $dest
$steamvr $reg show | grep -A3 -i 'external'" ;;
uninstall)
"$frame" --host "LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;;
"$frame" --host "$steamvr $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;;
send)
"$frame" --host "python3 -c 'import socket,sys; s=socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM); s.sendto(sys.argv[1].encode(), \"\\0ft_pointer\")' $(printf %q "${2:?command}")" ;;
probe) "$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' ;;
probe) "$frame" -C pointer/probe "$steamvr ./build/vrprobe" ;;
aimhere)
read -r yaw pitch < <("$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p')
read -r yaw pitch < <("$frame" -C pointer/probe "$steamvr ./build/vrprobe" | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p')
[ -n "${yaw:-}" ] || { echo "head pose not valid (headset off?)" >&2; exit 1; }
"$0" send "aim $yaw $pitch" && echo "aimed at yaw $yaw pitch $pitch" ;;
log) "$frame" --host "grep -iE 'ft_pointer' ~/.local/share/Steam/logs/vrserver.txt | tail -n ${2:-20}" ;;
+14
View File
@@ -346,6 +346,20 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
struct screen *sc = s->screens[e->screen];
struct wlr_surface *surface = sc->toplevel->base->surface;
if (e->type == FT_FRONT) {
// A spin brought this panel to the front: typing goes to it, as after a click there,
// and ft-floatd makes its window (or the top one on a screen) KWin's active window.
wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
s->keys_clicked = true;
char msg[32];
snprintf(msg, sizeof msg, "front %d", e->screen + 1);
struct sockaddr_un addr = {.sun_family = AF_UNIX};
const char name[] = "frametop_float";
memcpy(addr.sun_path + 1, name, sizeof name - 1);
sendto(s->relay_fd, msg, strlen(msg), MSG_DONTWAIT, (struct sockaddr *)&addr,
offsetof(struct sockaddr_un, sun_path) + 1 + sizeof name - 1);
return;
}
const uint32_t t = now_ms();
const double x = e->x / s->scale[e->screen], y = e->y / s->scale[e->screen]; // KWin's units
switch (e->type) {
+15 -1
View File
@@ -459,7 +459,21 @@ void Hide() {
bool Shown() { return g_shown; }
void SetLasers(bool on) {
if (Create()) vr::VROverlay()->SetOverlayFlag(g_overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, on);
static vr::VROverlayHandle_t set = vr::k_ulOverlayHandleInvalid; // the overlay `was` is for
static bool was = false;
if (!Create() || (set == g_overlay && was == on)) return;
set = g_overlay, was = on;
vr::VROverlay()->SetOverlayFlag(g_overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, on);
}
bool Aimed(const vr::HmdMatrix34_t &laser) {
if (!g_shown) return false;
vr::VROverlayIntersectionParams_t in{};
in.eOrigin = vr::TrackingUniverseStanding;
in.vSource = {laser.m[0][3], laser.m[1][3], laser.m[2][3]};
in.vDirection = {-laser.m[0][2], -laser.m[1][2], -laser.m[2][2]};
vr::VROverlayIntersectionResults_t out{};
return vr::VROverlay()->ComputeOverlayIntersection(g_overlay, &in, &out);
}
void Poll(void (*handle)(const Event &, void *), void *data) {
+2
View File
@@ -26,6 +26,8 @@ const vr::HmdMatrix34_t &Pose();
void EndDragBy(uint32_t device);
// Controllers' lasers work the panel with the dashboard closed, like the screens'.
void SetLasers(bool on);
// A laser (its pose, aiming along -z) points at the panel.
bool Aimed(const vr::HmdMatrix34_t &laser);
// The panel's input: key presses and releases, and Closed for its Close key.
void Poll(void (*handle)(const Event &, void *), void *data);
void Destroy();
+293 -39
View File
@@ -12,9 +12,7 @@
// - a resize tab on the bottom right corner: drag it to set the width (the height
// follows the screen's resolution).
// - a reset button left of the bar: every screen back in its layout, around where you
// are now (`ft-layout apply`, like Meta+Shift+R). Where the screens leave the
// controllers to a VR game, aiming a controller at it turns SteamVR's laser mouse on for
// that button alone (UpdateResetLaser), so it can be clicked in a game.
// are now (`ft-layout apply`, like Meta+Shift+R).
// The controls are translucent, like SteamVR's own, and brighten under a laser. They
// are invisible until a laser (a controller's, or the 3D mouse's) lands on or passes very close to
// one of them (UpdateControls).
@@ -44,7 +42,10 @@
// default it's off while a game (a scene app) runs: the screens stay up over the game,
// the controllers stay in it, and the 3D mouse (its own laser mode) or the dashboard
// works the screens. Modes: always, outside_games (default), dashboard (never on its
// own; also for flatscreen games, which aren't scene apps).
// own; also for flatscreen games, which aren't scene apps). Where the mode leaves the
// controllers to the game, pointing a controller at a panel (a screen, a floating window
// and its popups, their controls, the keyboard) turns the laser on for it until you
// point away, like SteamVR's own floating windows (UpdateAim).
// - during a VR game the screens hide unless the dashboard is open (g_inGames, default),
// or stay visible over it; the hotkey still shows them.
// - paused ("pause on", from the input relay when Frametop pauses for a VR game,
@@ -189,8 +190,12 @@ Mat TipOffset(vr::TrackedDeviceIndex_t dev) {
Tip tip{model, Identity(), false, now};
vr::RenderModel_ControllerMode_State_t mode{};
vr::RenderModel_ComponentState_t state{};
if (model[0] && vr::VRRenderModels()->GetComponentStateForDevicePath(model, vr::k_pch_Controller_Component_Tip,
vr::k_ulInvalidInputValueHandle, &mode, &state))
// GetComponentState, not GetComponentStateForDevicePath: without an input source handle
// the latter fails for every component while a VR game runs, and the rays came from the
// pose, 40 degrees above the laser. The tip doesn't move with the buttons.
vr::VRControllerState_t buttons{};
if (model[0] && vr::VRRenderModels()->GetComponentState(model, vr::k_pch_Controller_Component_Tip, &buttons, &mode,
&state))
tip.offset = state.mTrackingToComponentLocal, tip.found = true;
cache[dev] = tip;
return tip.offset;
@@ -237,7 +242,7 @@ constexpr double kRollSnap = 2.5; // degrees from level where rolling snaps l
constexpr double kRollStep = 5; // degrees per scroll notch on the roll button
constexpr float kChromeIdle = 0.55f; // the controls' opacity without a laser on them
constexpr long kControlsLinger = 35; // ticks (~0.4 s) the controls stay after a laser leaves
constexpr long kResetLinger = 45; // ticks (~0.5 s) the reset button keeps the laser mouse on
constexpr long kAimLinger = 25; // ticks (~0.3 s) a panel keeps the laser on after the aim leaves it
long g_tick = 0; // ft_vr_poll calls
bool g_vr = false; // connected to SteamVR (ft-screens --no-vr runs without it)
constexpr vr::TrackedDeviceIndex_t kNone = vr::k_unTrackedDeviceIndexInvalid;
@@ -275,8 +280,7 @@ struct Screen {
float controls = 0; // the controls' fade, 0 (hidden) .. 1
bool controlsUp = false; // the controls' overlays are shown
long nearUntil = 0; // a laser was near the controls until this tick
long resetNearUntil = 0; // a hand controller aimed at the reset button until this tick
bool resetLaser = false; // the reset button has MakeOverlaysInteractiveIfVisible
long aimUntil = 0; // a hand controller pointed at it until this tick (UpdateAim)
vr::TrackedDeviceIndex_t pinTarget = kNone; // moving: rides on this controller when let go
vr::TrackedDeviceIndex_t onWrist = kNone; // moving: the laser is in this controller's ring
bool barLit = false;
@@ -910,26 +914,20 @@ void UpdateAttention() {
// Controllers' lasers on the screens (see the top): the flag follows the mode and whether a
// VR game runs.
// VR game runs, and where the mode leaves the controllers to the game, whether one points at
// the panel (UpdateAim).
bool LasersByMode() { return g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning); }
long g_keyboardAimUntil = 0;
void UpdateLasers() {
const bool want = g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning);
const bool byMode = LasersByMode();
for (auto &[i, s] : g_screens) {
const bool want = byMode || (s.visible && g_tick < s.aimUntil);
if (s.lasers == want) continue;
s.lasers = want;
vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, want);
}
}
// The reset button in a VR game (or the dashboard mode), where the screens don't keep
// SteamVR's laser mouse on: aiming a hand controller at it turns the laser mouse on for that
// button alone, so the trigger clicks it, and the game gets the controllers back
// kResetLinger ticks after the aim leaves it (a wider zone than the one that turns it on).
void UpdateResetLaser(Screen &s) {
const bool want = s.resetButton != vr::k_ulOverlayHandleInvalid && s.visible && !s.lasers &&
g_tick < s.resetNearUntil;
if (want == s.resetLaser) return;
s.resetLaser = want;
vr::VROverlay()->SetOverlayFlag(s.resetButton, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, want);
if (keyboard::Shown()) keyboard::SetLasers(byMode || g_tick < g_keyboardAimUntil);
}
// The distance from a laser's line to a point ahead of it, or -1 when it's behind.
@@ -947,11 +945,10 @@ double RayDistance(const Mat &d, const Mat &c) {
// kControlsLinger ticks after it leaves, and while in use.
void UpdateControls() {
std::vector<Mat> lasers;
std::vector<bool> hands; // the laser is a hand controller's (not the 3D mouse's)
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
Mat d;
if (vr::VRSystem()->GetTrackedDeviceClass(i) == vr::TrackedDeviceClass_Controller && LaserPose(i, &d))
lasers.push_back(d), hands.push_back(IsHandController(i));
lasers.push_back(d);
}
for (auto &[i, s] : g_screens) {
Mat p;
@@ -977,16 +974,7 @@ void UpdateControls() {
break;
}
}
if (s.resetButton != vr::k_ulOverlayHandleInvalid && !s.lasers) {
const Mat c = Mul(p, offsets[6]);
const double aim = s.grip * (s.resetLaser ? 2.0 : 0.9);
for (size_t k = 0; k < lasers.size(); ++k) {
const double r = RayDistance(lasers[k], c);
if (hands[k] && r >= 0 && r <= aim) s.resetNearUntil = g_tick + kResetLinger;
}
}
}
UpdateResetLaser(s);
const bool inUse = s.drag != Drag::None || std::any_of(std::begin(s.hover), std::end(s.hover), [](bool h) { return h; });
const bool want = s.visible && (inUse || g_tick < s.nearUntil);
// The controls stay shown while their screen is, just fully transparent when not
@@ -1020,19 +1008,21 @@ void SendFloat(const std::string &msg) {
std::printf("to ft-floatd: %s\n", msg.c_str());
}
// A new panel: the pointer helper reads SteamVR's list of panels only every 20 s, so it's told
// at once ("overlay <key>"), or the mouse couldn't click a menu until then.
void AnnounceOverlay(const char *key) {
// To the pointer helper (@ft_pointer_helper), from an unbound socket.
void SendPointer(const std::string &msg) {
static const int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
sockaddr_un addr{};
addr.sun_family = AF_UNIX;
const char name[] = "ft_pointer_helper";
std::memcpy(addr.sun_path + 1, name, sizeof name - 1);
const std::string msg = std::string("overlay ") + key;
sendto(fd, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<sockaddr *>(&addr),
socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1));
}
// A new panel: the pointer helper reads SteamVR's list of panels only every 20 s, so it's told
// at once ("overlay <key>"), or the mouse couldn't click a menu until then.
void AnnounceOverlay(const char *key) { SendPointer(std::string("overlay ") + key); }
// Where a device's ray meets the screen's plane, in the screen's x (right) and y (up),
// metres from its centre.
bool RayOnPlane(const Mat &p, const Mat &d, double *x, double *y) {
@@ -1419,6 +1409,7 @@ struct Press {
double x = 0, y = 0; // ...and where on it, in buffer pixels
double distance = 1; // from the laser's start to the last panel it met
long upAt = -1; // the pointer helper saw left come up: release it at this tick
int64_t idleSince = -1; // the pressing controller has held nothing since (ms, ReleaseStuck)
};
Press g_press;
vr::VROverlayHandle_t g_catcher = vr::k_ulOverlayHandleInvalid;
@@ -1455,6 +1446,47 @@ void ReleaseAway(uint32_t button, void (*handle)(const struct ft_event *, void *
handle(&e, data);
}
// Whether a hand controller holds anything (a button down): 1 yes, 0 no, -1 unknown. SteamVR
// answers overlay apps only while a VR game runs (checked 2026-10-04); outside games it can't
// say. The Frame controller's axes have no types, so the buttons are all there is.
int ControllerHolds(vr::TrackedDeviceIndex_t dev) {
vr::VRControllerState_t st{};
if (!vr::VRSystem()->GetControllerState(dev, &st, sizeof st)) return -1;
return st.ulButtonPressed ? 1 : 0;
}
// Releases SteamVR never sends. Pausing, or hiding the screen a button went down on, takes the
// laser off it mid-click (the pause gesture's second thumbstick click does that), and SteamVR's
// laser mouse then forgets the button: no release comes, the catcher stayed up, and the game
// never got its controllers back (2026-10-04). So a held button is released here when it has
// no visible screen to come up on, or when its hand controller has held nothing for kStuckMs
// (only known during VR games, where the lost release took the game's controllers).
// The pointer helper's virtual controller has its own word for that ("up").
constexpr int64_t kStuckMs = 1000;
void ReleaseStuck(void (*handle)(const struct ft_event *, void *), void *data) {
if (!g_press.buttons) {
g_press.idleSince = -1;
return;
}
const char *why = nullptr;
const auto it = g_screens.find(g_press.screen);
if (g_paused) why = "paused";
else if (g_press.screen >= 0 && (it == g_screens.end() || !it->second.visible)) why = "its screen hid";
else if (IsHandController(g_press.device)) {
const int64_t now = NowMs();
if (ControllerHolds(g_press.device) != 0) g_press.idleSince = -1;
else if (g_press.idleSince < 0) g_press.idleSince = now;
else if (now - g_press.idleSince >= kStuckMs) why = "the controller holds nothing";
}
if (!why) return;
std::printf("a held button can't come up on a screen (%s): released\n", why);
const vr::TrackedDeviceIndex_t dev = g_press.device;
for (uint32_t b = 0; b < 32; ++b)
if (g_press.buttons & (1u << b)) ReleaseAway(BTN_LEFT + b, handle, data);
g_press.buttons = 0, g_press.upAt = -1, g_press.idleSince = -1;
EndDragsBy(dev);
}
void ShowCatcher(bool on) {
if (on == g_catcherShown || g_catcher == vr::k_ulOverlayHandleInvalid) return;
g_catcherShown = on;
@@ -1500,6 +1532,137 @@ Screen *Find(int one_based) {
return it == g_screens.end() ? nullptr : &it->second;
}
// ---------------------------------------------------------------- the lazy susan
// "spin next|prev|<degrees>": the panels in the room (screens and floating windows, not
// pinned ones) turn together about a vertical axis through your head, so the next panel to
// your right (next) or left (prev) glides to straight ahead, or the ring turns by that many
// degrees (positive turns it left, like next). Their arrangement stays as it is: the room
// turns instead of you. A spin that arrives during one adds to it, from where the panels are
// headed, so quick taps carry on smoothly. Grabbing a panel, or a command that places it,
// takes it out of the spin where it is. The 3D mouse's pointer goes to straight ahead.
constexpr double kSpinSeconds = 0.3; // how long a spin takes
constexpr double kSpinAhead = 8; // degrees: a panel this near straight ahead is the current one
constexpr double kSpinFocus = 30; // degrees: when a spin settles, the panel this near ahead gets typing
struct Spin {
bool on = false;
double cx = 0, cz = 0; // the axis
double from = 0, to = 0; // radians, turned from the poses in base (positive: to the left)
Clock::time_point start;
std::map<int, Mat> base; // index -> its pose before the spin
int front = -1; // settled: this panel came to the front (ft_vr_poll reports it)
} g_spin;
// p turned a radians about the vertical axis through (cx, cz); positive turns it to the left.
Mat Turned(const Mat &p, double a, double cx, double cz) {
const double c = std::cos(a), s = std::sin(a);
Mat r = Identity();
r.m[0][0] = float(c), r.m[0][2] = float(s);
r.m[2][0] = float(-s), r.m[2][2] = float(c);
r.m[0][3] = float(cx - c * cx - s * cz);
r.m[2][3] = float(cz + s * cx - c * cz);
return Mul(r, p);
}
bool Spinnable(const Screen &s) { return s.pinned == kNone && (!s.floating || s.floatOn) && s.drag == Drag::None; }
double SpinNow() {
if (!g_spin.on) return g_spin.to;
const double t = std::min(1.0, std::chrono::duration<double>(Clock::now() - g_spin.start).count() / kSpinSeconds);
return g_spin.from + (g_spin.to - g_spin.from) * t * t * (3 - 2 * t);
}
void UpdateSpin() {
if (!g_spin.on) return;
const double a = SpinNow();
const bool done = Clock::now() - g_spin.start >= std::chrono::duration<double>(kSpinSeconds);
for (auto it = g_spin.base.begin(); it != g_spin.base.end();) {
auto s = g_screens.find(it->first);
if (s == g_screens.end() || !Spinnable(s->second)) {
it = g_spin.base.erase(it); // grabbed, pinned, or gone: it stays where it is now
continue;
}
SetAbsolute(s->second, Turned(it->second, a, g_spin.cx, g_spin.cz));
++it;
}
if (done) {
// The panel now nearest straight ahead (of where you face) gets typing and the active window.
Mat head;
double nearest = kSpinFocus;
if (DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head)) {
for (const auto &[i, p] : g_spin.base) {
const auto it = g_screens.find(i);
Mat q;
if (it == g_screens.end() || !it->second.visible || !ScreenPose(it->second, &q)) continue;
double f[3] = {-head.m[0][2], 0, -head.m[2][2]};
double d[3] = {q.m[0][3] - head.m[0][3], 0, q.m[2][3] - head.m[2][3]};
const double fl = std::sqrt(Dot3(f, f)), dl = std::sqrt(Dot3(d, d));
if (fl < 1e-6 || dl < 1e-6) continue;
const double a = std::acos(std::clamp(Dot3(f, d) / (fl * dl), -1.0, 1.0)) * 180 / M_PI;
if (a < nearest) nearest = a, g_spin.front = i;
}
}
g_spin.on = false;
g_spin.base.clear();
ArrangeDesktopSoon(); // KWin's outputs follow where the screens are now
}
}
void SpinCommand(const char *arg, char *reply, int size) {
Mat head;
if (!DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head))
return (void)std::snprintf(reply, size, "error no head pose (headset off?)");
if (g_spin.on) {
g_spin.from = SpinNow(); // carry on from where the panels are now
} else {
g_spin.base.clear();
for (auto &[i, s] : g_screens) {
Mat p;
if (Spinnable(s) && ScreenPose(s, &p)) g_spin.base[i] = p;
}
g_spin.cx = head.m[0][3], g_spin.cz = head.m[2][3];
g_spin.from = g_spin.to = 0;
}
if (g_spin.base.empty()) return (void)std::snprintf(reply, size, "error nothing to spin");
double turn; // degrees, positive to the left
const bool next = !std::strcmp(arg, "next");
if (next || !std::strcmp(arg, "prev")) {
// Each visible panel's bearing from where you face, to the right positive, as it will
// be when the spin so far ends; the nearest one past straight ahead comes to the front.
double fx = -head.m[0][2], fz = -head.m[2][2];
const double n = std::sqrt(fx * fx + fz * fz) + 1e-9;
fx /= n, fz /= n;
const double rx = -fz, rz = fx;
double best = 0;
bool found = false;
for (const auto &[i, p] : g_spin.base) {
const auto s = g_screens.find(i);
if (s == g_screens.end() || !s->second.visible) continue;
const Mat q = Turned(p, g_spin.to, g_spin.cx, g_spin.cz);
const double dx = q.m[0][3] - head.m[0][3], dz = q.m[2][3] - head.m[2][3];
double a = std::atan2(dx * rx + dz * rz, dx * fx + dz * fz) * 180 / M_PI;
if (!next) a = -a;
if (a <= kSpinAhead) a += 360;
if (!found || a < best) best = a, found = true;
}
if (!found || best >= 360 - kSpinAhead) {
if (!g_spin.on) g_spin.base.clear();
return (void)std::snprintf(reply, size, "ok 0 (no other panel)");
}
if (best > 180) best -= 360; // the short way round
turn = next ? best : -best;
} else {
char *end;
turn = std::strtod(arg, &end);
if (end == arg || *end) return (void)std::snprintf(reply, size, "error spin next|prev|<degrees>");
}
g_spin.to += turn * M_PI / 180;
g_spin.start = Clock::now();
g_spin.on = true;
SendPointer("recenter"); // the 3D mouse's pointer stays in front of you, on what comes there
std::snprintf(reply, size, "ok %.1f", turn);
}
uint32_t LinuxButton(uint32_t vrButton) {
switch (vrButton) {
case vr::VRMouseButton_Right: return BTN_RIGHT;
@@ -1514,6 +1677,81 @@ void ReleaseAwayBy(vr::TrackedDeviceIndex_t dev, uint32_t vrButton, void (*handl
if (g_press.buttons && dev == g_press.device) ReleaseAway(LinuxButton(vrButton), handle, data);
}
// Where a laser meets a panel's surface, in the panel's u (metres along it from the centre,
// along the arc when curved) and v (up). OpenVR curves a screen into a cylinder toward its
// front, centred `curve` metres in front of it (see OnSurface).
bool RayOnSurface(const Screen &s, const Mat &p, const Mat &laser, double *u, double *v) {
const Mat inv = Inverse(p);
const double o[3] = {inv.m[0][0] * laser.m[0][3] + inv.m[0][1] * laser.m[1][3] + inv.m[0][2] * laser.m[2][3] + inv.m[0][3],
inv.m[1][0] * laser.m[0][3] + inv.m[1][1] * laser.m[1][3] + inv.m[1][2] * laser.m[2][3] + inv.m[1][3],
inv.m[2][0] * laser.m[0][3] + inv.m[2][1] * laser.m[1][3] + inv.m[2][2] * laser.m[2][3] + inv.m[2][3]};
double d[3];
for (int i = 0; i < 3; ++i) d[i] = -(inv.m[i][0] * laser.m[0][2] + inv.m[i][1] * laser.m[1][2] + inv.m[i][2] * laser.m[2][2]);
if (s.curve <= 0) {
if (std::fabs(d[2]) < 1e-6) return false;
const double t = -o[2] / d[2];
if (t <= 0) return false;
*u = o[0] + d[0] * t, *v = o[1] + d[1] * t;
return true;
}
// x^2 + (z - r)^2 = r^2, on the screen's side of the axis (z < r).
const double r = s.curve, oz = o[2] - r;
const double a = d[0] * d[0] + d[2] * d[2], b = 2 * (o[0] * d[0] + oz * d[2]), c = o[0] * o[0] + oz * oz - r * r;
const double disc = b * b - 4 * a * c;
if (a < 1e-9 || disc < 0) return false;
for (double t : {(-b - std::sqrt(disc)) / (2 * a), (-b + std::sqrt(disc)) / (2 * a)}) {
const double x = o[0] + d[0] * t, z = oz + d[2] * t;
if (t <= 0 || z >= 0) continue;
*u = r * std::atan2(x, -z), *v = o[1] + d[1] * t;
return true;
}
return false;
}
// Where the mode leaves the controllers to a VR game (see the top): a hand controller
// pointing at a panel, its controls, or a floating window's popups keeps that panel's laser
// on (UpdateLasers) until kAimLinger ticks after it points away, like SteamVR's own floating
// windows. Leaving takes a wider margin than arriving, and a drag or a held button keeps it
// on. The keyboard is one overlay, so SteamVR's own intersection test does there.
void UpdateAim() {
if (LasersByMode()) return;
std::vector<Mat> lasers;
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
Mat d;
if (IsHandController(i) && LaserPose(i, &d)) lasers.push_back(d);
}
for (auto &[index, s] : g_screens) {
Mat p;
if (!s.visible || !ScreenPose(s, &p)) continue;
if (s.drag != Drag::None || (g_press.buttons && g_press.screen == index)) {
s.aimUntil = g_tick + kAimLinger;
continue;
}
const double m = s.grip * (g_tick < s.aimUntil ? 2.0 : 0.25), h = s.heightMetres();
// The panel and its controls: the bar row under it, the resize tab off its corner.
const double halfW = std::max(s.metres / 2 + s.grip, s.chrome / 2 + s.chrome * 0.12 + s.grip * 2) + m;
const double top = h / 2 + m, bottom = std::min(BarY(s) - s.grip, -(h / 2 + s.grip)) - m;
for (const Mat &l : lasers) {
double u, v;
if (!RayOnSurface(s, p, l, &u, &v)) continue;
bool on = std::fabs(u) <= halfW && v <= top && v >= bottom;
for (const auto &[k, sub] : s.subs) {
if (on || s.cropW <= 0) break;
const double su = (sub.x + sub.w / 2.0 - (s.cropX + s.cropW / 2.0)) * s.mpp;
const double sv = -(sub.y + sub.h / 2.0 - (s.cropY + s.cropH / 2.0)) * s.mpp;
on = std::fabs(u - su) <= sub.w * s.mpp / 2 + m && std::fabs(v - sv) <= sub.h * s.mpp / 2 + m;
}
if (on) {
s.aimUntil = g_tick + kAimLinger;
break;
}
}
}
if (keyboard::Shown())
for (const Mat &l : lasers)
if (keyboard::Aimed(l)) g_keyboardAimUntil = g_tick + kAimLinger;
}
const char *LasersName() {
switch (g_lasers) {
case Lasers::Always: return "always";
@@ -2036,6 +2274,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
if (ev.eventType == vr::VREvent_MouseButtonUp)
ReleaseAwayBy(ev.trackedDeviceIndex, ev.data.mouse.button, handle, data);
if (g_press.upAt >= 0 && g_tick >= g_press.upAt) ReleaseAway(BTN_LEFT, handle, data);
ReleaseStuck(handle, data);
RefreshChrome();
while (vr::VRSystem()->PollNextEvent(&ev, sizeof ev)) {
if (ev.eventType == vr::VREvent_Quit) {
@@ -2071,10 +2310,19 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
handle(&e, data);
}
++g_tick;
UpdateSpin();
if (g_spin.front >= 0) {
ft_event e{};
e.type = FT_FRONT;
e.screen = g_spin.front;
g_spin.front = -1;
handle(&e, data);
}
UpdateGame();
UpdateArrange();
UpdateVisibility();
UpdateAttention();
UpdateAim();
UpdateLasers();
UpdateControls();
UpdateGuides();
@@ -2101,7 +2349,7 @@ bool ft_vr_keyboard_show(int index) {
const double fx = -head.m[0][2], fz = -head.m[2][2], n = std::sqrt(fx * fx + fz * fz) + 1e-9;
const double at[3] = {head.m[0][3] + fx / n * kKeyboardAhead, head.m[1][3] - kKeyboardBelow,
head.m[2][3] + fz / n * kKeyboardAhead};
keyboard::SetLasers(g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning));
keyboard::SetLasers(LasersByMode());
g_steamInFront = SteamInFront();
if (g_steamInFront) {
g_asidePose = FacingPose(at, head);
@@ -2146,6 +2394,8 @@ void ft_vr_keyboard_hide(void) {
// ~100 ms (it landed on something else), KWin gets it anyway
// state -> "ok <mode> <manual 0|1> <wrist deg> <gesture hand> <gesture deg>
// <controllers> <game running 0|1> <ingames>"
// spin next|prev|<degrees> turn every panel in the room about your head (see the lazy
// susan) -> "ok <degrees turned>"
// cutouts on|off|state hand cutouts (see handcut.h) -> "ok <on|off> <ready|idle|unavailable>
// <last composite ms> ms, predict <on|off> lead <ms> ms"
// cutouts predict on|off move the hands ahead along their velocity (on by default)
@@ -2183,6 +2433,7 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
Screen *s = Find(n);
if (!s) return (void)std::snprintf(reply, size, "error no screen %d", n);
EndDrag(*s);
g_spin.base.erase(n - 1);
SetAbsolute(*s, PanelPose(x, y, z, yaw, pitch, roll));
std::snprintf(reply, size, "ok");
} else if (std::sscanf(cmd, "width %d %lf", &n, &w) == 2) {
@@ -2349,6 +2600,7 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
Mat m{};
for (int k = 0; k < 12; ++k) m.m[k / 4][k % 4] = r[k];
EndDrag(*s);
g_spin.base.erase(n - 1);
SetAbsolute(*s, m);
std::snprintf(reply, size, "ok");
} else if (int k0; std::sscanf(cmd, "sub %d %d %d %d %d %d", &n, &k0, &x0, &y0, &w0, &h0) == 6 ||
@@ -2376,6 +2628,8 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
if ((g_press.buttons & ButtonBit(BTN_LEFT)) && g_press.device != kNone && !IsHandController(g_press.device))
g_press.upAt = g_tick + 9;
std::snprintf(reply, size, "ok");
} else if (std::sscanf(cmd, "spin %15s", word) == 1) {
SpinCommand(word, reply, size);
} else if (std::strncmp(cmd, "state", 5) == 0) {
std::snprintf(reply, size, "ok %s %d %.0f %s %.0f %s %d %s", ModeName(), g_manual ? 1 : 0, g_wristAngle,
g_gestureHand.c_str(), g_gestureAngle, LasersName(), g_gameRunning ? 1 : 0,
+2 -1
View File
@@ -17,7 +17,8 @@ struct ft_dmabuf {
int fd[4];
};
enum ft_event_type { FT_MOTION, FT_BUTTON, FT_SCROLL, FT_LEAVE, FT_QUIT, FT_KEY, FT_KEYBOARD_CLOSED };
// FT_FRONT: a spin (the lazy susan) brought panel `screen` to straight ahead: typing goes there.
enum ft_event_type { FT_MOTION, FT_BUTTON, FT_SCROLL, FT_LEAVE, FT_QUIT, FT_KEY, FT_KEYBOARD_CLOSED, FT_FRONT };
struct ft_event {
enum ft_event_type type;
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Update the lines of ~/.config/frametop.conf that still read exactly as an older
# frametop.conf.example wrote them. A line you changed stays as it is. install.sh and
# hands/rec/install.sh run this on every install and update.
# HANDS_SWAP_SIDES=0: the example's value until 2026-10-05. It made ft-hands keep ft-camd's side
# camera names even when they're backwards (some SteamVR restarts swap them), so hands landed
# beside their cutouts and the hand recorder labelled the side cameras wrong. auto tells from
# the hands.
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
. "$root/scripts/_env.sh"
on_frame_script <<'EOF'
f=~/.config/frametop.conf
[ -f "$f" ] || exit 0
old="HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)"
new="HANDS_SWAP_SIDES=auto # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them"
if grep -qxF "$old" "$f"; then
tmp=$(mktemp)
awk -v old="$old" -v new="$new" '$0 == old { print new; next } { print }' "$f" > "$tmp"
cat "$tmp" > "$f" # in place: the file keeps its owner and mode
rm -f "$tmp"
echo "settings: HANDS_SWAP_SIDES=0 (the old default) is now auto: hand tracking tells the side cameras apart itself"
fi
EOF
+4
View File
@@ -36,6 +36,10 @@ check "distrobox" on_frame 'test -x ~/.local/bin/distrobox && ~/.local/bin/distr
check "container $FRAME_BOX" on_frame "podman ps -a --filter name=^$FRAME_BOX\$ --format '{{.Image}} {{.Status}}' | grep ."
check "repo on the Frame" on_frame 'pwd'
check "free space in ~" on_frame "df -h ~ | awk 'NR==2{print \$4\" free\"}'"
# A taskbar saved on a screen the desktop doesn't have is hidden; the desktop's next start
# moves it to the first screen (session/fix-panels.py).
check "taskbar on a screen" on_frame 'set -o pipefail; [ -f session/fix-panels.py ] || { echo "not checked (older checkout)"; exit 0; }
python3 session/fix-panels.py --check | paste -sd ";" | sed "s/;/; /g"'
echo "what Frametop needs from SteamOS:"
if [ "$FRAME_LOCAL" = 1 ]; then
+43 -1
View File
@@ -26,7 +26,8 @@ for u in frametop-input-relay frametop-pointer frametop-power; do
done
echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
# SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md).
XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
section "What Frametop needs from SteamOS (scripts/update-check.py)"
python3 "$repo/scripts/update-check.py" 2>&1 || true
@@ -42,12 +43,53 @@ for i, s in enumerate(d.get("screens", []), 1):
print("visibility:", d.get("visibility"))
PY
section "Plasma panels and outputs"
python3 "$repo/session/fix-panels.py" --check 2>&1 || true
python3 - "$repo" <<'PY' 2>&1 || true
import json, subprocess, sys
sys.path.insert(0, sys.argv[1] + "/layout")
import ft_layout
env = ft_layout.nested_env()
if not env:
sys.exit(print("desktop not running: no live outputs or panels"))
outs = json.loads(subprocess.run(["kscreen-doctor", "-j"], capture_output=True, text=True, env=env, timeout=10).stdout or "{}")
for o in outs.get("outputs", []):
size = o.get("size") or {}
print(f"output {o.get('name')}: enabled={o.get('enabled')} priority={o.get('priority')} {size.get('width')}x{size.get('height')}")
js = "print(JSON.stringify(panels().map(p => ({id: p.id, screen: p.screen, location: p.location}))))"
r = subprocess.run(["qdbus6", "org.kde.plasmashell", "/PlasmaShell", "org.kde.PlasmaShell.evaluateScript", js],
capture_output=True, text=True, env=env, timeout=10)
print("live panels:", (r.stdout or r.stderr).strip())
PY
section "Input relay (last 60 lines)"
journalctl --user -u frametop-input-relay -n 60 --no-pager -o short 2>/dev/null
section "Pointer helper (last 60 lines)"
journalctl --user -u frametop-pointer -n 60 --no-pager -o short 2>/dev/null
section "Power service (last 30 lines)"
journalctl --user -u frametop-power -n 30 --no-pager -o short 2>/dev/null
section "Gaze"
echo "frametop-gaze: $(systemctl --user is-enabled frametop-gaze 2>/dev/null) / $(systemctl --user is-active frametop-gaze 2>/dev/null)"
echo "frametop-eyegrab (our eye tracker's frame grabber): $(systemctl is-enabled frametop-eyegrab 2>/dev/null) / $(systemctl is-active frametop-eyegrab 2>/dev/null)"
for f in calibration.json eyes/calibration.json; do # SteamVR's correction, our tracker's calibration
p=~/.local/state/frametop/gaze/$f
echo "$f: $([ -f "$p" ] && date -r "$p" '+%F %T' || echo none)"
done
python3 - "$repo" <<'PY' 2>&1 || true
import json, subprocess, sys
out = subprocess.run([sys.executable, sys.argv[1] + "/gaze/ft-gazectl", "status"], capture_output=True, text=True, timeout=10)
text = (out.stdout or out.stderr).strip()
try:
print("status:", json.dumps(json.loads(text), separators=(",", ":")))
except ValueError:
print("status:", text or "no answer from the gaze service")
PY
section "Gaze checks and calibration dots (last 10)"
tail -n 10 ~/.local/state/frametop/gaze/checks.jsonl 2>/dev/null | cut -c1-400 || true
section "Gaze service (last 60 lines, podman's left out)"
journalctl --user -u frametop-gaze -n 300 --no-pager -o short 2>/dev/null | grep -v ' podman\[' | tail -n 60
section "Our eye tracker's frame grabber (last 20 lines)"
journalctl -u frametop-eyegrab -n 20 --no-pager -o short 2>/dev/null
for f in /tmp/frametop-session.log /tmp/frametop-screens.log /tmp/frametop-layout.log; do
section "$f (last 60 lines)"
tail -n 60 "$f" 2>/dev/null || echo "missing"
+23
View File
@@ -35,6 +35,9 @@ KNOWN_GOOD = os.path.join(HOME, ".local/state/frametop/known-good.json")
STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64"
LAUNCHER = os.path.join(HOME, ".local/share/applications/deckard-nested-desktop.desktop")
VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath")
# The Frametop desktop has its own XDG_CONFIG_HOME (session/frametop-session.sh). An install from a
# terminal there, before pointer/driver/install.sh set SteamVR's, left vrpathreg's registry here.
STRAY_VRPATHS = os.path.join(HOME, ".config/frametop/openvr/openvrpaths.vrpath")
BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness" # as in power/ft-powerd.cpp
EYE_MMAP = "/dev/shm/eye-server.mmap"
VRSERVER = "http://127.0.0.1:27062" # its web socket: input/vrws.py
@@ -49,6 +52,8 @@ PACKAGES = {
"after a desktop restart; floating a window; no blur behind the taskbar's menus",
"plasma-workspace": "the taskbar and panels after a desktop restart; no DiscoverNotifier or "
"ibus-daemon inside the desktop",
"at-spi2-core": "an AT-SPI-aware app appears on the nested desktop's accessibility bus; "
"the registry stops and comes back after a desktop restart",
"gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked",
"bluez": "a Bluetooth mouse reconnecting after it sleeps",
}
@@ -162,6 +167,14 @@ def check_host():
("/usr/bin/kwin_wayland_wrapper", "FAIL", "the desktop can't start KWin"),
("/usr/bin/startplasma-wayland", "FAIL", "the desktop can't start Plasma"),
("/usr/bin/dbus-run-session", "FAIL", "the desktop can't start its session bus"),
("/usr/bin/python3", "warn", "nested accessibility can't run its startup helper"),
("/usr/bin/gdbus", "warn", "nested accessibility can't discover or check its bus"),
("/usr/lib/at-spi-bus-launcher", "warn", "nested accessibility can't start its bus"),
("/usr/lib/at-spi2-registryd", "warn", "nested accessibility has no fallback registry"),
("/usr/share/dbus-1/services/org.a11y.Bus.service", "warn",
"nested accessibility can't activate its bus"),
("/usr/share/dbus-1/accessibility-services/org.a11y.atspi.Registry.service", "warn",
"nested accessibility can't activate its registry natively"),
(f"{STEAMVR_BIN}/vrcmd", "FAIL", "the 3D mouse can't find panels"),
(f"{STEAMVR_BIN}/vrpathreg", "warn", "the pointer driver can't be installed or removed"),
("/usr/share/deckard/mesavars.sh", "warn", "the desktop starts without SteamOS's Mesa settings"),
@@ -217,6 +230,14 @@ def check_host():
else:
report("FAIL", "pointer driver", "not registered with SteamVR; run pointer/driver/install.sh install, "
"then restart SteamVR")
try:
with open(STRAY_VRPATHS) as f:
stray = json.load(f).get("runtime") is None
except (OSError, ValueError, AttributeError):
stray = False
if stray:
report("warn", "OpenVR path registry", f"{STRAY_VRPATHS} has no SteamVR in it, and hides SteamVR from "
"OpenVR programs started in the Frametop desktop; pointer/driver/install.sh install removes it")
session = launcher_session()
if session is None:
@@ -464,6 +485,8 @@ def main():
# A terminal in the desktop has its session's runtime dir and bus; systemctl needs the real ones.
os.environ["XDG_RUNTIME_DIR"] = f"/run/user/{UID}"
os.environ["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path=/run/user/{UID}/bus"
# And its own config folder, where SteamVR's path registry isn't: OpenVR and vrcmd need ours.
os.environ["XDG_CONFIG_HOME"] = os.path.join(HOME, ".config")
versions = current_versions()
check_versions(versions)
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env python3
"""Bring back a taskbar that Plasma saved against a screen this desktop doesn't have.
Plasma 6.2.5 ties each panel to a screen number (lastScreen in its containment), and the
numbers rank the enabled outputs by priority: 0 is the primary screen. A panel whose
number is past the screen count gets no view and stays hidden; Plasma never moves it,
even on a later start (sanitizeScreenLayout() only remaps a panel whose number has no
desktop, and the Frametop desktop keeps a desktop for every spare output it has seen).
So a taskbar saved on a spare output (issue #18), or on a screen that a smaller layout
dropped, was lost until the config was deleted.
The session runs this before Plasma starts, so nothing races Plasma for the file. Each
panel numbered at or past the screen count moves to screen 0, with its system tray's
containment, and keeps its widgets and settings. A panel is left where it is when screen
0 already has a panel on that edge: it comes back by itself if the screens do. The file
is backed up once per repair (<file>.ft-bak), and the changes go through kwriteconfig6.
fix-panels.py [--screens N] [--file APPLETSRC] [--check]
--screens the desktop's screen count (default: the configured layout's)
--file default: $XDG_CONFIG_HOME/plasma-org.kde.plasma.desktop-appletsrc, with
XDG_CONFIG_HOME defaulting to the Frametop desktop's ~/.config/frametop
--check change nothing; list the panels, and exit 1 if one is lost
"""
import argparse
import os
import re
import shutil
import subprocess
import sys
EDGES = {3: "top", 4: "bottom", 5: "left", 6: "right"} # Plasma::Types::Location
SYSTRAY = "org.kde.plasma.private.systemtray"
GROUP = re.compile(r"\[([^\]]*)\]")
def parse(text):
"""KConfig text -> {(group, subgroup, ...): {key: value}}."""
groups, cur = {}, None
for line in text.splitlines():
line = line.strip()
if line.startswith("["):
cur = tuple(GROUP.findall(line))
groups.setdefault(cur, {})
elif cur is not None and "=" in line and not line.startswith("#"):
k, v = line.split("=", 1)
groups[cur][re.sub(r"\[\$.*\]$", "", k.strip())] = v.strip()
return groups
def number(v, default=-1):
try:
return int(v)
except (TypeError, ValueError):
return default
def panels(groups):
"""The panels, by containment id: location, lastScreen, and the ids of their system
trays' own containments (which sit on the same edge and screen as the panel)."""
trays, found = {}, {}
for g, keys in groups.items():
if len(g) == 5 and g[0] == "Containments" and g[2] == "Applets" and g[4] == "Configuration":
tray = keys.get("SystrayContainmentId")
if tray:
trays.setdefault(g[1], []).append(tray)
owned = {t for ts in trays.values() for t in ts}
for g, keys in groups.items():
if len(g) != 2 or g[0] != "Containments" or number(g[1]) <= 0 or g[1] in owned:
continue
loc = number(keys.get("location"), 0)
if loc in EDGES and keys.get("plugin") != SYSTRAY:
found[g[1]] = {"location": loc, "screen": number(keys.get("lastScreen")),
"plugin": keys.get("plugin", "?"), "trays": trays.get(g[1], [])}
return dict(sorted(found.items(), key=lambda kv: number(kv[0])))
def plan(groups, screens):
"""(moves, kept): moves are (panel id, from screen, containment ids to put on screen 0);
kept are (panel id, from screen, why) for lost panels left alone."""
found = panels(groups)
taken = {(p["screen"], p["location"]) for p in found.values() if 0 <= p["screen"] < screens}
moves, kept = [], []
for pid, p in found.items():
if p["screen"] < screens:
continue # on a screen this desktop has (Plasma puts -1 on the first one itself)
if (0, p["location"]) in taken:
kept.append((pid, p["screen"], f"the first screen already has a {EDGES[p['location']]} panel"))
continue
taken.add((0, p["location"]))
moves.append((pid, p["screen"], [pid, *p["trays"]]))
return moves, kept
def default_screens():
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "layout"))
import ft_layout
return ft_layout.screen_count()
def main(argv):
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--screens", type=int)
ap.add_argument("--file")
ap.add_argument("--check", action="store_true")
a = ap.parse_args(argv)
config = os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config/frametop")
path = a.file or os.path.join(config, "plasma-org.kde.plasma.desktop-appletsrc")
screens = a.screens if a.screens else default_screens()
try:
with open(path) as f:
groups = parse(f.read())
except FileNotFoundError:
if a.check:
print("no Plasma config yet (Plasma makes the default taskbar)")
return 0
moves, kept = plan(groups, screens)
if a.check:
found = panels(groups)
for pid, p in found.items():
lost = " (lost: no such screen)" if p["screen"] >= screens else ""
print(f"panel {pid}: screen {p['screen']}, {EDGES[p['location']]}{lost}")
if not found:
print("no panels")
print(f"{screens} screen(s)")
return 1 if moves or kept else 0
if moves:
shutil.copy2(path, path + ".ft-bak")
for pid, was, ids in moves:
for cid in ids:
subprocess.run(["kwriteconfig6", "--file", os.path.abspath(path), "--group", "Containments",
"--group", cid, "--key", "lastScreen", "0"], check=True)
print(f"frametop: panel {pid} was saved on screen {was}, which this desktop doesn't have "
f"({screens} screen(s)); moved it to the first screen (backup: {path}.ft-bak)", file=sys.stderr)
for pid, was, why in kept:
print(f"frametop: panel {pid} is saved on screen {was}, which this desktop doesn't have; "
f"left there: {why}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+21 -1
View File
@@ -29,7 +29,7 @@ for var in $(compgen -e); do
case $var in
LD_LIBRARY_PATH | LD_PRELOAD | STEAM_* | Steam* | SRT_* | PRESSURE_VESSEL_* | MANGOHUD_* | \
ENABLE_VK_LAYER_VALVE_steam_overlay_* | STEAMVIDEOTOKEN | QT_IM_MODULE | GTK_IM_MODULE | \
XMODIFIERS) unset "$var" ;;
XMODIFIERS | AT_SPI_BUS_ADDRESS) unset "$var" ;;
esac
done
@@ -190,6 +190,21 @@ if [ "$remote" = 1 ]; then
"$here/remote-ctl.sh" start
fi
# AT-SPI: start the registry inside Plasma's autostart, after KWin has published the
# nested displays. Starting it before startplasma could bind to the host's X display.
# This config belongs only to Frametop; the host desktop's autostart is unchanged.
autostart=$XDG_CONFIG_HOME/autostart/frametop-atspi.desktop
mkdir -p "$(dirname "$autostart")"
cat > "$autostart" <<EOF
[Desktop Entry]
Type=Application
Name=Frametop accessibility
Exec="$here/ft-atspi"
X-KDE-autostart-phase=2
OnlyShowIn=KDE;
NoDisplay=true
EOF
# ft-floatd (floating windows) runs inside the Plasma session, on its D-Bus: started from
# the session's autostart, which only this desktop reads (XDG_CONFIG_HOME above).
autostart=$XDG_CONFIG_HOME/autostart/frametop-floatd.desktop
@@ -272,4 +287,9 @@ fi
# with both, apps would open twice.
kwriteconfig6 --file "$XDG_CONFIG_HOME/ksmserverrc" --group General --key loginMode emptySession
# Plasma keeps a panel on a screen number, and never moves one whose screen this desktop
# doesn't have, like a spare output or a screen a smaller layout dropped. Put such a panel
# back on the first (primary) screen before Plasma reads the file (session/fix-panels.py).
python3 "$here/fix-panels.py" --screens "$screens" || true
dbus-run-session startplasma-wayland
+1 -1
View File
@@ -40,7 +40,7 @@ POINTER_HEAD_DEADZONE=0.5 # keyboard clicks at the gaze (Meta+J, Meta+K): degre
POINTER_KEY_TAP=0.25 # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
HANDS_SWAP_SIDES=auto # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them
HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on
HANDS_CAMERAS=auto # hand tracking: auto (by the light) | mono (the IR cameras; also keeps ft-camd off the colour ones) | color | all
HANDS_BRIGHT=all # hand tracking, auto: the cameras in bright light, all | color
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/python3
"""Start AT-SPI on this nested desktop's live bus, after KWin has set its displays."""
import ast
import os
import signal
import subprocess
import sys
import time
DBUS = "org.freedesktop.DBus"
DBUS_PATH = "/org/freedesktop/DBus"
REGISTRY = "org.a11y.atspi.Registry"
def call(address, destination, path, method, *args):
return subprocess.run(
["gdbus", "call", "--address", address, "--dest", destination,
"--object-path", path, "--method", method, "--timeout", "5", *args],
check=True, capture_output=True, text=True, timeout=6).stdout.strip()
def stop(child):
if child.poll() is None:
child.terminate()
try:
child.wait(timeout=3)
except subprocess.TimeoutExpired:
child.kill()
child.wait()
def run_registry(address, session):
# SteamOS's registryd stays alive after a bus disconnect. Keep only our own
# child tied to both private buses; no host PID lookup or broad process kill.
registry = subprocess.Popen(["/usr/lib/at-spi2-registryd"])
try:
while registry.poll() is None:
try:
call(session, DBUS, DBUS_PATH, DBUS + ".GetId")
call(address, DBUS, DBUS_PATH, DBUS + ".GetId")
except subprocess.SubprocessError:
break # Normal session shutdown or loss of its accessibility bus.
# Each check starts two gdbus processes, and SteamVR needs the CPU.
time.sleep(5)
finally:
stop(registry)
def start():
session = os.environ.get("DBUS_SESSION_BUS_ADDRESS")
runtime = os.environ.get("XDG_RUNTIME_DIR", "")
if not session or os.path.basename(runtime) != "frametop":
return # Do not autolaunch a bus or touch the host desktop.
os.environ.pop("AT_SPI_BUS_ADDRESS", None)
reply = call(session, "org.a11y.Bus", "/org/a11y/bus", "org.a11y.Bus.GetAddress")
address, = ast.literal_eval(reply)
if not isinstance(address, str) or not address.startswith("unix:"):
raise ValueError("org.a11y.Bus returned no local accessibility bus")
if call(address, DBUS, DBUS_PATH, DBUS + ".NameHasOwner", REGISTRY) == "(true,)":
return
# Activation otherwise inherits the accessibility daemon's old environment (and
# DBUS_SESSION_BUS_ADDRESS is the accessibility bus itself). Pin the live bus.
env = {key: os.environ.get(key, "") for key in (
"DISPLAY", "WAYLAND_DISPLAY", "XAUTHORITY", "XDG_RUNTIME_DIR",
"XDG_CONFIG_HOME", "DBUS_SESSION_BUS_ADDRESS")}
env["AT_SPI_BUS_ADDRESS"] = address
call(address, DBUS, DBUS_PATH, DBUS + ".UpdateActivationEnvironment", repr(env))
try:
call(address, DBUS, DBUS_PATH, DBUS + ".StartServiceByName", REGISTRY, "0")
except subprocess.CalledProcessError:
# SteamOS's launcher can select dbus-broker because we are in a user unit,
# but the private session bus has no systemd activation manager. Reuse its
# bus rather than starting another launcher/bus. Never replace an owner.
if call(address, DBUS, DBUS_PATH, DBUS + ".NameHasOwner", REGISTRY) == "(true,)":
return
os.environ["AT_SPI_BUS_ADDRESS"] = address
# registryd itself refuses to queue behind an existing registry, including
# races with native activation. The watcher cleans up only our own child.
run_registry(address, session)
if __name__ == "__main__":
signal.signal(signal.SIGTERM, lambda signum, frame: sys.exit(0))
try:
start()
except (OSError, ValueError, SyntaxError, subprocess.SubprocessError) as error:
print(f"frametop: accessibility unavailable: {error}", file=sys.stderr)
+1 -1
View File
@@ -10,7 +10,7 @@ cg=$(systemctl --user show -p ControlGroup --value "$unit" 2>/dev/null)
# The desktop's own processes stay in the unit and stop with it: the session, KWin,
# Plasma, and the session services it started (portals, input methods, kded, wallet...).
own='^(frametop-sessi|dbus-|startplasma|plasma|kwin|Xwayland|ksmserver|krdpserver|Xvnc|xfreerdp|ft-layout|'
own='^(frametop-sessi|dbus-|startplasma|plasma|kwin|Xwayland|ksmserver|krdpserver|Xvnc|xfreerdp|ft-layout|ft-atspi|'
own+='kded|kactivitymanage|kaccess|kglobalaccel|kscreen|kwalletd|ksecretd|polkit-kde|org_kde_|baloo|'
own+='xembedsniproxy|gmenudbusmenu|DiscoverNotifie|kimpanel|ibus|xdg-|at-spi|dconf-service|fusermount|agent)'
keep=()
File diff suppressed because it is too large. Load diff
+162
View File
@@ -0,0 +1,162 @@
#!/usr/bin/env python3
"""Tests for session/fix-panels.py: issue #18's config (the taskbar saved on spare output 8
of a 3-screen desktop), panels that are fine, an edge that's taken, and a real run of
kwriteconfig6 on a copy. Nothing here touches the running desktop or its config.
python3 session/tests/test_fix_panels.py
"""
import importlib.util
import os
import subprocess
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.realpath(__file__))
SCRIPT = os.path.join(HERE, "..", "fix-panels.py")
spec = importlib.util.spec_from_file_location("fix_panels", SCRIPT)
fp = importlib.util.module_from_spec(spec)
spec.loader.exec_module(fp)
def desktop(cid, screen):
return f"""[Containments][{cid}]
activityId=7e1d1a3c-0000-4000-8000-000000000000
formfactor=0
immutability=1
lastScreen={screen}
location=0
plugin=org.kde.plasma.folder
wallpaperplugin=org.kde.image
"""
def panel(cid, screen, location=4, tray=None):
text = f"""[Containments][{cid}]
activityId=
formfactor=2
immutability=1
lastScreen={screen}
location={location}
plugin=org.kde.panel
wallpaperplugin=org.kde.image
[Containments][{cid}][Applets][{cid}1]
immutability=1
plugin=org.kde.plasma.kickoff
"""
if tray:
text += f"""
[Containments][{cid}][Applets][{cid}2]
immutability=1
plugin=org.kde.plasma.systemtray
[Containments][{cid}][Applets][{cid}2][Configuration]
PreloadWeight=90
SystrayContainmentId={tray}
[Containments][{tray}]
activityId=
formfactor=2
immutability=1
lastScreen={screen}
location={location}
plugin=org.kde.plasma.private.systemtray
popupHeight=432
"""
return text
ISSUE_18 = "\n".join([desktop(1, 0), desktop(18, 1), desktop(19, 2)] + [desktop(20 + i, 3 + i) for i in range(8)]
+ [panel(10, 8, tray=11), "[ScreenMapping]\nitemsOnDisabledScreens=\n"])
class Plan(unittest.TestCase):
def test_issue_18(self):
moves, kept = fp.plan(fp.parse(ISSUE_18), 3)
self.assertEqual(moves, [("10", 8, ["10", "11"])])
self.assertEqual(kept, [])
def test_panel_on_a_screen_stays(self):
for screen in (0, 2, -1):
moves, kept = fp.plan(fp.parse(desktop(1, 0) + panel(2, screen, tray=8)), 3)
self.assertEqual((moves, kept), ([], []), screen)
def test_tray_is_not_a_panel(self):
found = fp.panels(fp.parse(panel(2, 0, tray=8)))
self.assertEqual(list(found), ["2"])
self.assertEqual(found["2"]["trays"], ["8"])
def test_desktops_never_move(self):
moves, _ = fp.plan(fp.parse(ISSUE_18), 1)
self.assertEqual([m[0] for m in moves], ["10"])
def test_edge_taken_on_first_screen(self):
text = panel(2, 0, tray=8) + panel(10, 8, tray=11)
moves, kept = fp.plan(fp.parse(text), 3)
self.assertEqual(moves, [])
self.assertEqual([k[:2] for k in kept], [("10", 8)])
def test_other_edge_is_free(self):
moves, kept = fp.plan(fp.parse(panel(2, 0) + panel(10, 4, location=3)), 3)
self.assertEqual(moves, [("10", 4, ["10"])])
self.assertEqual(kept, [])
def test_two_lost_on_one_edge(self):
moves, kept = fp.plan(fp.parse(panel(10, 5) + panel(12, 8)), 3)
self.assertEqual([m[0] for m in moves], ["10"])
self.assertEqual([k[0] for k in kept], ["12"])
def test_screen_count_went_down(self):
moves, _ = fp.plan(fp.parse(panel(2, 1, location=3)), 1)
self.assertEqual(moves, [("2", 1, ["2"])])
class Run(unittest.TestCase):
def setUp(self):
self.dir = tempfile.TemporaryDirectory()
self.path = os.path.join(self.dir.name, "plasma-org.kde.plasma.desktop-appletsrc")
with open(self.path, "w") as f:
f.write(ISSUE_18)
def tearDown(self):
self.dir.cleanup()
def run_script(self, *args):
return subprocess.run([sys.executable, SCRIPT, "--file", self.path, "--screens", "3", *args],
capture_output=True, text=True)
def test_check_reports_and_changes_nothing(self):
r = self.run_script("--check")
self.assertEqual(r.returncode, 1)
self.assertIn("panel 10: screen 8, bottom (lost", r.stdout)
with open(self.path) as f:
self.assertEqual(f.read(), ISSUE_18)
def test_repair(self):
r = self.run_script()
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn("moved it to the first screen", r.stderr)
with open(self.path) as f:
groups = fp.parse(f.read())
self.assertEqual(groups[("Containments", "10")]["lastScreen"], "0")
self.assertEqual(groups[("Containments", "11")]["lastScreen"], "0")
self.assertEqual(groups[("Containments", "10", "Applets", "101")]["plugin"], "org.kde.plasma.kickoff")
self.assertEqual(groups[("Containments", "25")]["lastScreen"], "8") # a spare's desktop
with open(self.path + ".ft-bak") as f:
self.assertEqual(f.read(), ISSUE_18)
# A second run finds nothing to do and leaves the backup alone.
os.remove(self.path + ".ft-bak")
r = self.run_script()
self.assertEqual((r.returncode, r.stderr), (0, ""))
self.assertFalse(os.path.exists(self.path + ".ft-bak"))
self.assertEqual(self.run_script("--check").returncode, 0)
def test_no_config_yet(self):
os.remove(self.path)
self.assertEqual(self.run_script().returncode, 0)
self.assertEqual(self.run_script("--check").returncode, 0)
if __name__ == "__main__":
unittest.main()
Executable
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env bash
# Uninstall Frametop from the Steam Frame. In a terminal on the headset (Konsole in the desktop,
# or over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
#
# or ~/frametop/uninstall.sh. It doesn't use the rest of the repo, so it also works when
# ~/frametop is gone or broken.
#
# It never stops what you're using now: the input relay carries the keyboard and mouse, and the
# desktop runs from the repo. So it goes in two steps:
# 1. Frametop stops starting. The launcher's Desktop entry goes back to the stock desktop, and
# Frametop's services, SteamVR driver, menu entries, and system files (our eye tracker's
# frame grabber and the Bluetooth fixes, with sudo) are removed. What runs now keeps running
# until you restart the headset.
# 2. After the restart, run it again. It deletes the code (~/frametop) and, if you want, your
# settings and the build container.
# When nothing of Frametop is running, one run does both.
#
# Options (piped, they go after "bash -s --"):
# --dir DIR where the repo is, if not ~/frametop
# --dry-run show what it would do, and change nothing
set -euo pipefail
shopt -s nullglob
usage() {
cat <<'EOF'
usage: uninstall.sh [--dir DIR] [--dry-run]
piped: curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- [options]
EOF
}
dry=0
apps=$HOME/.local/share/applications
override=$apps/deckard-nested-desktop.desktop
relay_unit=$HOME/.config/systemd/user/frametop-input-relay.service
driver=$HOME/.local/share/frametop/ft_pointer
vrpathreg=/opt/steamvr/bin/linuxarm64/vrpathreg
handsctl=$HOME/.local/bin/ft-handsctl
eyegrab_files=(/etc/systemd/system/frametop-eyegrab.service /etc/frametop/ft-eyegrab)
bt_files=(/etc/systemd/system/steamframe-bt-fixups.service /etc/systemd/system/bluetooth.service.d/steamframe.conf
/etc/steamframe/bt-fixups.sh)
step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
run() { # run a command, or with --dry-run, show it
if [ "$dry" = 1 ]; then printf ' would run: %s\n' "$*"; else "$@"; fi
}
ask() { # ask "question" default(y|n)
local hint answer
if [ "$dry" = 1 ]; then echo "$1 [dry run: yes]"; return 0; fi
hint=$([ "$2" = y ] && echo "Y/n" || echo "y/N")
read -r -p "$1 [$hint] " answer </dev/tty || answer=
answer=${answer:-$2}
[[ $answer =~ ^[Yy] ]]
}
exists() { local f; for f in "$@"; do [ -e "$f" ] && return 0; done; return 1; }
size() { du -shc "$@" 2>/dev/null | tail -1 | cut -f1; }
# Is this folder Frametop's code? Only then is it deleted.
is_repo() { [ "$1" != "$HOME" ] && [ -f "$1/desktops.sh" ] && [ -f "$1/session/frametop-session.sh" ]; }
find_repo() {
local p
[ -n "$dir" ] && { echo "$dir"; return; }
# The launcher entry and the relay's unit point into the repo, until step 1 removes them.
p=$(sed -n 's#^Exec=\(.*\)/session/frametop-session\.sh.*#\1#p' "$override" 2>/dev/null | head -1)
[ -z "$p" ] && p=$(sed -n 's#^ExecStart=/usr/bin/python3 \(.*\)/input/input-relay\.py.*#\1#p' "$relay_unit" 2>/dev/null | head -1)
[ -z "$p" ] && [ -f "${BASH_SOURCE[0]:-}" ] && p=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
echo "${p:-$HOME/frametop}"
}
# Frametop's programs that are running now (step 1 leaves them running until the restart).
running() {
local n out=()
for n in ft-screens ft-pointer ft-powerd ft-eyegrab ft-camd ft-hands; do
pgrep -x "$n" >/dev/null && out+=("$n")
done
pgrep -f '[i]nput/input-relay\.py' >/dev/null && out+=("input relay")
pgrep -f '[g]aze/ft-gazed' >/dev/null && out+=("gaze service")
pgrep -f '[f]rametop-session\.sh' >/dev/null && out+=("desktop session")
echo "${out[*]}"
}
main() {
local units=() entries=() names=() sys=0 repo now f
while [ $# -gt 0 ]; do
case $1 in
--dir) dir=${2:?--dir needs a folder}; shift ;;
--dry-run) dry=1 ;;
-h|--help) usage; return 0 ;;
*) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
esac
shift
done
if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
echo "This uninstalls Frametop from a Steam Frame (SteamOS, VR variant). Run it in a terminal on the headset." >&2
return 1
fi
if [ "$dry" = 0 ] && ! { : </dev/tty; } 2>/dev/null; then
echo "This asks questions, and there's no terminal to ask in. Run it in one (over SSH: ssh -t)." >&2
return 1
fi
[ "$dry" = 1 ] && echo "Dry run: nothing changes."
repo=$(find_repo)
# Step 1: what makes Frametop start. Nothing here stops a running program.
[ -f "$override" ] && grep -q 'Frametop' "$override" || override=
units=("$HOME"/.config/systemd/user/frametop-*.service)
for f in ft-input-settings ft-display-settings ft-layout-reset ft-screens-toggle ft-remote-settings ft-gazeprobe; do
[ -e "$apps/$f.desktop" ] && entries+=("$apps/$f.desktop")
done
[ -e "$apps/frametop-handrec.desktop" ] && entries+=("$apps/frametop-handrec.desktop")
entries+=("$apps"/frametop-profile-*.desktop) # one per layout profile (layout/ft_layout.py)
[ -L "$handsctl" ] || handsctl=
exists "${eyegrab_files[@]}" "${bt_files[@]}" && sys=1
if [ -n "$override" ] || [ ${#units[@]} -gt 0 ] || [ ${#entries[@]} -gt 0 ] || [ -d "$driver" ] ||
[ -n "$handsctl" ] || [ "$sys" = 1 ]; then
step "Step 1 of 2: stop Frametop from starting"
echo "This removes:"
[ -n "$override" ] && echo " - the launcher's Desktop entry (Launch a program -> Desktop opens the stock desktop again)"
for f in "${units[@]}"; do echo " - the service $(basename "$f")"; done
[ -d "$driver" ] && echo " - the 3D mouse's SteamVR driver (ft_pointer)"
[ ${#entries[@]} -gt 0 ] && echo " - ${#entries[@]} menu entries (Frametop Display Settings, Input Settings, ...)"
[ -n "$handsctl" ] && echo " - $handsctl"
exists "${eyegrab_files[@]}" && echo " - our eye tracker's frame grabber (a system service: needs your password)"
exists "${bt_files[@]}" && echo " - the Bluetooth fixes (system files: needs your password)"
echo "What runs now keeps running until you restart the headset, so your keyboard, mouse, and"
echo "this terminal keep working. Your settings and the code stay for now."
ask "Uninstall Frametop?" n || { echo "Nothing changed."; return 0; }
[ -n "$override" ] && run rm -f "$override"
if [ ${#units[@]} -gt 0 ]; then
for f in "${units[@]}"; do names+=("$(basename "$f")"); done
run systemctl --user disable "${names[@]}" 2>/dev/null || true
run rm -f "${units[@]}"
run systemctl --user daemon-reload
fi
if [ -d "$driver" ]; then
if [ -x "$vrpathreg" ]; then
LD_LIBRARY_PATH=$(dirname "$vrpathreg") run "$vrpathreg" removedriver "$driver" ||
echo "warning: SteamVR's vrpathreg couldn't unregister the driver; SteamVR may log that it's missing" >&2
fi
run rm -rf "$driver"
fi
[ ${#entries[@]} -gt 0 ] && run rm -f "${entries[@]}"
[ -n "$handsctl" ] && run rm -f "$handsctl"
if [ "$sys" = 1 ]; then
echo "The system files need your password (sudo)."
if ! run sudo bash -c '
for u in frametop-eyegrab steamframe-bt-fixups; do systemctl disable $u.service 2>/dev/null; done
rm -f "$@"
rmdir /etc/frametop /etc/steamframe /etc/systemd/system/bluetooth.service.d 2>/dev/null
systemctl daemon-reload; true' sys "${eyegrab_files[@]}" "${bt_files[@]}"; then
echo "warning: the system files weren't removed (no password?). Run this again to retry." >&2
fi
fi
[ "$dry" = 1 ] || echo "Frametop no longer starts."
fi
# Step 2: delete what's left, once nothing of Frametop runs.
now=$(running)
if [ -n "$now" ]; then
step "Restart the headset to finish"
echo "Still running from before: $now."
echo "They stop when the headset restarts. After that, run this again to delete the code"
echo "($repo) and, if you want, your settings and the build container:"
echo
if [ "$repo" = "$HOME/frametop" ]; then
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash"
else
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- --dir $(printf %q "$repo")"
fi
echo
if ask "Restart the headset now? This closes everything open, in VR and on the desktop." n; then
run systemctl reboot || echo "Couldn't restart it from here: restart the headset from Steam's power menu." >&2
fi
[ "$dry" = 1 ] || return 0
echo "Dry run: after the restart, step 2 would go like this."
fi
step "Step 2 of 2: delete what's left"
if [ -d "$repo" ] && is_repo "$repo"; then
if [ -f "$repo/.git" ]; then
echo "Leaving $repo: it's a git worktree. Remove it with git worktree remove."
else
local def=y
if [ -n "$(git -C "$repo" status --porcelain --untracked-files=no 2>/dev/null)" ] ||
[ -n "$(git -C "$repo" log --branches --not --remotes --oneline 2>/dev/null | head -1)" ]; then
echo "$repo has changes of its own (git -C $repo status)."
def=n
fi
if ask "Delete the Frametop code in $repo ($(size "$repo"))?" "$def"; then
cd "$HOME"
run rm -rf "$repo"
fi
fi
elif [ -e "$repo" ]; then
echo "Leaving $repo: it doesn't look like Frametop's code."
fi
# Rebuilt by the desktop at each start, so nothing to keep.
run rm -rf "$HOME/.local/share/frametop/apps" "$HOME/.local/share/kwin/decorations/kwin4_decoration_qml_frametop" \
"$HOME/.cache/frametop"
local settings=("$HOME"/.config/frametop.conf* "$HOME"/.config/frametop-*.json* "$HOME/.config/frametop-remote"
"$HOME/.config/frametop" "$HOME/.local/state/frametop")
local kept=()
for f in "${settings[@]}"; do [ -e "$f" ] && kept+=("$f"); done
if [ ${#kept[@]} -gt 0 ]; then
echo "Your settings: the screen layout and profiles, button maps, gaze calibration, the remote"
echo "desktop password, and the Frametop desktop's own Plasma setup (${kept[*]/#$HOME/\~})."
ask "Delete your settings too? Keep them to pick up where you left off if you reinstall." n &&
run rm -rf "${kept[@]}"
fi
local recs=()
for f in "$HOME/.local/share/frametop/eyes" "$HOME/.local/share/frametop/hands"; do [ -e "$f" ] && recs+=("$f"); done
if [ ${#recs[@]} -gt 0 ]; then
ask "Delete your eye and hand recordings in ~/.local/share/frametop ($(size "${recs[@]}"))?" n &&
run rm -rf "${recs[@]}"
fi
[ "$dry" = 1 ] || rmdir "$HOME/.local/share/frametop" 2>/dev/null || true
if command -v podman >/dev/null && podman container exists dev 2>/dev/null; then
echo "The build container (dev) holds Frametop's compilers and libraries, 1-2 GB. If you put"
echo "anything else in it, that goes too."
if ask "Delete the build container?" n; then
run "$HOME/.local/bin/distrobox" rm --force dev </dev/null
run podman image rm registry.fedoraproject.org/fedora-toolbox:44 >/dev/null 2>&1 || true
echo "distrobox stays in ~/.local/bin, for any other containers. To remove it too:"
echo " ~/dev/src/distrobox/uninstall --prefix ~/.local"
fi
fi
step "Done"
echo "Frametop is uninstalled."
}
dir=
main "$@"