Compare commits

..
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 33fb3bb611 update-check: KWin's blur and contrast effect ids, retest hints
The session now turns KWin's blur and contrast effects off by id (blurEnabled and
contrastEnabled in the desktop's kwinrc), and a KWin that renamed them would quietly
leave them on. The check looks for their built-in factories (KWin::blur_factory,
KWin::contrast_factory) in kwin_wayland, which it already reads for --output-count,
and warns if one is gone. The kwin and plasma-workspace retest hints gain the blur and
the hidden autostart entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:21:09 -06:00
DeeJanuzandClaude Opus 5.5 6cae2e0afe Remote Access: check the status every 5 s instead of every 2 s
While its window was open, Frametop Remote Access ran remote-ctl.sh status every 2 s,
and each run spawns bash, curl (the tailnet name from tailscaled) and python3 to parse
it. It now checks every 5 s, plus when the window comes to the front and once more 2 s
after turning remote access on or off or changing the password, so a change still
shows within a couple of seconds. A check doesn't start while one is still running.
Doing the check in-process would duplicate remote-ctl.sh's idea of "running", which
the session and the pause code share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:20:32 -06:00
DeeJanuzandClaude Opus 5.5 57617d4a14 ft-powerd: ask SteamVR every 100 ms, not on every input event
The loop polled the input devices with a 100 ms timeout and then, on every wake, did
SteamVR's part: PollNextEvent, the headset's activity level and every device's pose, all
IPC calls to vrserver. Input wakes it at once so the displays come on with the first
key or motion, but a moving mouse sends hundreds of events a second, so moving the mouse
meant hundreds of rounds of IPC a second instead of 10.

Every wake still drains the input devices and the control socket and counts input as
use straight away; SteamVR's part, and the backlight read that goes with it, now run
only when 100 ms have passed since the last time, and poll sleeps until then. Built in
the dev container (power/build.sh, no warnings); not run, since the live ft-powerd holds
@ft_powerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:19:49 -06:00
DeeJanuzandClaude Opus 5.5 b7dfe4759e Session: don't autostart Discover's notifier or IBus in the desktop
The nested Plasma session runs the system's XDG autostart entries. Discover's update
notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) started
plasma-discover --mode update inside it, 520-620 MB resident and about 9% of a core,
with flatpak-system-helper and AppStream downloads behind it. IBus started a nested
ibus-daemon with kimpanel and ibus-extension-gtk3, which no app in the desktop can use:
KWin's input method is ft-textinput (zwp_input_method_v1, focus reports only; the VR
keyboard types through ft-screens' seat), and the session already drops QT_IM_MODULE,
GTK_IM_MODULE and XMODIFIERS. Nothing in Frametop talks to IBus.

Before Plasma starts, the session script copies both entries into
$XDG_CONFIG_HOME/autostart with Hidden=true, which plasma-session honours for that
desktop only. It does this once ([Defaults] autostart=1 in frametoprc) and skips a name
the user already has a file for, so deleting the copy brings the program back. The
geoclue demo agent stays (it answers apps' location requests outside GNOME and idles at
0%), and orca's entry is OnlyShowIn GNOME-family desktops, so it never ran. Tested
against a temporary XDG_CONFIG_HOME, including an existing user ibus.desktop left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:18:31 -06:00
DeeJanuzandClaude Opus 5.5 5b6cfcd746 Session: blur, background contrast and animations off by default
The nested kwinrc had no [Plugins] group, so KWin ran its default blur and background
contrast effects, and kdeglobals had no AnimationDurationFactor, so animations ran at
full length. KWin renders through zink on Turnip, on the GPU vrcompositor needs, and
blur re-renders what's behind every translucent panel and menu; each animation frame is
another frame for KWin and ft-screens.

Before KWin starts, the session script now writes [Plugins] blurEnabled=false and
contrastEnabled=false to $XDG_CONFIG_HOME/kwinrc and [KDE] AnimationDurationFactor=0 to
its kdeglobals, each only if the desktop's own file has no value for it. It does this
once and records that in $XDG_CONFIG_HOME/frametoprc ([Defaults] effects=1), because
System Settings deletes a key put back to its default: without the marker, turning blur
back on wouldn't survive a restart. The ids blur and contrast are the built-in effects
of KWin 6.2.5 on SteamOS (both enabled by default in its plugin metadata). Tested
against a temporary XDG_CONFIG_HOME: fresh config, an existing blurEnabled=true kept,
and a deleted key not rewritten.

README and docs/reference.md say how to turn them back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:17:44 -06:00
DeeJanuzandClaude Opus 5.5 e1ee5ef395 Remote desktop: read the layout only after it changes
While FreeRDP ran, vnc-bridge.sh called ft-layout remote-view every 5 s, which scans
all of /proc for plasmashell and runs kscreen-doctor -j: about 4.4% of a core for a
layout that rarely changes.

It now stats the two files the answer depends on, the nested KWin's
~/.config/frametop/kwinoutputconfig.json (positions, scales, primary) and
~/.config/frametop-layout.json (screen sizes), once a second while FreeRDP runs. After
either changes it reads the layout every 2 s for 10 s, since KWin's outputs follow the
file a few seconds later; otherwise once a minute, in case a change touched neither.
With no VNC viewer connected nothing runs (previous commit).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:14:43 -06:00
DeeJanuzandClaude Opus 5.5 3e7248a04e Remote desktop: connect FreeRDP only while a VNC viewer is connected
vnc-bridge.sh kept FreeRDP connected to krdpserver from the moment remote desktop
started, so krdp captured and H.264-encoded every KWin redraw in software (openh264)
with nobody watching: krdpserver 55-78% of a core, xfreerdp 16-27%, Xvnc 6-11%, with 0
clients on :5900. krdp 6.7 creates its screencast session per RDP connection and drops
it when the connection closes, so krdpserver itself idles without one and stays up.

The bridge now counts established connections to Xvnc's port with ss, starts FreeRDP
when a viewer appears (the desktop shows about 3 s later; the VNC screen is black until
then) and stops it 45 s after the last one leaves (VNC_IDLE_SEC). Xvnc has no client
hook, so its log output, which it writes for every connection, wakes the bridge early;
otherwise it looks every 5 s while idle (0.1% of a core measured, against 0.9% for ss
once a second) and every second while FreeRDP runs. The layout check runs only while
FreeRDP runs.

While a viewer is connected the bridge sends "watch 15" to ft-screens (@ft_screens) at
once and every 5 s, so screens at a reduced frame rate (out of view, headset on a
stand) stream at full rate; it lapses by itself if the bridge dies, and an ft-screens
without the command just answers an error. The window search after starting FreeRDP
now ends when FreeRDP exits instead of polling for 30 s.

krdp on 127.0.0.1 with a fresh password, VNC on the tailnet address with VncAuth, and
remote-ctl.sh start/stop (pause and resume) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 09:14:25 -06:00
17 changed files with 311 additions and 376 deletions

No files matched your search

+2 -1
View File
@@ -144,7 +144,8 @@ This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 2026092
- Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels. - Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels.
- Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings. - Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings.
- On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md. - On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md.
- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. - Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. It costs almost nothing until a viewer connects; the picture then takes a few seconds to appear.
- The desktop has no blur behind panels and menus, and no window animations, so it leaves the headset's GPU to SteamVR. Turn them back on in the Frametop desktop's System Settings (Desktop Effects, and Animation speed under General Behavior); Frametop won't turn them off again.
- Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power. - Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power.
## Reporting problems ## Reporting problems
+6
View File
@@ -177,6 +177,12 @@ The private runtime directory also moves the session's document portal to `$XDG_
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)). A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompositor needs to hit its frame time, and on the Frame that costs CPU too. The nested session started with KWin's defaults: blur and background contrast on (no `[Plugins]` group in its kwinrc) and animations at full length. Blur re-renders what's behind every translucent panel and menu each time it changes, and every animated frame is one more frame for KWin and ft-screens to draw and send. They're off by default in the Frametop desktop. The session script writes them before KWin starts, only where the desktop's own config has no value, once: System Settings deletes a setting put back to KDE's default rather than writing it, so without the marker in `frametoprc` a user who turned blur back on would lose it at the next start. The effect ids (`blur`, `contrast`) are the ones built into KWin 6.2.5 on SteamOS; KWin reads `<id>Enabled` from `[Plugins]`.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
Remote desktop is a chain (krdp, then FreeRDP inside Xvnc) because nothing on SteamOS serves KWin over VNC directly. Kept connected all the time, it cost about a core with nobody watching: krdpserver 55 to 78% (it encodes H.264 in software with openh264: VA-API finds no driver for the Frame's GPU in the container), FreeRDP 16 to 27%, Xvnc 6 to 11%, and the bridge's layout check every 5 seconds another 4%. krdp creates its screencast session per RDP connection and drops it when the connection closes (`SessionController::onNewConnection` in krdp 6.7), so an idle krdpserver costs nothing and can stay up; only the RDP connection has to go. The bridge connects FreeRDP when a VNC client appears and disconnects 45 seconds after the last one leaves. Xvnc has no hook for its clients, so the bridge counts established connections to its port with `ss`, woken early by Xvnc's log output; looking with `ss` once a second cost about 0.9% of a core in bash, against about 0.1% this way. `Xvnc -inetd` from a systemd socket would start a server per connection and lose sharing between viewers. The layout check (`ft-layout remote-view`, which scans `/proc` for plasmashell and runs `kscreen-doctor -j`) now runs only while FreeRDP runs, and then only after `kwinoutputconfig.json` or `frametop-layout.json` changes, with one check a minute in case a change touched neither.
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`. Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
## Displays off on a stand ## Displays off on a stand
+12
View File
@@ -18,6 +18,16 @@ desktops.sh start | stop | restart | status | log [lines]
When the VR launcher starts the desktop, it inherits the Steam client's environment. The session script drops the client's runtime from it (`LD_LIBRARY_PATH`, the `STEAM_*` settings, and the Steam overlay's Vulkan layer), so apps in the desktop use the system's libraries, including its video codecs, just as they would after a normal login. When the VR launcher starts the desktop, it inherits the Steam client's environment. The session script drops the client's runtime from it (`LD_LIBRARY_PATH`, the `STEAM_*` settings, and the Steam overlay's Vulkan layer), so apps in the desktop use the system's libraries, including its video codecs, just as they would after a normal login.
KWin's blur and background contrast effects and its animations are off in this desktop, because KWin draws on the headset's GPU, which SteamVR needs. The session script turns them off once, the first time it starts (it leaves a setting you already have alone, and marks it done in `~/.config/frametop/frametoprc`), so turning them back on sticks. In the Frametop desktop, System Settings → Window Management → Desktop Effects has Blur and Background Contrast, and General Behavior has Animation speed. Or from a terminal, then restart the desktop:
```
kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key blurEnabled true
kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEnabled true
kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1
```
Two of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, and IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`), and skips a name you already have a file for. Delete a copy to start that program again.
Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key. Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
Restarting the desktop closes its windows. Before the unit stops, `session/keep-apps.sh` moves every program started in the desktop into a systemd scope of its own, so background work such as servers, tmux, and builds keeps running. An app that shuts down its own helper processes when its window closes will still lose them; run that kind of work outside the desktop, for example as a systemd user service. Restarting the desktop closes its windows. Before the unit stops, `session/keep-apps.sh` moves every program started in the desktop into a systemd scope of its own, so background work such as servers, tmux, and builds keeps running. An app that shuts down its own helper processes when its window closes will still lose them; run that kind of work outside the desktop, for example as a systemd user service.
@@ -259,6 +269,8 @@ It listens on port 5900 on the Frame's Tailscale address only, not the LAN, so i
No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency. krdp streams every screen; the VNC screen is the primary's size, and the FreeRDP window is shifted so the primary fills it (`ft-layout remote-view` gives the offset). krdp's own `--monitor` would stream just one screen, but it maps the pointer as if that screen sat at 0,0, so clicks would miss. When the layout changes, the VNC screen resizes and FreeRDP reconnects within a few seconds. No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency. krdp streams every screen; the VNC screen is the primary's size, and the FreeRDP window is shifted so the primary fills it (`ft-layout remote-view` gives the offset). krdp's own `--monitor` would stream just one screen, but it maps the pointer as if that screen sat at 0,0, so clicks would miss. When the layout changes, the VNC screen resizes and FreeRDP reconnects within a few seconds.
FreeRDP runs only while a VNC viewer is connected, because while it's connected krdp captures and encodes every redraw. With no viewer, krdp has no RDP connection and so captures nothing, and Xvnc shows a black screen. When a viewer connects, the bridge starts FreeRDP, and the desktop appears about 3 seconds later; FreeRDP stops 45 seconds after the last viewer leaves (`VNC_IDLE_SEC` in the bridge's environment). The bridge looks for viewers with `ss` whenever Xvnc logs something, as it does for every connection, and every 5 seconds otherwise. While a viewer is connected, the bridge asks ft-screens to draw every screen at full rate (`watch 15` on `@ft_screens`, renewed every 5 seconds), so screens you aren't looking at in the headset, or a headset on a stand, don't stream at a low rate. It reads the primary screen's place again (`ft-layout remote-view`) only while FreeRDP runs, after `~/.config/frametop/kwinoutputconfig.json` or `~/.config/frametop-layout.json` changes, and once a minute.
With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` and `KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screens or inject input. The second one lets scripts take screenshots through KWin's `org.kde.KWin.ScreenShot2` D-Bus interface. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop. With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` and `KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screens or inject input. The second one lets scripts take screenshots through KWin's `org.kde.KWin.ScreenShot2` D-Bus interface. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop.
## Limits ## Limits
+2 -2
View File
@@ -2,7 +2,7 @@
The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it. The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it.
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on. The gaze service asks it for only the sources it uses (`--sources`, and `sources LIST` on its stdin): our tracker and mmap set 1 with Own tracker, about 700 bytes a line instead of 1.3 KB, and every source while a check or the calibration runs. So SteamVR's gaze action, the only source that calls into vrserver (twice a sample), is read only then. The probe gets them all. - `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log. - `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below. - `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments. - `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
@@ -63,7 +63,7 @@ Ground rules, for anyone changing it:
- **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only). - **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only).
- **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`. - **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`.
- **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs. - **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs.
- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. ft-eyes keeps OpenCV and numpy to one thread: their pools of one per core spun idle workers at about a quarter of a core, for frames this small. It also runs at nice 10 with SCHED_BATCH, and ft-gaze at nice 5 (not batch, since each sample goes on to the pointer): both run in the dev container's podman scope, out of reach of the gaze service's unit, on the cores vrcompositor and vrserver use at nice 0. Replays, scoring, and training go to a PC. - **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. Replays, scoring, and training go to a PC.
## Headset fit ## Headset fit
+16 -85
View File
@@ -4,14 +4,7 @@
// Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source // Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source
// hit-tested against the Frametop screens: // hit-tested against the Frametop screens:
// //
// Options: -v (log action errors), --watch-stdin (quit when stdin closes; until then, a line // Options: -v (log action errors), --watch-stdin (quit when stdin closes).
// "sources LIST" on stdin switches the sources as --sources does), --sources LIST (comma-
// separated: action, mmap1, mmap2, left, right, own, and eye for EYE; or all, the default).
// Sources left out are read not at all and print as {"ok":0} ("eye" as null), so every line
// keeps the same keys. The gaze service asks for the ones it uses (own and mmap1 with our
// tracker), and all of them while a check or the calibration runs. Only the action costs
// SteamVR anything (two calls into vrserver per sample), and a line with every source is
// about 1.5 KB, 130 KB a second through podman's relay.
// //
// {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>, // {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
// "head":{"yaw":..,"pitch":..,"hit":HIT}, head forward ray (for head nudging) // "head":{"yaw":..,"pitch":..,"hit":HIT}, head forward ray (for head nudging)
@@ -65,7 +58,6 @@
#include <algorithm> #include <algorithm>
#include <atomic> #include <atomic>
#include <cerrno>
#include <chrono> #include <chrono>
#include <climits> #include <climits>
#include <cmath> #include <cmath>
@@ -80,7 +72,6 @@
#include <fcntl.h> #include <fcntl.h>
#include <sys/mman.h> #include <sys/mman.h>
#include <sys/resource.h>
#include <sys/socket.h> #include <sys/socket.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/un.h> #include <sys/un.h>
@@ -398,30 +389,6 @@ std::string SrcJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t
return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}"; return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}";
} }
// Sources (--sources, "sources LIST" on stdin): a bit each.
enum : unsigned { kAction = 1, kMmap1 = 2, kMmap2 = 4, kLeft = 8, kRight = 16, kOwn = 32, kEye = 64, kAll = 127 };
bool ParseSources(const std::string &list, unsigned &mask) {
static const struct {
const char *name;
unsigned bit;
} names[] = {{"action", kAction}, {"mmap1", kMmap1}, {"mmap2", kMmap2}, {"left", kLeft},
{"right", kRight}, {"own", kOwn}, {"eye", kEye}, {"all", kAll}};
unsigned m = 0;
size_t at = 0;
while (at <= list.size()) {
const size_t comma = std::min(list.find(',', at), list.size());
const std::string name = list.substr(at, comma - at);
bool known = false;
for (const auto &n : names)
if (name == n.name) m |= n.bit, known = true;
if (!known) return false;
at = comma + 1;
}
mask = m;
return true;
}
// Head poses of the last half second, so a sample can use the pose at its own time. // Head poses of the last half second, so a sample can use the pose at its own time.
class PoseHistory { class PoseHistory {
public: public:
@@ -459,42 +426,17 @@ std::string ExeDir() {
int main(int argc, char **argv) { int main(int argc, char **argv) {
bool verbose = false, watchStdin = false; bool verbose = false, watchStdin = false;
std::atomic<unsigned> sources{kAll};
for (int i = 1; i < argc; ++i) { for (int i = 1; i < argc; ++i) {
if (std::strcmp(argv[i], "-v") == 0) verbose = true; if (std::strcmp(argv[i], "-v") == 0) verbose = true;
if (std::strcmp(argv[i], "--watch-stdin") == 0) watchStdin = true; if (std::strcmp(argv[i], "--watch-stdin") == 0) watchStdin = true;
if (std::strcmp(argv[i], "--sources") == 0 && i + 1 < argc) {
unsigned m;
if (ParseSources(argv[++i], m))
sources = m;
else
std::fprintf(stderr, "ft-gaze: --sources %s: unknown source (all used)\n", argv[i]);
}
} }
// Nice 5, before any thread starts (they inherit it): we run in the dev container's podman
// scope, beside vrcompositor and vrserver at nice 0, and the gaze service's unit doesn't
// reach us. Not SCHED_BATCH, as ft-eyes is: that would let each wakeup wait out another
// task's turn, and each sample goes on to the pointer.
errno = 0;
const int nice0 = getpriority(PRIO_PROCESS, 0);
if (errno == 0 && nice0 < 5 && setpriority(PRIO_PROCESS, 0, 5) != 0)
std::fprintf(stderr, "ft-gaze: nice: %s\n", std::strerror(errno));
// --watch-stdin: quit when stdin closes. The probe runs us through distrobox, which // --watch-stdin: quit when stdin closes. The probe runs us through distrobox, which
// passes neither its signals nor a closed stdout on to us, but does pass stdin's end. // passes neither its signals nor a closed stdout on to us, but does pass stdin's end.
// Lines on stdin until then: "sources LIST".
std::atomic<bool> stdinClosed{false}; std::atomic<bool> stdinClosed{false};
if (watchStdin) if (watchStdin)
std::thread([&stdinClosed, &sources] { std::thread([&stdinClosed] {
char c[256]; char c[256];
std::string line; while (read(0, c, sizeof c) > 0) {
ssize_t n;
while ((n = read(0, c, sizeof c)) > 0) {
line.append(c, size_t(n));
for (size_t nl; (nl = line.find('\n')) != std::string::npos; line.erase(0, nl + 1)) {
unsigned m;
if (line.compare(0, 8, "sources ") == 0 && ParseSources(line.substr(8, nl - 8), m)) sources = m;
}
if (line.size() > 4096) line.clear(); // no newline in sight: not ours
} }
stdinClosed = true; stdinClosed = true;
}).detach(); }).detach();
@@ -567,7 +509,6 @@ int main(int argc, char **argv) {
if (fresh && hp.bPoseIsValid) { if (fresh && hp.bPoseIsValid) {
lastEmit = now; lastEmit = now;
const unsigned want = sources;
const auto list = screens.Get(); const auto list = screens.Get();
const vr::HmdMatrix34_t &headNow = hp.mDeviceToAbsoluteTracking; const vr::HmdMatrix34_t &headNow = hp.mDeviceToAbsoluteTracking;
vr::HmdMatrix34_t headThen = headNow; vr::HmdMatrix34_t headThen = headNow;
@@ -576,7 +517,7 @@ int main(int argc, char **argv) {
// SteamVR's action: a room-space origin and fixation point, turned into the head // SteamVR's action: a room-space origin and fixation point, turned into the head
// frame so every source reports the same kind of angles. // frame so every source reports the same kind of angles.
std::string action = "{\"ok\":0}"; std::string action = "{\"ok\":0}";
if (!inGame && (want & kAction)) { if (!inGame) {
vr::VRActiveActionSet_t active{}; vr::VRActiveActionSet_t active{};
active.ulActionSet = set; active.ulActionSet = set;
active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin; active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
@@ -619,33 +560,26 @@ int main(int argc, char **argv) {
return std::string(b); return std::string(b);
}; };
char extra[256]; char extra[256];
if (want & kMmap1) { std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1), s.open[0], s.open[1], lr(s.left1, s.right1));
s.open[0], s.open[1], lr(s.left1, s.right1)); m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1));
m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1)); std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
} m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
if (want & kMmap2) { left = SrcJson(list, headThen, s.left2);
std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2)); right = SrcJson(list, headThen, s.right2);
m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
}
if (want & kLeft) left = SrcJson(list, headThen, s.left2);
if (want & kRight) right = SrcJson(list, headThen, s.right2);
// "new" compares with the last sample, so the last measurement is kept either way.
const float *m = s.meas; const float *m = s.meas;
const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1]; const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1];
const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3]; const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3];
std::memcpy(lastMeas, m, sizeof lastMeas); std::memcpy(lastMeas, m, sizeof lastMeas);
if (want & kEye) { std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}",
std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}", (m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR));
(m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR)); eye = extra;
eye = extra;
}
} }
// Our tracker: its own sample time picks the head pose, like the mmap's. // Our tracker: its own sample time picks the head pose, like the mmap's.
std::string own = "{\"ok\":0}"; std::string own = "{\"ok\":0}";
OwnSample o; OwnSample o;
if ((want & kOwn) && ownFile.Read(o) && now - o.t < 0.1) { if (ownFile.Read(o) && now - o.t < 0.1) {
vr::HmdMatrix34_t headOwn = headNow; vr::HmdMatrix34_t headOwn = headNow;
history.At(o.t, headOwn); history.At(o.t, headOwn);
auto pair = [](bool ok, float a, float b) { auto pair = [](bool ok, float a, float b) {
@@ -690,10 +624,7 @@ int main(int argc, char **argv) {
break; break;
} }
if (stdinClosed) break; if (stdinClosed) break;
// 250 passes a second: a new sample is printed within 4 ms (2 on average) of appearing, std::this_thread::sleep_for(std::chrono::milliseconds(2));
// and the pose history has a pose within 2 ms of any sample's time (a 0.2 degree head
// turn at 100 degrees a second). Every 2 ms read the pose and the events twice as often.
std::this_thread::sleep_for(std::chrono::milliseconds(4));
} }
screens.Stop(); screens.Stop();
vr::VR_Shutdown(); vr::VR_Shutdown();
+3 -38
View File
@@ -76,12 +76,6 @@ off. A check asked for while idle waits for the tracker to start (at most WAKE_S
tracker's next click after it starts again re-seats it, as after the headset was off, so turning tracker's next click after it starts again re-seats it, as after the headset was off, so turning
gaze mode on after a while opens the quick check. gaze mode on after a while opens the quick check.
ft-gaze prints only the sources this uses ("sources LIST" on its stdin, see wanted_sources):
own and mmap1 with our tracker, left, right and mmap1 with SteamVR's eyes, and every source
while a check or the calibration runs or is asked for, since those record them all. So
SteamVR's gaze action, the one source that calls into vrserver, is read only then (or with
--source action).
Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
our tracker asks for a click, and the full calibration when gaze mode comes on without one, our tracker asks for a click, and the full calibration when gaze mode comes on without one,
both in a panel fixed to the headset (gaze/panel/ft-gazepanel, which this service runs too). both in a panel fixed to the headset (gaze/panel/ft-gazepanel, which this service runs too).
@@ -266,7 +260,6 @@ class Service:
self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own") self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own")
self.checks = Checks(self, self.sel) self.checks = Checks(self, self.sel)
self.proc = None self.proc = None
self.proc_sources = None # what ft-gaze was last told to print
self.buf = b"" self.buf = b""
self.restart_at = 0.0 self.restart_at = 0.0
self.running = True self.running = True
@@ -466,31 +459,6 @@ class Service:
# --- ft-gaze --- # --- ft-gaze ---
def wanted_sources(self):
"""The sources ft-gaze should print (its --sources): what on_sample and the checks read.
mmap1 always (blinks, lost eyes, and the headset going on, from its openness and
variances); everything while a check runs or waits, since they record every source."""
c = self.checks
if c.active or c.pending:
return "all"
kind = self.kind
if kind == "own":
return "own,mmap1"
if kind == "eyes":
return "left,right,mmap1"
return ",".join(dict.fromkeys((self.source, "mmap1", "mmap2"))) # mmap2: each eye, for the fallback
def sync_sources(self):
want = self.wanted_sources()
if not self.proc or want == self.proc_sources:
return
try:
self.proc.stdin.write(f"sources {want}\n".encode())
self.proc.stdin.flush()
except (OSError, ValueError):
return # it's stopping: read_stdout notices
self.proc_sources = want
def start_helper(self): def start_helper(self):
if not HELPER.exists(): if not HELPER.exists():
log(f"ft-gaze isn't built: run {REPO}/gaze/build.sh") log(f"ft-gaze isn't built: run {REPO}/gaze/build.sh")
@@ -501,11 +469,9 @@ class Service:
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False) subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox" distrobox = Path.home() / ".local" / "bin" / "distrobox"
# ft-gaze quits when its stdin closes: the one thing distrobox passes on. # ft-gaze quits when its stdin closes: the one thing distrobox passes on.
sources = self.wanted_sources() self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env,
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
"--sources", sources], env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE, start_new_session=True)
stderr=subprocess.PIPE, start_new_session=True)
self.proc_sources = sources
os.set_blocking(self.proc.stdout.fileno(), False) os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False) os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout") self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
@@ -910,7 +876,6 @@ class Service:
elif key.data == "stderr" and self.proc: elif key.data == "stderr" and self.proc:
self.read_stderr() self.read_stderr()
self.checks.tick() self.checks.tick()
self.sync_sources()
if now >= next_periodic: if now >= next_periodic:
self.periodic() self.periodic()
next_periodic = now + 1.0 next_periodic = now + 1.0
+1 -6
View File
@@ -47,7 +47,6 @@
#include <drm_fourcc.h> #include <drm_fourcc.h>
#include <fcntl.h> #include <fcntl.h>
#include <gbm.h> #include <gbm.h>
#include <poll.h>
#include <sys/socket.h> #include <sys/socket.h>
#include <sys/un.h> #include <sys/un.h>
#include <unistd.h> #include <unistd.h>
@@ -525,11 +524,7 @@ int main(int argc, char **argv) {
if (!shown) ov->ShowOverlay(h), shown = true; if (!shown) ov->ShowOverlay(h), shown = true;
dirty = false; dirty = false;
} }
// Until a command comes, or 10 ms while shown (SteamVR's events). Hidden, it waits up to a std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
// second: it woke 20 to 30 times a second for nothing, the main cost left with gaze idle.
// A closed stdin (--watch-stdin) wakes it too, so quitting doesn't wait.
pollfd fds[2] = {{sock, POLLIN, 0}, {0, POLLIN, 0}};
poll(fds, watchStdin ? 2 : 1, visible ? 10 : 1000);
} }
ov->DestroyOverlay(h); ov->DestroyOverlay(h);
buffers.Drop(); buffers.Drop();
+6 -26
View File
@@ -37,26 +37,14 @@ gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
logs = [] logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg) gazed.log = gazecheck.log = lambda msg: logs.append(msg)
# The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes. It logs the # The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes.
# sources it was asked for: "argv LIST" (--sources), then "line LIST" for each "sources LIST".
tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-") tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-")
FAKE = os.path.join(tmp, "ft-gaze") FAKE = os.path.join(tmp, "ft-gaze")
SOURCES_LOG = os.path.join(tmp, "sources.log")
with open(FAKE, "w") as f: with open(FAKE, "w") as f:
f.write('''import json, os, select, sys, time f.write('''import json, select, sys, time
log = open(sys.argv[1], "a", buffering=1)
log.write("argv " + (sys.argv[sys.argv.index("--sources") + 1] if "--sources" in sys.argv else "-") + "\\n")
buf = b""
while True: while True:
if select.select([sys.stdin], [], [], 1 / 90)[0]: if select.select([sys.stdin], [], [], 1 / 90)[0] and not sys.stdin.read(1):
data = os.read(0, 4096) break
if not data:
break
buf += data
while b"\\n" in buf:
line, buf = buf.split(b"\\n", 1)
if line.startswith(b"sources "):
log.write("line " + line[8:].decode() + "\\n")
eye = {"hy": 1.0, "hp": 2.0} eye = {"hy": 1.0, "hp": 2.0}
print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001], print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
"open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True) "open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True)
@@ -67,9 +55,8 @@ started = []
def start_helper(self): def start_helper(self):
"""ft-gaze, straight from here instead of the dev container.""" """ft-gaze, straight from here instead of the dev container."""
import selectors import selectors
self.proc = subprocess.Popen([sys.executable, FAKE, SOURCES_LOG, "--sources", self.wanted_sources()], self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stderr=subprocess.PIPE)
self.proc_sources = self.wanted_sources()
os.set_blocking(self.proc.stdout.fileno(), False) os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False) os.set_blocking(self.proc.stderr.fileno(), False)
self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout") self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
@@ -155,24 +142,17 @@ check("lease over (2 s + IDLE_AFTER): idle", wait(lambda: not svc.awake, 4.5), T
time.sleep(0.5) time.sleep(0.5)
logs.clear() logs.clear()
open(SOURCES_LOG, "w").close()
count = len(started) count = len(started)
check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first") check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first")
check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True) check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True)
check("once the tracker sends, it ran (and says why it couldn't open)", check("once the tracker sends, it ran (and says why it couldn't open)",
wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True) wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True)
check("nothing left pending", svc.checks.pending, None) check("nothing left pending", svc.checks.pending, None)
sources = open(SOURCES_LOG).read().split("\n")
check("ft-gaze started with every source for the check", sources[0], "argv all")
in_use = svc.wanted_sources()
check("then only those in use (SteamVR's tracker: not the action, not own)",
(f"line {in_use}" in sources, in_use != "all", "action" in in_use, "own" in in_use), (True, True, False, False))
check("idle again after", wait(lambda: not svc.awake, 3), True) check("idle again after", wait(lambda: not svc.awake, 3), True)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True) print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
svc.running = False svc.running = False
time.sleep(0.7) time.sleep(0.7)
os.remove(FAKE) os.remove(FAKE)
os.remove(SOURCES_LOG)
os.rmdir(tmp) os.rmdir(tmp)
os._exit(1 if failures else 0) os._exit(1 if failures else 0)
+1 -6
View File
@@ -9,12 +9,7 @@ the search runs again with a smaller closing.
import cv2 import cv2
import numpy as np import numpy as np
# One thread: OpenCV's pool of one per core costs more than it saves on a frame this small DARK = 30 # pupil pixels are below this (the face around it is 40-180)
# (a 140-240 px window while it follows the pupil). Its idle workers spun and yielded about
# 14,000 times a second each, a quarter of a core, beside SteamVR's compositor.
cv2.setNumThreads(1)
DARK = 30 # pupil pixels are below this (the face around it is 40-180)
MIN_AREA = 150 # pupil area range in pixels MIN_AREA = 150 # pupil area range in pixels
MAX_AREA = 20000 MAX_AREA = 20000
MIN_FILL = 0.75 # blob area / fitted-ellipse area MIN_FILL = 0.75 # blob area / fitted-ellipse area
-7
View File
@@ -63,13 +63,6 @@ reading only.
frame when its camera starts the frame after next (no slot is rewritten sooner than three frame when its camera starts the frame after next (no slot is rewritten sooner than three
frames), ignores late writes to the frame just finished, and saves from a separate frames), ignores late writes to the frame just finished, and saves from a separate
thread. fit1 may hold a few percent of torn frames. thread. fit1 may hold a few percent of torn frames.
- `--share` (2026-10-03) checks only the slot each camera writes next, from the two before
(the orders above, learned again if they change; all four slots for 2 s after a frame turns
up elsewhere), sleeps until 2.5 ms before the next frame is due, then looks every 1 ms with
0.5 ms of timer slack. Against the 0.3 ms poll of all eight slots, on simulated cameras:
207 wakeups a second instead of 1,486, 0.8% of a core instead of 3.6% (more on the real
DMA-BUF memory), no torn or skipped frames, and a frame's start seen 1.35 ms late on
average instead of 0.76. `--rec` still polls all slots every 0.3 ms, for its times.
- Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so - Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so
recordings are empty then. recordings are empty then.
- **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time): - **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time):
+16 -73
View File
@@ -46,7 +46,6 @@
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <sys/mman.h> #include <sys/mman.h>
#include <sys/prctl.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/syscall.h> #include <sys/syscall.h>
#include <time.h> #include <time.h>
@@ -289,53 +288,20 @@ static int eye_buffer(void) {
// Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass // Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass
// (forever if negative), a stop signal comes, the tracker process goes away, or keep() // (forever if negative), a stop signal comes, the tracker process goes away, or keep()
// (checked about every 0.25 s, when given) says to stop. Looks every `poll_us`. // (checked about every 0.25 s, when given) says to stop.
// //
// A frame lands over several milliseconds, in bursts, and its last bursts can come after // A frame lands over several milliseconds, in bursts, and its last bursts can come after
// the camera has started its next frame. A slot isn't rewritten until at least three frames // the camera has started its next frame. A slot isn't rewritten until at least three frames
// later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when // later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when
// its camera starts the frame after next. Changes to the slot just finished are late bursts, // its camera starts the frame after next. Changes to the slot just finished are late bursts,
// not a new frame. A frame's time is when its slot first changed. // not a new frame. A frame's time is when its slot first changed.
//
// Each look checks only the slot each camera writes next: the one that followed the last two
// before (after[][], seeded with the orders above and learned as frames come). Every check
// reads 256 words spread over a frame in DMA-BUF memory, so all eight slots each time was
// most of the cost. A camera with nothing in its expected slot for 1.5 frames, or with no
// order known yet, has all its slots checked until a frame comes. When that finds a frame
// somewhere else (the order changed, or a frame was missed), all its slots are checked for
// SCAN_AFTER_MISS, as before, while after[][] learns the new order. A slot's fingerprint is
// taken again when it stops being one of the two in use, so a check later sees only a new frame.
// Between frames it sleeps until FRAME_EARLY before the next one is due (the cameras run at
// 90 fps, within a ms of each other), then looks every `poll_us`.
#define FRAME_DUE (1.5 / 90) // s: a camera that hasn't started a frame by then gets all its slots checked
#define SCAN_AFTER_MISS 2.0 // s of checking all slots after a frame came in an unexpected one
#define CAMS_IDLE 0.5 // s without a frame from either camera: look 4 times less often
#define FRAME_EARLY 0.0025 // s before a frame is due to start looking for it
// When to start looking for the next frame: FRAME_EARLY before the first camera's is due. A
// camera already late (it lost its order, or stopped) means now.
static double next_due(const double last[2], double t) {
double due = 1e300;
for (int cam = 0; cam < 2; cam++) {
double d = last[cam] + 1.0 / 90 - FRAME_EARLY;
if (t - last[cam] > CAMS_IDLE) continue; // stopped: the other one sets the pace
if (d < due) due = d;
}
return due < 1e300 ? due : t;
}
static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double), static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double),
int (*keep)(void), unsigned poll_us) { int (*keep)(void)) {
static const int order[2][8] = {{3, 0, 1, 2, 3, 0, 1, 2}, {7, 5, 4, 6, 5, 7, 6, 4}};
int after[EYE_SLOTS][EYE_SLOTS];
memset(after, -1, sizeof after);
for (int c = 0; c < 2; c++)
for (int i = 0; i < 8; i++) after[order[c][i]][order[c][(i + 1) % 8]] = order[c][(i + 2) % 8];
uint64_t sig[EYE_SLOTS]; uint64_t sig[EYE_SLOTS];
double first[EYE_SLOTS]; double first[EYE_SLOTS];
int cur[2] = {-1, -1}, prev[2] = {-1, -1}; int cur[2] = {-1, -1}, prev[2] = {-1, -1};
for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0; for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0;
double start = now(), checked = start, kept = start, last[2] = {start, start}, scan_until[2] = {0, 0}; double start = now(), checked = start, kept = start;
char proc[64]; char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid); snprintf(proc, sizeof proc, "/proc/%d", pid);
while ((seconds < 0 || now() - start < seconds) && !stop_rec) { while ((seconds < 0 || now() - start < seconds) && !stop_rec) {
@@ -349,36 +315,18 @@ static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8
if (!keep()) return; if (!keep()) return;
kept = t; kept = t;
} }
for (int cam = 0; cam < 2; cam++) { for (int k = 0; k < EYE_SLOTS; k++) {
int next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1; uint64_t s = frame_sig(bufs[b].p + slot_start(k));
int all = next < 0 || t - last[cam] > FRAME_DUE || t < scan_until[cam]; if (s == sig[k]) continue;
int from = all ? cam * 4 : next, to = all ? cam * 4 + 4 : next + 1; sig[k] = s;
for (int k = from; k < to; k++) { int cam = k >= 4;
uint64_t s = frame_sig(bufs[b].p + slot_start(k)); if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst
if (s == sig[k]) continue; if (prev[cam] >= 0) done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
sig[k] = s; prev[cam] = cur[cam];
if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst cur[cam] = k;
if (next >= 0 && k != next) scan_until[cam] = t + SCAN_AFTER_MISS; first[k] = t;
if (prev[cam] >= 0) {
done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
after[prev[cam]][cur[cam]] = k;
// Out of use from now on: later changes are a new frame.
sig[prev[cam]] = frame_sig(bufs[b].p + slot_start(prev[cam]));
}
prev[cam] = cur[cam];
cur[cam] = k;
first[k] = t;
last[cam] = t;
next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1;
}
} }
double wait = poll_us * 1e-6; usleep(300);
if (t - last[0] > CAMS_IDLE && t - last[1] > CAMS_IDLE) {
wait *= 4;
} else if (next_due(last, t) - t > wait) {
wait = next_due(last, t) - t;
}
usleep((useconds_t)(wait * 1e6));
} }
} }
@@ -408,7 +356,7 @@ static int rec(double seconds, const char *dir, int pid) {
pthread_t writer; pthread_t writer;
pthread_create(&writer, NULL, ring_writer, NULL); pthread_create(&writer, NULL, ring_writer, NULL);
double start = now(); double start = now();
poll_frames(b, seconds, pid, rec_frame, NULL, 300); // 0.3 ms: recordings' times poll_frames(b, seconds, pid, rec_frame, NULL);
pthread_mutex_lock(&ring.mu); pthread_mutex_lock(&ring.mu);
ring.done = 1; ring.done = 1;
pthread_cond_signal(&ring.cv); pthread_cond_signal(&ring.cv);
@@ -435,10 +383,6 @@ static int rec(double seconds, const char *dir, int pid) {
#define SHARE_SLOTS 8 #define SHARE_SLOTS 8
#define SHARE_MAGIC 0x31434546u // "FEC1" #define SHARE_MAGIC 0x31434546u // "FEC1"
#define WANT_FRESH 3.0 // seconds a touch of the --want file lasts #define WANT_FRESH 3.0 // seconds a touch of the --want file lasts
// How often --share looks for frames, with a timer slack that lets the kernel group the
// wakeups: a frame reaches ft-eyes 1 to 1.5 ms after its camera starts the next, not 0.3.
#define SHARE_POLL_US 1000
#define SHARE_SLACK_NS 500000
typedef struct { typedef struct {
uint32_t magic, version, width, height, slots, entry_size; uint32_t magic, version, width, height, slots, entry_size;
@@ -518,7 +462,6 @@ static int share_loop(const char *path) {
.slots = SHARE_SLOTS, .entry_size = (uint32_t)esize}; .slots = SHARE_SLOTS, .entry_size = (uint32_t)esize};
signal(SIGINT, on_stop); signal(SIGINT, on_stop);
signal(SIGTERM, on_stop); signal(SIGTERM, on_stop);
if (prctl(PR_SET_TIMERSLACK, SHARE_SLACK_NS, 0, 0, 0) != 0) perror("PR_SET_TIMERSLACK");
fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "", fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "",
want_path ? want_path : ""); want_path ? want_path : "");
int pid = -1, idle = -1, missing = 0; int pid = -1, idle = -1, missing = 0;
@@ -543,7 +486,7 @@ static int share_loop(const char *path) {
if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid); if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid);
idle = 0, missing = 0; idle = 0, missing = 0;
h->tracker_pid = pid; h->tracker_pid = pid;
poll_frames(eye_buffer(), -1, pid, share_frame, wanted, SHARE_POLL_US); poll_frames(eye_buffer(), -1, pid, share_frame, wanted);
struct stat st; struct stat st;
char proc[64]; char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid); snprintf(proc, sizeof proc, "/proc/%d", pid);
+4 -42
View File
@@ -50,7 +50,6 @@ import json
import math import math
import mmap import mmap
import os import os
import select
import socket import socket
import struct import struct
import sys import sys
@@ -59,12 +58,7 @@ import time
from collections import deque from collections import deque
from pathlib import Path from pathlib import Path
# One thread for numpy's BLAS and OpenMP, set before numpy loads: it would start one per core import numpy as np
# (8 here) for small arrays that never need them. eyes_pupil keeps OpenCV to one as well.
for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS"):
os.environ.setdefault(_var, "1")
import numpy as np # noqa: E402
sys.path.insert(0, str(Path(__file__).resolve().parent)) sys.path.insert(0, str(Path(__file__).resolve().parent))
import eyes_model # noqa: E402 import eyes_model # noqa: E402
@@ -88,15 +82,6 @@ GAP = 3.0 # s without frames: the headset was off, and may sit diffe
CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation) CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation)
CALIB_MIN = 15 # frames an eye needs in a calibration dot's window CALIB_MIN = 15 # frames an eye needs in a calibration dot's window
CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot
# Waiting for frames. They come only as a counter in shared memory, so there's nothing to block
# on: sleep until a camera's next frame is due, then look every POLL. ft-eyegrab passes each one
# on within a ms or two of when its camera starts the one after, so they arrive 11.1 ms apart,
# give or take that.
PERIOD = 1 / 90 # s between a camera's frames
EARLY = 0.002 # start looking this long before a frame is due
POLL = 0.001 # then this often until it comes
STALLED = 0.1 # s without a frame: that camera stopped (headset off), and isn't waited for
IDLE_POLL = 0.02 # how often to look while both are stopped
class Cams: class Cams:
@@ -367,7 +352,7 @@ def wait_for_cams(sock, tracker, want):
return Cams() return Cams()
except (OSError, ValueError, RuntimeError): except (OSError, ValueError, RuntimeError):
serve(sock, tracker) serve(sock, tracker)
select.select([sock], [], [], 0.2) time.sleep(0.2)
def serve(sock, tracker): def serve(sock, tracker):
@@ -387,24 +372,7 @@ def serve(sock, tracker):
pass pass
def below_steamvr():
"""Nice 10 and SCHED_BATCH, for this thread and those it starts. ft-eyes runs in the dev
container's podman scope, where the gaze service's unit doesn't reach it, so it ran at
nice 0 on the cores vrcompositor and vrserver use. Batch also lets a waking ft-eyes wait
for the running task's turn instead of taking the core: a frame a few ms late costs the
gaze little, a late compositor frame costs a dropped frame in the headset."""
try:
os.setpriority(os.PRIO_PROCESS, 0, max(os.getpriority(os.PRIO_PROCESS, 0), 10))
except OSError as e:
print(f"ft-eyes: nice: {e}", file=sys.stderr, flush=True)
try:
os.sched_setscheduler(0, os.SCHED_BATCH, os.sched_param(0))
except (OSError, AttributeError) as e:
print(f"ft-eyes: SCHED_BATCH: {e}", file=sys.stderr, flush=True)
def main(): def main():
below_steamvr()
verbose = "-v" in sys.argv verbose = "-v" in sys.argv
if "--watch-stdin" in sys.argv: if "--watch-stdin" in sys.argv:
# Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our # Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our
@@ -430,7 +398,6 @@ def main():
print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True) print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True)
seen = [cams.count(0), cams.count(1)] seen = [cams.count(0), cams.count(1)]
report = time.monotonic() report = time.monotonic()
arrived = [0.0, 0.0] # when each camera's newest frame was seen (monotonic)
while True: while True:
serve(sock, tracker) serve(sock, tracker)
want() want()
@@ -440,7 +407,6 @@ def main():
if n == seen[c]: if n == seen[c]:
continue continue
seen[c] = n seen[c] = n
arrived[c] = time.monotonic()
got = cams.frame(c, n - 1) # only the newest: never fall behind got = cams.frame(c, n - 1) # only the newest: never fall behind
if got: if got:
eyes[c].feed(*got) eyes[c].feed(*got)
@@ -459,6 +425,8 @@ def main():
shifts += [float(v) for v in e.shift.value] shifts += [float(v) for v in e.shift.value]
pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan] pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan]
out.write(latest, (yaw, pitch), flags, per, shifts, pupils) out.write(latest, (yaw, pitch), flags, per, shifts, pupils)
else:
time.sleep(0.001)
now = time.monotonic() now = time.monotonic()
if now - report >= 5: if now - report >= 5:
if verbose: if verbose:
@@ -476,12 +444,6 @@ def main():
print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True) print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True)
cams = wait_for_cams(sock, tracker, want) cams = wait_for_cams(sock, tracker, want)
seen = [cams.count(0), cams.count(1)] seen = [cams.count(0), cams.count(1)]
# Until the next frame is due (a command on the socket wakes us sooner).
wait = IDLE_POLL
for c in (0, 1):
if now - arrived[c] < STALLED:
wait = min(wait, max(arrived[c] + PERIOD - EARLY - now, POLL))
select.select([sock], [], [], wait)
if __name__ == "__main__": if __name__ == "__main__":
+72 -64
View File
@@ -306,14 +306,19 @@ int main() {
std::printf("displays on: %s\n", why.c_str()); std::printf("displays on: %s\n", why.c_str());
}; };
// SteamVR is asked every 100 ms (events, the activity level, poses: IPC calls to
// vrserver). Input wakes the loop too, to wake the displays at once, but a moving mouse
// sends hundreds of events a second, so those wakes only drain the devices.
constexpr auto kTick = std::chrono::milliseconds(100);
auto lastVr = now - kTick;
std::vector<pollfd> fds; std::vector<pollfd> fds;
while (!stopSignal) { while (!stopSignal) {
// Sleep until input or 100 ms: input wakes the displays at once, poses every tick.
fds.clear(); fds.clear();
fds.push_back({ctl, POLLIN, 0}); fds.push_back({ctl, POLLIN, 0});
for (auto &[path, d] : inputs) for (auto &[path, d] : inputs)
if (d.fd >= 0) fds.push_back({d.fd, POLLIN, 0}); if (d.fd >= 0) fds.push_back({d.fd, POLLIN, 0});
poll(fds.data(), fds.size(), 100); const auto wait = std::chrono::ceil<std::chrono::milliseconds>(kTick - (Clock::now() - lastVr));
poll(fds.data(), fds.size(), std::clamp<int>(wait.count(), 0, kTick.count()));
now = Clock::now(); now = Clock::now();
const bool grace = now < graceUntil; const bool grace = now < graceUntil;
std::string use; // why this tick counts as use std::string use; // why this tick counts as use
@@ -329,71 +334,74 @@ int main() {
lastConf = now; lastConf = now;
} }
vr::VREvent_t ev; if (now - lastVr >= kTick) {
while (sys->PollNextEvent(&ev, sizeof ev)) { lastVr = now;
if (ev.eventType == vr::VREvent_Quit) { vr::VREvent_t ev;
stopSignal = SIGTERM; while (sys->PollNextEvent(&ev, sizeof ev)) {
sys->AcknowledgeQuit_Exiting(); if (ev.eventType == vr::VREvent_Quit) {
} else if (ev.eventType == vr::VREvent_TrackedDeviceActivated || stopSignal = SIGTERM;
ev.eventType == vr::VREvent_TrackedDeviceDeactivated) { sys->AcknowledgeQuit_Exiting();
lastClasses = now - std::chrono::hours(1); } else if (ev.eventType == vr::VREvent_TrackedDeviceActivated ||
} ev.eventType == vr::VREvent_TrackedDeviceDeactivated) {
} lastClasses = now - std::chrono::hours(1);
if (now - lastClasses > std::chrono::seconds(5)) {
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i)
classes[i] = sys->GetTrackedDeviceClass(i);
lastClasses = now;
}
// SteamVR's standby (headset taken off) turns the displays off itself; putting the
// headset on again (or the mount covering the sensor again) counts as use.
// A change counts once it has held for a second (the first reading after connecting
// is Idle).
const auto level = sys->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
const bool awayNow = level == vr::k_EDeviceActivityLevel_Idle || level == vr::k_EDeviceActivityLevel_Standby ||
level == vr::k_EDeviceActivityLevel_Idle_Timeout;
if (awayNow == away) awaySince = now;
if (awayNow != away && now - awaySince >= std::chrono::seconds(1)) {
away = awayNow;
std::printf("SteamVR: headset %s\n", away ? "off (SteamVR's standby has the displays)" : "on");
if (!away) use = "headset on again";
}
// Raw poses: recentering or a new play area doesn't move anything.
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, poses, vr::k_unMaxTrackedDeviceCount);
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
const auto c = classes[i];
if (c != vr::TrackedDeviceClass_HMD && c != vr::TrackedDeviceClass_Controller &&
c != vr::TrackedDeviceClass_GenericTracker)
continue;
const auto &p = poses[i];
if (!p.bPoseIsValid || p.eTrackingResult != vr::TrackingResult_Running_OK) {
anchors[i].set = false; // tracking back or a controller turned on: start over
continue;
}
if (!anchors[i].set || grace || now - anchors[i].at > kMoveWindow) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
continue;
}
double metres, degrees;
Distance(anchors[i], p.mDeviceToAbsoluteTracking, metres, degrees);
if (metres > moveMetres || degrees > moveDegrees) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
if (use.empty()) {
char why[96];
std::snprintf(why, sizeof why, "%s %u moved %.1f mm, %.2f deg",
c == vr::TrackedDeviceClass_HMD ? "headset" : "device", i, metres * 1000, degrees);
use = why;
} }
} }
} if (now - lastClasses > std::chrono::seconds(5)) {
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i)
classes[i] = sys->GetTrackedDeviceClass(i);
lastClasses = now;
}
if (off && ReadInt(kBacklight, -1) > 0) { // SteamVR's standby (headset taken off) turns the displays off itself; putting the
// Something else lit them (SteamVR leaving standby, the brightness setting). // headset on again (or the mount covering the sensor again) counts as use.
off = false; // A change counts once it has held for a second (the first reading after connecting
std::remove(SavedPath().c_str()); // is Idle).
use = "turned on elsewhere"; const auto level = sys->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
std::printf("displays on: turned on elsewhere\n"); const bool awayNow = level == vr::k_EDeviceActivityLevel_Idle || level == vr::k_EDeviceActivityLevel_Standby ||
level == vr::k_EDeviceActivityLevel_Idle_Timeout;
if (awayNow == away) awaySince = now;
if (awayNow != away && now - awaySince >= std::chrono::seconds(1)) {
away = awayNow;
std::printf("SteamVR: headset %s\n", away ? "off (SteamVR's standby has the displays)" : "on");
if (!away) use = "headset on again";
}
// Raw poses: recentering or a new play area doesn't move anything.
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, poses, vr::k_unMaxTrackedDeviceCount);
for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
const auto c = classes[i];
if (c != vr::TrackedDeviceClass_HMD && c != vr::TrackedDeviceClass_Controller &&
c != vr::TrackedDeviceClass_GenericTracker)
continue;
const auto &p = poses[i];
if (!p.bPoseIsValid || p.eTrackingResult != vr::TrackingResult_Running_OK) {
anchors[i].set = false; // tracking back or a controller turned on: start over
continue;
}
if (!anchors[i].set || grace || now - anchors[i].at > kMoveWindow) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
continue;
}
double metres, degrees;
Distance(anchors[i], p.mDeviceToAbsoluteTracking, metres, degrees);
if (metres > moveMetres || degrees > moveDegrees) {
SetAnchor(anchors[i], p.mDeviceToAbsoluteTracking, now);
if (use.empty()) {
char why[96];
std::snprintf(why, sizeof why, "%s %u moved %.1f mm, %.2f deg",
c == vr::TrackedDeviceClass_HMD ? "headset" : "device", i, metres * 1000, degrees);
use = why;
}
}
}
if (off && ReadInt(kBacklight, -1) > 0) {
// Something else lit them (SteamVR leaving standby, the brightness setting).
off = false;
std::remove(SavedPath().c_str());
use = "turned on elsewhere";
std::printf("displays on: turned on elsewhere\n");
}
} }
if (!use.empty()) { if (!use.empty()) {
lastUse = now; lastUse = now;
+17 -5
View File
@@ -83,6 +83,7 @@ class Window(Adw.ApplicationWindow):
self.status = {} self.status = {}
self.revealed = False self.revealed = False
self.setting = False # the switch is being set from the status, not by the user self.setting = False # the switch is being set from the status, not by the user
self.asking = False # a status check is running
self.toasts = Adw.ToastOverlay() self.toasts = Adw.ToastOverlay()
view = Adw.ToolbarView() view = Adw.ToolbarView()
@@ -139,16 +140,27 @@ class Window(Adw.ApplicationWindow):
"/tmp/frametop-remote.log and /tmp/frametop-vnc.log.") "/tmp/frametop-remote.log and /tmp/frametop-vnc.log.")
page.add(about) page.add(about)
# Each check runs remote-ctl.sh (bash, curl, and python3 for the tailnet name), so
# it runs every 5 seconds, when the window comes to the front, and after a change.
self.refresh() self.refresh()
GLib.timeout_add_seconds(2, self.refresh) GLib.timeout_add_seconds(5, self.refresh)
self.connect("notify::is-active", lambda win, _param: win.is_active() and win.refresh())
# --- status --- # --- status ---
def refresh(self): def refresh(self):
proc = Gio.Subprocess.new([CTL, "status"], Gio.SubprocessFlags.STDOUT_PIPE | Gio.SubprocessFlags.STDERR_SILENCE) if not self.asking:
proc.communicate_utf8_async(None, None, self.on_status) self.asking = True
proc = Gio.Subprocess.new([CTL, "status"], Gio.SubprocessFlags.STDOUT_PIPE | Gio.SubprocessFlags.STDERR_SILENCE)
proc.communicate_utf8_async(None, None, self.on_status)
return True return True
def refresh_soon(self):
"""Again in a moment: remote access takes a second or two to start or stop."""
self.refresh()
GLib.timeout_add_seconds(2, lambda: self.refresh() and False)
def on_status(self, proc, result): def on_status(self, proc, result):
self.asking = False
try: try:
_, out, _ = proc.communicate_utf8_finish(result) _, out, _ = proc.communicate_utf8_finish(result)
except GLib.Error: except GLib.Error:
@@ -197,7 +209,7 @@ class Window(Adw.ApplicationWindow):
self.toast("Remote access turns on at the next desktop restart") self.toast("Remote access turns on at the next desktop restart")
else: else:
self.toast("Remote access on") self.toast("Remote access on")
self.refresh() self.refresh_soon()
def on_reveal(self, button): def on_reveal(self, button):
self.revealed = button.get_active() self.revealed = button.get_active()
@@ -221,7 +233,7 @@ class Window(Adw.ApplicationWindow):
if self.status.get("running") == "1": if self.status.get("running") == "1":
subprocess.run([CTL, "restart"], stdin=subprocess.DEVNULL, capture_output=True) subprocess.run([CTL, "restart"], stdin=subprocess.DEVNULL, capture_output=True)
self.toast("New password set" + (": remote access restarted" if self.status.get("running") == "1" else "")) self.toast("New password set" + (": remote access restarted" if self.status.get("running") == "1" else ""))
self.refresh() self.refresh_soon()
def copy(self, text, done): def copy(self, text, done):
if text: if text:
+14 -6
View File
@@ -46,8 +46,9 @@ PACKAGES = {
"deckard-steamvr-rel": "the 3D mouse on the dashboard, on SteamVR Settings, and on a SteamVR " "deckard-steamvr-rel": "the 3D mouse on the dashboard, on SteamVR Settings, and on a SteamVR "
"window's grab bar; picking up a controller; mapped controller buttons; gaze", "window's grab bar; picking up a controller; mapped controller buttons; gaze",
"kwin": "clicks near the far edge of a screen whose scale isn't 1; every screen comes back " "kwin": "clicks near the far edge of a screen whose scale isn't 1; every screen comes back "
"after a desktop restart; floating a window", "after a desktop restart; floating a window; no blur behind the taskbar's menus",
"plasma-workspace": "the taskbar and panels after a desktop restart", "plasma-workspace": "the taskbar and panels after a desktop restart; no DiscoverNotifier or "
"ibus-daemon inside the desktop",
"gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked", "gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked",
"bluez": "a Bluetooth mouse reconnecting after it sleeps", "bluez": "a Bluetooth mouse reconnecting after it sleeps",
} }
@@ -176,14 +177,21 @@ def check_host():
try: try:
with open("/usr/bin/kwin_wayland", "rb") as f: with open("/usr/bin/kwin_wayland", "rb") as f:
# Qt keeps the option name as a UTF-16 string literal. kwin = f.read()
output_count = "output-count".encode("utf-16-le") in f.read()
except OSError: except OSError:
output_count = False kwin = b""
if output_count: # Qt keeps the option name as a UTF-16 string literal.
if "output-count".encode("utf-16-le") in kwin:
report("ok", "KWin", "has --output-count (one output per screen)") report("ok", "KWin", "has --output-count (one output per screen)")
else: else:
report("FAIL", "KWin", "no --output-count option: the desktop gets one screen at most") report("FAIL", "KWin", "no --output-count option: the desktop gets one screen at most")
# The session turns these built-in effects off by id (blurEnabled, contrastEnabled in kwinrc).
gone = [e for e in ("blur", "contrast") if b"KWin::%s_factory" % e.encode() not in kwin]
if not gone:
report("ok", "KWin effects", "blur and contrast, which the desktop turns off, keep their ids")
else:
report("warn", "KWin effects", f"no built-in {', '.join(gone)} effect: renamed? The desktop's "
"kwinrc may no longer turn it off (session/frametop-session.sh)")
if systemctl("cat", "steamvr.service"): if systemctl("cat", "steamvr.service"):
report("ok", "steamvr.service", "Frametop's services start and stop with it") report("ok", "steamvr.service", "Frametop's services start and stop with it")
+34
View File
@@ -234,6 +234,40 @@ elif [ "$(kreadconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme
kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete
fi fi
# KWin draws through zink on the headset's GPU, which vrcompositor needs, so the blur and
# background contrast behind panels and menus and the window animations cost frames for
# little. They're off by default in this desktop. Each is written once, before KWin first
# reads it, and only if this desktop's config doesn't have it yet; the marker keeps it
# from coming back, since System Settings deletes a setting put back to KDE's default.
# docs/reference.md says how to turn them on again.
default() { # file group key value
[ -n "$(kreadconfig6 --file "$1" --group "$2" --key "$3")" ] || kwriteconfig6 --file "$1" --group "$2" --key "$3" "$4"
}
frametoprc=$XDG_CONFIG_HOME/frametoprc
if [ "$(kreadconfig6 --file "$frametoprc" --group Defaults --key effects)" != 1 ]; then
default "$kwinrc" Plugins blurEnabled false
default "$kwinrc" Plugins contrastEnabled false
default "$XDG_CONFIG_HOME/kdeglobals" KDE AnimationDurationFactor 0
kwriteconfig6 --file "$frametoprc" --group Defaults --key effects 1
fi
# System autostart entries this desktop doesn't need, hidden for it alone by a copy with
# Hidden=true in its own autostart folder, once and unless there's a file of that name
# already. Discover's update notifier starts Discover itself to check for updates (about
# 600 MB and a share of a core, with Flatpak's helper and AppStream behind it); updates
# come with SteamOS and from Discover in the stock desktop. IBus can't reach the
# desktop's apps: KWin's input method is ft-textinput, and the session drops the
# variables that point apps at IBus or XIM. Deleting the copy brings an entry back.
if [ "$(kreadconfig6 --file "$frametoprc" --group Defaults --key autostart)" != 1 ]; then
for entry in org.kde.discover.notifier ibus; do
src=/etc/xdg/autostart/$entry.desktop dst=$XDG_CONFIG_HOME/autostart/$entry.desktop
[ -r "$src" ] && [ ! -e "$dst" ] || continue
mkdir -p "$(dirname "$dst")"
sed '/^\[Desktop Entry\]$/a Hidden=true' "$src" > "$dst"
done
kwriteconfig6 --file "$frametoprc" --group Defaults --key autostart 1
fi
# Profiles reopen apps (docs/profiles.md), so Plasma's own session restore stays off here; # Profiles reopen apps (docs/profiles.md), so Plasma's own session restore stays off here;
# with both, apps would open twice. # with both, apps would open twice.
kwriteconfig6 --file "$XDG_CONFIG_HOME/ksmserverrc" --group General --key loginMode emptySession kwriteconfig6 --file "$XDG_CONFIG_HOME/ksmserverrc" --group General --key loginMode emptySession
+105 -15
View File
@@ -11,6 +11,16 @@
# screen placed lower or further right. Instead the VNC screen is the primary's size, and # screen placed lower or further right. Instead the VNC screen is the primary's size, and
# the workspace-sized RDP window inside it is shifted so the primary fills it. The pointer # the workspace-sized RDP window inside it is shifted so the primary fills it. The pointer
# maps 1:1. When the layout changes, the VNC screen resizes and the RDP client reconnects. # maps 1:1. When the layout changes, the VNC screen resizes and the RDP client reconnects.
#
# The RDP client runs only while someone watches. While connected, krdp captures and
# H.264-encodes every redraw in software, about 60% of a core, with FreeRDP and Xvnc adding
# about 30% more, even with no VNC viewer. krdp starts its capture per RDP connection and
# stops it when the connection closes, so it idles without one. So FreeRDP starts when a
# VNC client connects (the screen is black for the few seconds that takes) and stops
# VNC_IDLE_SEC (45) seconds after the last one leaves. Xvnc has no hook for clients, so ss
# counts them: whenever Xvnc writes to its log (it logs each connection), every second
# while FreeRDP runs, and every 5 seconds otherwise. The log only wakes this script up;
# what it says doesn't matter.
set -eu set -eu
here=$(dirname "$(readlink -f "$0")") here=$(dirname "$(readlink -f "$0")")
@@ -57,16 +67,51 @@ stop_rdp() { pkill -f "[x]freerdp /v:127.0.0.1:$rdp_port " 2>/dev/null || true;
trap 'stop_rdp; pkill -f "[X]vnc $display " 2>/dev/null || true' EXIT trap 'stop_rdp; pkill -f "[X]vnc $display " 2>/dev/null || true' EXIT
box bash -c 'vncpasswd -f < "$1/vnc-password" > "$1/vnc-passwd.bin" && chmod 600 "$1/vnc-passwd.bin"' - "$creds" box bash -c 'vncpasswd -f < "$1/vnc-password" > "$1/vnc-passwd.bin" && chmod 600 "$1/vnc-passwd.bin"' - "$creds"
box Xvnc "$display" -geometry "${w}x${h}" -depth 24 \ exec {xlog}< <(box Xvnc "$display" -geometry "${w}x${h}" -depth 24 \
-interface "$addr" -rfbport "$vnc_port" \ -interface "$addr" -rfbport "$vnc_port" \
-SecurityTypes VncAuth -PasswordFile "$creds/vnc-passwd.bin" \ -SecurityTypes VncAuth -PasswordFile "$creds/vnc-passwd.bin" \
-AlwaysShared -desktop "Steam Frame (Frametop)" & -AlwaysShared -desktop "Steam Frame (Frametop)" 2>&1)
xvnc=$! xvnc=$!
sleep 2
# Keep an RDP connection open inside the VNC screen. Reconnect if it drops or the layout changes. # Wait up to $1 seconds, less if Xvnc logs something; its lines go on to this log.
nap() {
local line rc=0
IFS= read -rt "$1" -u "$xlog" line || rc=$?
if [ $rc -eq 0 ]; then
printf '%s\n' "$line"
while IFS= read -rt 0.1 -u "$xlog" line; do printf '%s\n' "$line"; done
elif [ $rc -le 128 ]; then
sleep 1 # Xvnc's output closed: it's exiting
fi
return 0
}
nap 2
# A connected VNC client: an established TCP connection to Xvnc's port. Any connection
# counts, authenticated or not; it's on the tailnet only.
clients() { [ -n "$(ss -Htn state established "( sport = :$vnc_port )" 2>/dev/null)" ]; }
# ft-screens drops screens you aren't looking at to a low frame rate, and krdp would
# stream that. "watch SECONDS" asks it for full rate on every screen for that long: sent
# when a client connects and renewed every few seconds while one stays, so it lapses by
# itself if this script dies. An older ft-screens just answers that it doesn't know it.
watch() {
if command -v socat >/dev/null; then
printf 'watch 15' | socat -u - ABSTRACT-SENDTO:ft_screens 2>/dev/null
else
python3 -c 'import socket; socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM).sendto(b"watch 15", "\0ft_screens")' 2>/dev/null
fi || true
}
# What remote-view depends on: KWin's saved outputs (positions, scales, primary) and
# Frametop's layout (screen sizes). It's read again only after one of these changes, and
# once a minute in case a change didn't touch them, and only while FreeRDP runs.
layout_files=("$HOME/.config/frametop/kwinoutputconfig.json" "$HOME/.config/frametop-layout.json")
stamp() { stat -c %y "${layout_files[@]}" 2>/dev/null || true; }
# Start FreeRDP inside the VNC screen, sized and shifted for $v.
# /cert:ignore is fine here: the connection never leaves this host. # /cert:ignore is fine here: the connection never leaves this host.
while kill -0 $xvnc 2>/dev/null; do start_rdp() {
read -r x y w h ww wh <<< "$v" read -r x y w h ww wh <<< "$v"
box env DISPLAY=$display bash -c ' box env DISPLAY=$display bash -c '
size=$1 x=$2 y=$3 ww=$4 wh=$5 creds=$6 rdp_port=$7 size=$1 x=$2 y=$3 ww=$4 wh=$5 creds=$6 rdp_port=$7
@@ -80,6 +125,7 @@ while kill -0 $xvnc 2>/dev/null; do
rdp=$! rdp=$!
# FreeRDP takes no negative position, so move its window once it is up. # FreeRDP takes no negative position, so move its window once it is up.
for _ in $(seq 60); do for _ in $(seq 60); do
kill -0 $rdp 2>/dev/null || break
win=$(xdotool search --class xfreerdp 2>/dev/null | tail -1) win=$(xdotool search --class xfreerdp 2>/dev/null | tail -1)
[ -n "$win" ] && break [ -n "$win" ] && break
sleep 0.5 sleep 0.5
@@ -88,14 +134,58 @@ while kill -0 $xvnc 2>/dev/null; do
wait $rdp wait $rdp
' vnc-rdp "${w}x$h" "$x" "$y" "$ww" "$wh" "$creds" "$rdp_port" || true & ' vnc-rdp "${w}x$h" "$x" "$y" "$ww" "$wh" "$creds" "$rdp_port" || true &
rdp=$! rdp=$!
while kill -0 $rdp 2>/dev/null; do }
sleep 5
now=$(view) || continue idle_sec=${VNC_IDLE_SEC:-45}
[ -n "$now" ] && [ "$now" != "$v" ] || continue rdp= # FreeRDP's job while it runs
echo "layout changed: $v -> $now" seen=0 # when a client was last seen ($SECONDS)
v=$now watched=-99 # when "watch" was last sent
stop_rdp stamp_v= # stamp() when $v was read
done recheck=0 # read the layout every 2 seconds until then
wait $rdp 2>/dev/null || true next_view=0 # next time to read it
sleep 2 while kill -0 $xvnc 2>/dev/null; do
if clients; then
if [ $((SECONDS - watched)) -ge 5 ]; then watch; watched=$SECONDS; fi
seen=$SECONDS
if [ -z "$rdp" ]; then
echo "VNC client connected, starting the RDP client"
now=$(view) && [ -n "$now" ] && v=$now
stamp_v=$(stamp) recheck=0 next_view=$((SECONDS + 60))
start_rdp
fi
elif [ "$seen" -ne 0 ]; then
watched=-99
if [ $((SECONDS - seen)) -ge "$idle_sec" ]; then
seen=0
if [ -n "$rdp" ]; then
echo "no VNC client for ${idle_sec}s, stopping the RDP client"
stop_rdp
wait "$rdp" 2>/dev/null || true
rdp=
fi
fi
fi
if [ -n "$rdp" ] && ! kill -0 "$rdp" 2>/dev/null; then
# It dropped, or the layout changed: reconnect next round if a client is still there.
wait "$rdp" 2>/dev/null || true
rdp=
nap 2
continue
fi
if [ -n "$rdp" ]; then
# A layout change shows up in KWin's outputs a few seconds after the files change.
s=$(stamp)
[ "$s" = "$stamp_v" ] || { stamp_v=$s; recheck=$((SECONDS + 10)) next_view=$SECONDS; }
if [ "$SECONDS" -ge "$next_view" ]; then
next_view=$((SECONDS + 60))
[ "$SECONDS" -ge "$recheck" ] || next_view=$((SECONDS + 2))
now=$(view) || now=
if [ -n "$now" ] && [ "$now" != "$v" ]; then
echo "layout changed: $v -> $now"
v=$now
stop_rdp
fi
fi
fi
if [ -n "$rdp" ] || [ "$seen" -ne 0 ]; then nap 1; else nap 5; fi
done done