mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 05:00:08 +02:00
Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 DeeJanuzandClaude Opus 5.5
|
540d8c425c | ||
|
Codex
|
ba6ccdfbd1 | ||
|
Codex
|
77f28f0922 | ||
|
DeeJanuzandClaude Opus 5.5
|
85532a54f3 | ||
|
DeeJanuzandClaude Opus 5.5
|
3864741f53 | ||
|
DeeJanuzandClaude Opus 5.5
|
12ad93d24c | ||
|
DeeJanuzandClaude Opus 5.5
|
a7f9a8dd80 | ||
|
DeeJanuzandClaude Opus 5.5
|
ac142f6e4a | ||
|
DeeJanuzandClaude Opus 5.5
|
b417534425 |
No files matched your search
@@ -13,6 +13,8 @@ Two settings apps come with it: Frametop Display Settings for the screens, profi
|
||||
|
||||
Frametop is an independent project, not made by or affiliated with Valve.
|
||||
|
||||
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
|
||||
|
||||
## Install on the headset
|
||||
|
||||
You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
|
||||
@@ -27,7 +29,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
|
||||
|
||||
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
|
||||
|
||||
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you three things along the way: whether to install gaze mode (experimental, yes by default) and the Bluetooth fixes, then whether to restart SteamVR. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
|
||||
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
|
||||
|
||||
After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings.
|
||||
|
||||
@@ -93,7 +95,7 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh
|
||||
|
||||
### Gaze mode (experimental)
|
||||
|
||||
In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later). Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
|
||||
In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
|
||||
|
||||
| Do this | To get this |
|
||||
| --- | --- |
|
||||
@@ -139,7 +141,7 @@ In a terminal on the headset, run:
|
||||
cd ~/frametop && scripts/report.sh
|
||||
```
|
||||
|
||||
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file.
|
||||
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
|
||||
|
||||
## Update
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# Controller desktop click stability
|
||||
|
||||
Trigger presses reach KDE immediately, but controller motion within 8 logical
|
||||
pixels of the press stays at that position until release. Releasing without a motion outside this
|
||||
zone delivers the click at the original position, even if the hand moved during
|
||||
release. Moving outside the zone begins a normal drag immediately; returning to
|
||||
the zone does not turn it back into a click. There is no hold-duration timer.
|
||||
|
||||
This filters overlay pointer content events on desktop monitors only, and only
|
||||
presses from hand controllers start it. The 3D mouse (whose laser comes from the
|
||||
`ft_pointer` virtual controller), SteamVR UI, separate screen grab bars and
|
||||
floating-app title-bar carrying are unaffected. Multi-button gestures keep their
|
||||
existing behavior. A motion onto another desktop monitor starts a drag;
|
||||
cross-monitor motion is not stabilized.
|
||||
|
||||
CLI (runtime preferences, reset to 8 on desktop restart):
|
||||
|
||||
```sh
|
||||
input/ft-clickctl status
|
||||
input/ft-clickctl threshold 8
|
||||
input/ft-clickctl threshold 0 # disable without a restart
|
||||
```
|
||||
|
||||
Thresholds are 0–64 logical pixels, normalized to each panel's KDE scale.
|
||||
Status reports held state, suppressed motions, stabilized clicks and drags.
|
||||
Changing the threshold while a controller button is held is refused.
|
||||
|
||||
This is a separate contribution from desktop mouse/controller ownership. Its
|
||||
hardware validation must check small controls, intentional text selection,
|
||||
long presses, cross-monitor dragging and simultaneous mouse use. The existing
|
||||
renderer laser remains tracked; this change stabilizes desktop input rather
|
||||
than smoothing the visual laser. Default threshold is a starting point to test.
|
||||
|
||||
Run `scripts/test-controller-click.sh` for the isolated gesture-state tests.
|
||||
+1
-1
@@ -175,7 +175,7 @@ Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens
|
||||
|
||||
The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused.
|
||||
|
||||
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it.
|
||||
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
|
||||
|
||||
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
|
||||
|
||||
|
||||
+2
-2
@@ -110,7 +110,7 @@ pointer/helper/run.sh status | log | restart
|
||||
pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer
|
||||
```
|
||||
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
|
||||
## Frametop Input Settings
|
||||
|
||||
@@ -206,7 +206,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
|
||||
- Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
|
||||
- The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
|
||||
- A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
|
||||
- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode coming on without a calibration opens Calibrate by itself.
|
||||
- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
|
||||
|
||||
[gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
|
||||
|
||||
|
||||
+3
-3
@@ -7,7 +7,7 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
|
||||
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
|
||||
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
|
||||
|
||||
Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`); the probe and our own tracker are installed by hand.
|
||||
Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
|
||||
|
||||
```
|
||||
gaze/run.sh install # the gaze service: builds ft-gaze and the panel, starts with SteamVR
|
||||
@@ -23,7 +23,7 @@ gaze/probe/ft-gazeprobe --screen 1
|
||||
Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
|
||||
|
||||
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
|
||||
- **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
|
||||
- **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
|
||||
- **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
|
||||
|
||||
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
|
||||
@@ -33,7 +33,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
|
||||
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
|
||||
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
|
||||
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
|
||||
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
|
||||
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
|
||||
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
|
||||
|
||||
Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
|
||||
|
||||
+10
-1
@@ -36,8 +36,17 @@ def main():
|
||||
if r is None:
|
||||
sys.exit("the pointer helper isn't running")
|
||||
print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
|
||||
if ask("ft_gazed", "status", 0.3) is None:
|
||||
st = ask("ft_gazed", "status", 0.3)
|
||||
if st is None:
|
||||
print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
|
||||
elif r == "ok on":
|
||||
try:
|
||||
calibrated = (json.loads(st).get("checks") or {}).get("calibrated")
|
||||
except ValueError:
|
||||
calibrated = None
|
||||
if calibrated is False:
|
||||
print("note: no calibration for this eye tracker yet; it opens in the headset "
|
||||
"(if it can't, ft-gazectl status says why: checks.problem)")
|
||||
elif cmd in ("status", "forget", "reload"):
|
||||
r = ask("ft_gazed", cmd)
|
||||
if r is None:
|
||||
|
||||
+29
-10
@@ -3,8 +3,12 @@
|
||||
|
||||
Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
|
||||
again when the file changes:
|
||||
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes,
|
||||
ft-gaze's source "own"; this service runs it, see below)
|
||||
GAZE_TRACKER=auto|own|steam
|
||||
our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own";
|
||||
this service runs it, see below) or SteamVR's. auto (the default)
|
||||
is ours when it's installed (gaze/tracker/install.sh: the frame
|
||||
grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
|
||||
switches when ours is installed or removed.
|
||||
GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
|
||||
off it was at your recent nudges; left or right counts that eye twice
|
||||
as much as the other. Either eye alone carries the gaze when the
|
||||
@@ -119,6 +123,7 @@ EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker
|
||||
EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh)
|
||||
EYES_SOCKET = "\0ft_eyes" # its control socket
|
||||
EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service)
|
||||
EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service")) # tracker/install.sh
|
||||
CONF = Path.home() / ".config" / "frametop.conf"
|
||||
CALIBRATION = STATE / "calibration.json"
|
||||
LESSONS = STATE / "pointer-lessons.json"
|
||||
@@ -126,6 +131,7 @@ LESSON_LOG = STATE / "pointer-lessons.jsonl"
|
||||
SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here
|
||||
SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
|
||||
TRACKERS = ("steam", "own")
|
||||
TRACKER_SETTINGS = ("auto",) + TRACKERS
|
||||
BIASES = ("auto", "left", "right")
|
||||
NUDGE_MAX = 55.0 # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
|
||||
HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
|
||||
@@ -153,8 +159,14 @@ def log(msg):
|
||||
print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def own_installed():
|
||||
"""Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
|
||||
return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
|
||||
|
||||
|
||||
def read_settings():
|
||||
"""(tracker, eye bias, nudge max) from frametop.conf, defaults for anything missing or unknown."""
|
||||
"""(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything
|
||||
missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed."""
|
||||
conf = {}
|
||||
try:
|
||||
for line in CONF.read_text().splitlines():
|
||||
@@ -164,13 +176,15 @@ def read_settings():
|
||||
conf[k.strip()] = v.strip().lower()
|
||||
except OSError:
|
||||
pass
|
||||
tracker = conf.get("GAZE_TRACKER", "steam")
|
||||
setting = conf.get("GAZE_TRACKER", "auto")
|
||||
setting = setting if setting in TRACKER_SETTINGS else "auto"
|
||||
tracker = setting if setting != "auto" else "own" if own_installed() else "steam"
|
||||
bias = conf.get("GAZE_EYE", "auto")
|
||||
try:
|
||||
nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0) # the helper's range
|
||||
except ValueError:
|
||||
nudge = NUDGE_MAX
|
||||
return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto", nudge
|
||||
return tracker, setting, bias if bias in BIASES else "auto", nudge
|
||||
|
||||
|
||||
def mtime(path):
|
||||
@@ -185,7 +199,7 @@ class Service:
|
||||
self.override, self.verbose, self.to = source, verbose, to
|
||||
self.source = source or "mmap1" # the older path's source
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
self.tracker, self.bias, self.nudge_max = read_settings()
|
||||
self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings()
|
||||
self.conf_mtime = mtime(CONF)
|
||||
self.models = {name: Correction() for name in SOURCES}
|
||||
self.mode = DEFAULT_MODEL
|
||||
@@ -246,9 +260,12 @@ class Service:
|
||||
|
||||
def load_settings(self):
|
||||
self.conf_mtime = mtime(CONF)
|
||||
tracker, bias, self.nudge_max = read_settings()
|
||||
tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
|
||||
if (tracker, bias) != (self.tracker, self.bias):
|
||||
log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
|
||||
auto = ""
|
||||
if self.tracker_setting == "auto":
|
||||
auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
|
||||
log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
|
||||
self.tracker, self.bias = tracker, bias
|
||||
for w in self.weights.values():
|
||||
w.bias = bias
|
||||
@@ -733,7 +750,8 @@ class Service:
|
||||
st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
|
||||
lessons=max(len(m) for m in w.misses), own_running=bool(own),
|
||||
own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
|
||||
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists())
|
||||
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
|
||||
tracker_setting=self.tracker_setting, own_installed=own_installed())
|
||||
st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
|
||||
if self.last_sample else None, "headset_on": self.steam.wearing(),
|
||||
"headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
|
||||
@@ -749,7 +767,8 @@ class Service:
|
||||
if mtime(CALIBRATION) != self.cal_mtime:
|
||||
self.load_calibration()
|
||||
self.refit()
|
||||
if mtime(CONF) != self.conf_mtime:
|
||||
if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
|
||||
and own_installed() != (self.tracker == "own")):
|
||||
self.load_settings()
|
||||
want = self.eyes_wanted()
|
||||
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
|
||||
|
||||
+23
-10
@@ -640,31 +640,44 @@ def cross_validate(points, mode):
|
||||
return errs
|
||||
|
||||
|
||||
def steady_samples(samples, vergence_jump=1.5):
|
||||
def steady_samples(samples, vergence_jump=1.5, why=None):
|
||||
"""The samples of one look at one spot where the tracker had both eyes: none in a blink
|
||||
(openness under half its median over the samples), none where it had lost an eye (its
|
||||
variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
|
||||
vergence) is more than `vergence_jump` degrees from its median over the samples. The
|
||||
vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
|
||||
fraction of one far off), so only a jump away from what it was during this look means
|
||||
the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept."""
|
||||
the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
|
||||
`why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
|
||||
"lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
|
||||
if why is None:
|
||||
why = {}
|
||||
# Openness: a blink is a sharp drop from what it was during this look. Not a fixed
|
||||
# level: looking down, the upper lids come down with the eyes, and in bright light you
|
||||
# squint, so the reading can stay under 0.5 for the whole look while the tracker follows
|
||||
# the eyes fine (a calibration dot at the bottom of the bright round failed that way).
|
||||
opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
|
||||
floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
|
||||
opened = []
|
||||
seen = []
|
||||
for smp in samples:
|
||||
o = (smp["src"].get("mmap1") or {}).get("open")
|
||||
if not o or min(o) >= floor:
|
||||
opened.append(smp)
|
||||
m1 = smp["src"].get("mmap1") or {}
|
||||
o = m1.get("open")
|
||||
lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
|
||||
# A lost eye's openness reads 0 too, so a lost eye is named before a blink.
|
||||
key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
|
||||
"blink" if o and min(o) < floor else None
|
||||
if key:
|
||||
why[key] = why.get(key, 0) + 1
|
||||
else:
|
||||
seen.append(smp)
|
||||
|
||||
def vergence(smp):
|
||||
return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
|
||||
opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST]
|
||||
have = [v for v in map(vergence, opened) if v is not None]
|
||||
have = [v for v in map(vergence, seen) if v is not None]
|
||||
if len(have) < 5:
|
||||
return opened
|
||||
return seen
|
||||
med = statistics.median(have)
|
||||
return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
|
||||
kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
|
||||
if len(kept) < len(seen):
|
||||
why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept)
|
||||
return kept
|
||||
+142
-23
@@ -17,9 +17,12 @@ directions: look at each one.
|
||||
size, and the tracker's error with it, changes with brightness), each the middle and
|
||||
a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
|
||||
RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
|
||||
when gaze mode comes on without a calibration for the tracker in use, and on
|
||||
"calibrate". Frametop's screens hide while it runs. Quitting it while there's still
|
||||
no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
|
||||
whenever gaze mode is on without a calibration for the tracker in use and someone's
|
||||
in the headset, and on "calibrate". One that closes unfinished (ignored, too few
|
||||
dots) opens again only once the headset comes off and on, or gaze mode off and on.
|
||||
Frametop's screens hide while it runs. Quitting it while there's still no
|
||||
calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
|
||||
Why gaze mode can't work yet goes in the status ("problem"), for Input Settings.
|
||||
|
||||
fit the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
|
||||
card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
|
||||
@@ -34,7 +37,10 @@ works however far off the tracker is; a left click or Meta+J (the pointer helper
|
||||
"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
|
||||
click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
|
||||
the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
|
||||
it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
|
||||
it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
|
||||
gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
|
||||
click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
|
||||
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
|
||||
renewed every second; the helper shows it again by itself when that stops).
|
||||
|
||||
What a capture teaches:
|
||||
@@ -90,6 +96,7 @@ ROUND_BG = (0.03, 0.33, 0.8)
|
||||
ROUND_NAMES = ("dark", "medium", "bright")
|
||||
RING_SCALE = (1.0, 0.5, 1.0)
|
||||
PANEL_RETRY = 10.0
|
||||
FULL_RETRY = 10.0 # seconds before an automatic calibration that failed to start tries again
|
||||
FIT_TIMEOUT = 300.0 # seconds the fit check stays up
|
||||
FIT_EVERY = 0.5 # seconds between its cards' updates (each is a new picture for the panel)
|
||||
FIT_HINT_WIDTH = 95 # characters a hint line holds in the panel
|
||||
@@ -117,6 +124,42 @@ def check_dots(kind, own):
|
||||
return out
|
||||
|
||||
|
||||
# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long):
|
||||
# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the
|
||||
# headset's fit causes, in the calibration's panel.
|
||||
STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
|
||||
"lost_right": ("right eye lost", "SteamVR lost your right eye"),
|
||||
"lost_both": ("both eyes lost", "SteamVR lost both eyes"),
|
||||
"blink": ("blinked", "you blinked"),
|
||||
"vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
|
||||
FIT_HINT = "check the headset fit"
|
||||
ACCEPT_WAIT = 1.5 # seconds after a click with no capture before the panel says what it waits for
|
||||
|
||||
|
||||
def reject_reason(reply, why):
|
||||
"""Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's
|
||||
drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause."""
|
||||
if reply is None:
|
||||
if not why:
|
||||
return "no reading", "SteamVR sent no reading for that look", False
|
||||
key = max(why, key=why.get)
|
||||
return *STEAM_REASONS[key], key.startswith("lost")
|
||||
words = reply.removeprefix("fail ").split()
|
||||
# ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
|
||||
if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
|
||||
eye = words[1]
|
||||
if "seen" in words:
|
||||
return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
|
||||
if "moved" in words:
|
||||
return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
|
||||
if not reply:
|
||||
return "no answer", "our tracker didn't answer", False
|
||||
if reply.startswith("our tracker isn't running"):
|
||||
return "tracker not running", "our tracker isn't running", False
|
||||
text = reply.removeprefix("fail ")
|
||||
return text[:24], f"our tracker said: {text}", False
|
||||
|
||||
|
||||
def spread(points):
|
||||
"""The median point and the spread around it (1.4826 x the median distance: a standard
|
||||
deviation that one stray sample can't move far)."""
|
||||
@@ -167,7 +210,10 @@ class Checks:
|
||||
self.check = None
|
||||
self.gaze_on = None
|
||||
self.gaze_heard = 0.0
|
||||
self.want_full_until = 0.0 # gaze mode came on before the tracker said whether it's calibrated
|
||||
self.full_armed = True # gaze mode on without a calibration opens the full one (need_full)
|
||||
self.full_blocked = None # why it can't open now
|
||||
self.full_retry_at = 0.0
|
||||
self.full_failed = None # why the last calibration failed (too few dots), for problem()
|
||||
self.last_quick = 0.0
|
||||
self.sample_at = 0.0 # the tracker last sent anything
|
||||
self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc")
|
||||
@@ -282,11 +328,49 @@ class Checks:
|
||||
return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
|
||||
|
||||
def on_gaze_on(self):
|
||||
self.full_armed = True
|
||||
self.need_full("gaze mode came on without a calibration")
|
||||
|
||||
def need_full(self, reason):
|
||||
"""Gaze mode is on without a calibration: open the full one, once per arming (see the top),
|
||||
or note why it can't open."""
|
||||
cal = self.calibrated()
|
||||
if cal is False:
|
||||
self.start("full", "gaze mode came on without a calibration")
|
||||
elif cal is None:
|
||||
self.want_full_until = time.monotonic() + 20
|
||||
if cal:
|
||||
self.full_armed = True # missing one later (the other tracker picked) is news again
|
||||
if not self.gaze_on or self.check or not self.full_armed or cal is not False:
|
||||
self.full_blocked = None
|
||||
return
|
||||
now = time.monotonic()
|
||||
if not self.can_run():
|
||||
why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
|
||||
else "no eyes seen (is the headset on?)")
|
||||
elif now < self.full_retry_at:
|
||||
return
|
||||
else:
|
||||
reply = self.start("full", reason)
|
||||
why = None if reply == "ok" else reply.removeprefix("error ")
|
||||
if why:
|
||||
self.full_retry_at = now + FULL_RETRY
|
||||
if why and why != self.full_blocked:
|
||||
log(f"the calibration can't open: {why}")
|
||||
self.full_blocked = why
|
||||
if not why:
|
||||
self.full_armed = False
|
||||
|
||||
def problem(self):
|
||||
"""Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said
|
||||
yet is None: Input Settings has its own line for our tracker."""
|
||||
if not self.gaze_on or self.calibrated() is not False:
|
||||
return None
|
||||
if self.check and self.check["kind"] == "full":
|
||||
return "Not calibrated yet: the calibration is open in the headset"
|
||||
if self.full_blocked:
|
||||
return f"Not calibrated, and the calibration can't open: {self.full_blocked}"
|
||||
if not self.full_armed:
|
||||
if self.full_failed:
|
||||
return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate"
|
||||
return "Not calibrated: the calibration closed unfinished. Use Calibrate"
|
||||
return "Not calibrated: the calibration opens in the headset"
|
||||
|
||||
def auto_quick(self, reason):
|
||||
now = time.monotonic()
|
||||
@@ -318,7 +402,7 @@ class Checks:
|
||||
now = time.monotonic()
|
||||
self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
|
||||
"started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
|
||||
"skipped": 0, "captured": 0, "points": {}}
|
||||
"skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
|
||||
log(f"{kind} check: {reason}")
|
||||
if kind == "full":
|
||||
st = ask(SCREENS, "state", 0.5).split()
|
||||
@@ -402,7 +486,7 @@ class Checks:
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
|
||||
self.last_progress = None
|
||||
c["shown"] = time.monotonic()
|
||||
c["run"], c["accept"], c["done_at"] = [], False, None
|
||||
c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None
|
||||
|
||||
def on_sample(self, s):
|
||||
self.sample_at = time.monotonic()
|
||||
@@ -428,6 +512,7 @@ class Checks:
|
||||
if "hy" not in src:
|
||||
return
|
||||
now = time.monotonic()
|
||||
c["gaze_at"] = now
|
||||
if now < c["shown"] + CHECK_SETTLE:
|
||||
return
|
||||
g = (src["hy"], src["hp"])
|
||||
@@ -488,7 +573,9 @@ class Checks:
|
||||
if ok:
|
||||
svc.weights["own"].add(miss)
|
||||
else:
|
||||
steady = steady_samples(samples)
|
||||
why = {}
|
||||
steady = steady_samples(samples, why=why)
|
||||
rec["dropped"] = why
|
||||
reads = {}
|
||||
for name in ("action", "mmap1", "mmap2", "left", "right"):
|
||||
pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
|
||||
@@ -522,23 +609,41 @@ class Checks:
|
||||
if svc.kind == "eyes":
|
||||
svc.weights["steam"].add(miss)
|
||||
svc.dirty = True
|
||||
if not ok:
|
||||
short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
|
||||
rec["reason"] = long
|
||||
c["reasons"][long] = c["reasons"].get(long, 0) + 1
|
||||
if fit:
|
||||
c["fit_reasons"].add(long)
|
||||
rec["accepted"] = ok
|
||||
self.log_check(rec)
|
||||
if not ok:
|
||||
c["tries"] += 1
|
||||
log(f"{c['kind']} check dot {c['i'] + 1}: not taken ({rec.get('reply', '')})")
|
||||
if c["tries"] >= 2 or c["kind"] != "full":
|
||||
log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
|
||||
full = c["kind"] == "full"
|
||||
if c["tries"] >= 2 or not full:
|
||||
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
|
||||
self.skip()
|
||||
else:
|
||||
self.note(f"Not taken: {long}. Look at the dot and click again")
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
|
||||
c["run"], c["accept"] = [], False
|
||||
c["run"], c["accept"], c["accept_at"] = [], False, None
|
||||
c["shown"] = time.monotonic() # settle again, then retry
|
||||
return
|
||||
c["captured"] += 1
|
||||
c["tries"] = 0
|
||||
self.note("")
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
|
||||
c["done_at"] = time.monotonic() + DONE_PAUSE
|
||||
|
||||
def note(self, text):
|
||||
"""The panel's warning line, over the instructions (empty: none). It stays until the
|
||||
next dot is taken, so a skipped dot's reason is still there at the one after it."""
|
||||
c = self.check
|
||||
if c.get("note") != text:
|
||||
c["note"] = text
|
||||
self.to_panel(f"note {text}".rstrip())
|
||||
|
||||
def skip(self):
|
||||
c = self.check
|
||||
yaw, pitch, _ = c["dots"][c["i"]]
|
||||
@@ -567,12 +672,20 @@ class Checks:
|
||||
return
|
||||
n = len(c["dots"])
|
||||
if c["captured"] < n * 2 / 3:
|
||||
log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays")
|
||||
reasons = c["reasons"]
|
||||
main = max(reasons, key=reasons.get) if reasons else None
|
||||
self.full_failed = main
|
||||
log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays"
|
||||
+ (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else ""))
|
||||
self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
|
||||
if main:
|
||||
fit = main in c["fit_reasons"]
|
||||
self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else ""))
|
||||
self.to_panel("dot 0 0 off")
|
||||
c["done_at"] = time.monotonic() + 3.0
|
||||
c["done_at"] = time.monotonic() + (8.0 if main else 3.0) # time to read why
|
||||
c["closing"] = True
|
||||
return
|
||||
self.full_failed = None
|
||||
if c["own"]:
|
||||
reply = ask(EYES, "calib-fit", 10.0)
|
||||
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
|
||||
@@ -642,6 +755,8 @@ class Checks:
|
||||
if self.check and self.check["kind"] == "fit":
|
||||
self.check["fit"].toggle_guide(time.monotonic())
|
||||
elif self.check:
|
||||
if not self.check["accept"]:
|
||||
self.check["accept_at"] = time.monotonic()
|
||||
self.check["accept"] = True
|
||||
return "ok"
|
||||
if cmd == "calquit":
|
||||
@@ -688,6 +803,13 @@ class Checks:
|
||||
return
|
||||
if c["done_at"]:
|
||||
return
|
||||
if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
|
||||
# Clicked, but no capture yet (see on_sample): say what it's waiting for.
|
||||
full = c["kind"] == "full"
|
||||
if now - c.get("gaze_at", 0.0) > 0.5:
|
||||
self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
|
||||
else:
|
||||
self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
|
||||
if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
|
||||
self.close("ignored")
|
||||
elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
|
||||
@@ -703,18 +825,13 @@ class Checks:
|
||||
self.gaze_on = None # the helper isn't answering
|
||||
if self.check:
|
||||
self.to_helper("calpanel 1")
|
||||
if self.want_full_until:
|
||||
cal = self.calibrated()
|
||||
if cal is not None or now > self.want_full_until:
|
||||
self.want_full_until = 0.0
|
||||
if cal is False and self.gaze_on:
|
||||
self.start("full", "gaze mode came on without a calibration")
|
||||
if not self.eyes_seen(AWAY_MIN):
|
||||
self.away, self.back_since = True, None
|
||||
elif self.back_since is not None and not self.eyes_seen():
|
||||
self.back_since = None # gone again before DON_DELAY
|
||||
elif self.back_since is not None and now - self.back_since >= DON_DELAY:
|
||||
self.away, self.back_since = False, None
|
||||
self.full_armed = True # a calibration that closed unfinished opens again
|
||||
self.auto_quick("the headset went on")
|
||||
if svc.kind == "own" and now - svc.own_at < 5:
|
||||
reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
|
||||
@@ -723,10 +840,12 @@ class Checks:
|
||||
elif not self.reseat_seen and self.can_run():
|
||||
self.reseat_seen = True
|
||||
self.auto_quick("our tracker asked for a click")
|
||||
self.need_full("gaze mode is on without a calibration")
|
||||
|
||||
def status(self):
|
||||
c = self.check
|
||||
st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
|
||||
"problem": self.problem(),
|
||||
"panel": self.panel_proc is not None,
|
||||
"last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
|
||||
if c:
|
||||
|
||||
@@ -22,6 +22,8 @@
|
||||
// look, capture (a ring filling to progress), done,
|
||||
// fail, off
|
||||
// title <text> / text <text> a line at the top / at the bottom (empty to clear)
|
||||
// note <text> a warning line just above the bottom one, in orange (red on
|
||||
// the bright round): why a dot wasn't taken (empty to clear)
|
||||
// eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
|
||||
// fit: an eye's card (0 left): its state in that colour, the
|
||||
// tracker's signal, the share of the last 10 s it was seen
|
||||
@@ -143,7 +145,7 @@ struct Panel {
|
||||
double wDeg = kQuickDeg; // across
|
||||
std::vector<uint8_t> px;
|
||||
double bg = 0.05;
|
||||
std::string title, text;
|
||||
std::string title, text, note;
|
||||
bool dotOn = false;
|
||||
double dotYaw = 0, dotPitch = 0, progress = 0;
|
||||
std::string state = "off";
|
||||
@@ -268,6 +270,9 @@ void Draw(Panel &p) {
|
||||
const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
|
||||
Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
|
||||
Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
|
||||
// Under the full calibration's lowest dots (RING degrees down) and over the text.
|
||||
if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05);
|
||||
else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3);
|
||||
if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
|
||||
if (!p.dotOn || p.state == "off") return;
|
||||
double x, y;
|
||||
@@ -468,7 +473,7 @@ int main(int argc, char **argv) {
|
||||
p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
|
||||
p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
|
||||
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
|
||||
p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off";
|
||||
p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
|
||||
p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
|
||||
place();
|
||||
visible = dirty = true; // shown with its first picture
|
||||
@@ -496,6 +501,8 @@ int main(int argc, char **argv) {
|
||||
p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
|
||||
} else if (!std::strncmp(buf, "text", 4)) {
|
||||
p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
|
||||
} else if (!std::strncmp(buf, "note", 4)) {
|
||||
p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true;
|
||||
} else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
|
||||
reply = visible ? "ok shown" : "ok hidden";
|
||||
} else {
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
# Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
|
||||
# service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
|
||||
# SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
|
||||
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker
|
||||
# is Own tracker or the gaze probe uses it.
|
||||
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
|
||||
# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
|
||||
# probe uses it. install.sh offers this after gaze mode.
|
||||
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
|
||||
# the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
|
||||
# (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
|
||||
|
||||
@@ -14,6 +14,8 @@ Two programs, each a user service that stops when SteamVR does:
|
||||
|
||||
They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
|
||||
|
||||
For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
|
||||
|
||||
```
|
||||
ft-handsctl on | off # on the Frame: start or stop hand tracking (SteamVR must be running)
|
||||
ft-handsctl status # the services, and ft-hands' last status lines
|
||||
@@ -21,6 +23,8 @@ ft-handsctl log [lines]
|
||||
ft-handsctl cutouts on|off|state # ft-screens' hand cutouts, without stopping tracking
|
||||
ft-handsctl gestures # pinches and grips, live (tools/watch_gestures.py --distance)
|
||||
|
||||
hands/ft-cutouts on | off | status # the cutouts only: no pinches or grips
|
||||
|
||||
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), install disabled
|
||||
hands/run.sh start|stop # start or stop the services
|
||||
hands/run.sh restart # after changing a setting
|
||||
@@ -104,6 +108,7 @@ Options:
|
||||
- `--models DIR`: where the models are.
|
||||
- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
|
||||
- `--no-publish`: don't write the hands and gestures files.
|
||||
- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
|
||||
- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
|
||||
- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
|
||||
- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
|
||||
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# ft-cutouts: the hand cutouts without pinches and grips. Your hands show through Frametop's
|
||||
# screens, but ft-hands runs with --no-gestures, so nothing clicks or drags. ft-handsctl on is
|
||||
# the full version; each stops the other. Like it, this doesn't start with SteamVR, and it
|
||||
# stops when SteamVR does. On the Frame.
|
||||
#
|
||||
# ft-cutouts on | off | status
|
||||
#
|
||||
# Runs this checkout's hands/build as transient user units, so it doesn't need hands/run.sh
|
||||
# install: only a build (hands/build.sh) and ft-camd's capabilities (hands/run.sh caps).
|
||||
set -euo pipefail
|
||||
here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
|
||||
camd=frametop-cutouts-camd.service hands=frametop-cutouts-hands.service
|
||||
|
||||
die() { echo "$*" >&2; exit 1; }
|
||||
|
||||
run_unit() { # unit, the ft-handsctl unit it replaces, description, then the command
|
||||
local unit=$1 conflicts=$2 what=$3
|
||||
shift 3
|
||||
systemctl --user -q is-active "$unit" && return
|
||||
systemd-run --user --quiet --collect --unit="$unit" --description="Frametop hand cutouts: $what" \
|
||||
-p PartOf=steamvr.service -p After=steamvr.service -p Conflicts="$conflicts" \
|
||||
-p Restart=always -p RestartSec=5 -p TimeoutStopSec=5 "$@"
|
||||
}
|
||||
|
||||
case ${1:-status} in
|
||||
on)
|
||||
systemctl --user -q is-active steamvr.service || die "SteamVR isn't running"
|
||||
[ -x "$here/build/ft-camd" ] && [ -x "$here/build/ft-hands" ] || die "not built: hands/build.sh"
|
||||
grep -qa -- --no-gestures "$here/build/ft-hands" || die "ft-hands predates --no-gestures: hands/build.sh"
|
||||
getcap "$here/build/ft-camd" | grep -q cap_sys_ptrace ||
|
||||
die "ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)"
|
||||
if systemctl --user -q is-active frametop-camd.service frametop-hands.service; then
|
||||
echo "stopping ft-handsctl's hand tracking (it has pinches and grips)"
|
||||
fi
|
||||
run_unit $camd frametop-camd.service "the camera broker" "$here/build/ft-camd" --status 60
|
||||
if ! systemctl --user -q is-active $hands; then
|
||||
"$here/../scripts/container-up.sh" # so stopping the unit can't take the container down
|
||||
pkill -x ft-hands || true
|
||||
fi
|
||||
run_unit $hands frametop-hands.service "hand tracking, no gestures" "$HOME/.local/bin/distrobox" enter dev -- \
|
||||
"$here/build/ft-hands" --status 60 --no-gestures
|
||||
"$here/ft-handsctl" cutouts on >/dev/null || true
|
||||
sleep 4
|
||||
"$0" status ;;
|
||||
off)
|
||||
systemctl --user stop $hands $camd 2>/dev/null || true
|
||||
systemctl --user -q is-active frametop-hands.service || pkill -x ft-hands || true
|
||||
echo "hand cutouts off" ;;
|
||||
status)
|
||||
for u in $camd $hands; do echo "$u: $(systemctl --user is-active $u || true)"; done
|
||||
echo "ft-screens cutouts: $("$here/ft-handsctl" cutouts state 2>&1)"
|
||||
inv=$(systemctl --user show -p InvocationID --value $hands) # this run's lines only
|
||||
[ -n "$inv" ] && journalctl --user _SYSTEMD_INVOCATION_ID="$inv" --no-pager -o cat |
|
||||
grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
|
||||
*) sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
|
||||
esac
|
||||
+19
-7
@@ -3,9 +3,12 @@
|
||||
// Python prototype: the same scheduling, with the models on a few threads.
|
||||
//
|
||||
// ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
|
||||
// [--no-publish] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
|
||||
// [--no-publish] [--no-gestures] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
|
||||
// (--help lists them all)
|
||||
//
|
||||
// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
|
||||
// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
|
||||
//
|
||||
// Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
|
||||
// own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
|
||||
// hands) they expose for the room and the hands come out dark. The two Arcturus colour
|
||||
@@ -150,7 +153,7 @@ struct Lighting {
|
||||
int main(int argc, char **argv) {
|
||||
double seconds = 0, status = 5;
|
||||
int threads = 3, niceness = 5;
|
||||
bool int8 = false, publish = true, track = true, swap_sides = false;
|
||||
bool int8 = false, publish = true, track = true, swap_sides = false, gestures_on = true;
|
||||
std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
|
||||
std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
|
||||
// SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
|
||||
@@ -189,6 +192,7 @@ int main(int argc, char **argv) {
|
||||
else if (a == "--nice" && more) niceness = std::atoi(argv[++i]);
|
||||
else if (a == "--int8") int8 = true;
|
||||
else if (a == "--no-publish") publish = false;
|
||||
else if (a == "--no-gestures") gestures_on = false;
|
||||
else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
|
||||
@@ -217,6 +221,7 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
else {
|
||||
std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
|
||||
" [--no-gestures] (hands for the cutouts only: no pinches or grips)\n"
|
||||
" [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
|
||||
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
|
||||
" [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
|
||||
@@ -265,12 +270,14 @@ int main(int argc, char **argv) {
|
||||
return true;
|
||||
};
|
||||
if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
|
||||
(publish && (!pub.open(err) || !gestures.open(pinch, grip, err))) ||
|
||||
(publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
|
||||
(!record.empty() && !start_recording(record, err))) {
|
||||
std::fprintf(stderr, "%s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
|
||||
// so a reader can't take an earlier run's file for this one's
|
||||
if (publish && !gestures_on) unlink((run_dir() + "/gestures").c_str());
|
||||
|
||||
std::map<std::string, int> index; // mono calibration name -> ring camera
|
||||
std::map<std::string, int> color; // colour calibration name (color_video<N>) -> ring camera
|
||||
@@ -541,9 +548,11 @@ int main(int argc, char **argv) {
|
||||
const auto hands = tracker.step(images, int64_t(tmin));
|
||||
const uint64_t capture = uint64_t(int64_t(tmin) - raw_off); // CLOCK_MONOTONIC
|
||||
const std::vector<Seen> views = tracker.views_now();
|
||||
if (gestures_on) {
|
||||
grip.update(hands, views, int64_t(capture));
|
||||
pinch.update(hands, views, int64_t(capture), grip.gripping());
|
||||
if (gesture_log && capture - t_glog >= 100'000'000) {
|
||||
}
|
||||
if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
|
||||
t_glog = capture;
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (pinch.world_d[k] >= 0)
|
||||
@@ -554,7 +563,8 @@ int main(int argc, char **argv) {
|
||||
// a gesture down or closing gets the full rate, even while the palm holds still
|
||||
next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
|
||||
0.005) * 1e9);
|
||||
if (publish) pub.write(hands, capture), gestures.write(pinch, grip, capture);
|
||||
if (publish) pub.write(hands, capture);
|
||||
if (publish && gestures_on) gestures.write(pinch, grip, capture);
|
||||
for (const Pinch::Event &e : grip.events)
|
||||
std::printf("grip %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
|
||||
e.point[0], e.point[1], e.point[2]);
|
||||
@@ -593,6 +603,7 @@ int main(int argc, char **argv) {
|
||||
s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
|
||||
!rec ? "" : (" recorded " + std::to_string(rec->written()) + " dropped " +
|
||||
std::to_string(rec->dropped())).c_str());
|
||||
if (gestures_on) {
|
||||
std::printf(" pinches: left %u right %u (held back, palm down: %d %d) grips: left %u right %u",
|
||||
pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
|
||||
grip.side(0).begins, grip.side(1).begins);
|
||||
@@ -604,6 +615,7 @@ int main(int argc, char **argv) {
|
||||
: "open",
|
||||
pinch.side(k).distance, grip.curl[k]);
|
||||
std::printf("\n");
|
||||
}
|
||||
for (const Hand *h : hands)
|
||||
std::printf(" hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m scale %.2f speed %.2f m/s\n", h->id,
|
||||
h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale,
|
||||
@@ -616,8 +628,8 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
}
|
||||
if (!color.empty()) unlink(want_file.c_str());
|
||||
if (publish) {
|
||||
pub.write({}, mono_ns());
|
||||
if (publish) pub.write({}, mono_ns());
|
||||
if (publish && gestures_on) {
|
||||
pinch.release(int64_t(mono_ns())); // a drag in progress ends, as lost
|
||||
grip.release(int64_t(mono_ns()));
|
||||
gestures.write(pinch, grip, mono_ns());
|
||||
|
||||
@@ -162,8 +162,12 @@ def key_bindings(rules):
|
||||
counts as its own)."""
|
||||
bound = rules.get("key_bindings")
|
||||
return dict(bound) if isinstance(bound, dict) else dict(DEFAULT_KEY_BINDINGS)
|
||||
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias.
|
||||
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. GAZE_TRACKER
|
||||
# can also be auto (the default): ours when it's installed, else SteamVR's.
|
||||
GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
|
||||
# Our tracker's frame grabber (gaze/tracker/install.sh), on the host: this runs in the dev
|
||||
# container, which has the host's /etc under /run/host.
|
||||
EYEGRAB = ("/etc/frametop/ft-eyegrab", "/run/host/etc/frametop/ft-eyegrab")
|
||||
GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"}
|
||||
# Pointer settings: key, label, default, min, max, step, unit.
|
||||
POINTER_SETTINGS = [
|
||||
@@ -961,9 +965,25 @@ class Backend(QObject):
|
||||
|
||||
@Property(str, notify=gazeChanged)
|
||||
def gazeTracker(self):
|
||||
"""Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER)."""
|
||||
v = read_conf().get("GAZE_TRACKER", "steam")
|
||||
return v if v in GAZE_TRACKERS else "steam"
|
||||
"""Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER; auto is
|
||||
ours when it's installed)."""
|
||||
v = read_conf().get("GAZE_TRACKER", "auto")
|
||||
if v in GAZE_TRACKERS:
|
||||
return v
|
||||
return "own" if self.gazeOwnInstalled else "steam"
|
||||
|
||||
@Property(bool, notify=gazeChanged)
|
||||
def gazeTrackerAuto(self):
|
||||
"""GAZE_TRACKER is auto (or missing): ours once it's installed, without a choice here."""
|
||||
return read_conf().get("GAZE_TRACKER", "auto") not in GAZE_TRACKERS
|
||||
|
||||
@Property(bool, notify=gazeChanged)
|
||||
def gazeOwnInstalled(self):
|
||||
"""Is our own tracker installed? The gaze service says (it also checks ft-eyes' Python in
|
||||
its checkout); without it, whether the frame grabber is."""
|
||||
if "own_installed" in self._gaze:
|
||||
return bool(self._gaze["own_installed"])
|
||||
return any(os.path.exists(p) for p in EYEGRAB)
|
||||
|
||||
@Property(str, notify=gazeChanged)
|
||||
def gazeEye(self):
|
||||
|
||||
+33
-4
@@ -902,6 +902,23 @@ Kirigami.ApplicationWindow {
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
Controls.Label {
|
||||
// Why the gaze pointer, on, can't follow your eyes yet, so it isn't left looking
|
||||
// like a plain mouse. The calibration part is the gaze service's (gaze/gazecheck.py).
|
||||
readonly property var checks: gpage.status.checks || {}
|
||||
readonly property string why: !backend.gazeServiceInstalled
|
||||
? "The gaze service isn't installed: run gaze/run.sh install in the Frametop folder, in a terminal"
|
||||
: !backend.gazeServiceRunning ? "The gaze service isn't running (it starts with SteamVR)"
|
||||
: backend.gazeTracker === "own" && !gpage.status.eyegrab
|
||||
? "Our eye tracker needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
|
||||
: checks.problem || ""
|
||||
visible: backend.gazeMode > 0 && why !== ""
|
||||
text: why
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
|
||||
wrapMode: Text.WordWrap
|
||||
color: checks.check ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
ColumnLayout {
|
||||
Kirigami.FormData.label: "Mouse left button:"
|
||||
Repeater {
|
||||
@@ -949,16 +966,28 @@ Kirigami.ApplicationWindow {
|
||||
}
|
||||
RowLayout {
|
||||
Kirigami.FormData.label: "Eye tracker:"
|
||||
Controls.RadioButton {
|
||||
text: "Own tracker (recommended)"
|
||||
checked: backend.gazeTracker === "own"
|
||||
onToggled: if (checked) backend.setGazeTracker("own")
|
||||
}
|
||||
Controls.RadioButton {
|
||||
text: "SteamVR"
|
||||
checked: backend.gazeTracker === "steam"
|
||||
onToggled: if (checked) backend.setGazeTracker("steam")
|
||||
}
|
||||
Controls.RadioButton {
|
||||
text: "Own tracker"
|
||||
checked: backend.gazeTracker === "own"
|
||||
onToggled: if (checked) backend.setGazeTracker("own")
|
||||
}
|
||||
Controls.Label {
|
||||
// Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
|
||||
visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
|
||||
text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
|
||||
+ "Frametop folder, in a terminal (asks for sudo)"
|
||||
+ (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
|
||||
: ", then pick Own tracker here and calibrate it once.")
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
|
||||
wrapMode: Text.WordWrap
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
Controls.Label {
|
||||
// The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber
|
||||
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Controller desktop click stability over local IPC; no SteamVR client."""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import socket
|
||||
|
||||
def request(command):
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
|
||||
client.bind(''); client.settimeout(2)
|
||||
client.sendto(command.encode(), '\0ft_screens')
|
||||
response=client.recv(8192).decode()
|
||||
if response.startswith('error'): raise RuntimeError(response)
|
||||
return response
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
sub=parser.add_subparsers(dest='command', required=True)
|
||||
sub.add_parser('status')
|
||||
sub.add_parser('threshold').add_argument('pixels', type=float)
|
||||
args=parser.parse_args()
|
||||
try:
|
||||
if args.command == 'threshold':
|
||||
if not math.isfinite(args.pixels) or not 0 <= args.pixels <= 64:
|
||||
raise ValueError('Threshold must be 0–64 logical pixels; 0 disables stabilization')
|
||||
if request(f'controller-click {args.pixels:g}') != 'ok': raise RuntimeError('Threshold rejected')
|
||||
print(json.dumps(json.loads(request('controller-click?')), indent=2))
|
||||
except (OSError, ValueError, RuntimeError) as error: parser.exit(1, str(error)+'\n')
|
||||
if __name__ == '__main__': main()
|
||||
+51
-14
@@ -1,13 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install everything on the Steam Frame: the build container, Frametop (multi-screen
|
||||
# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode and the
|
||||
# Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
|
||||
# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode, our own
|
||||
# eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
|
||||
# for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
|
||||
# (It also works from a PC over SSH; see "Developing from a PC" in the README.)
|
||||
#
|
||||
# Usage: ./install.sh [--yes] [--no-bluetooth]
|
||||
# --yes don't ask; installs gaze mode, skips the Bluetooth fixes and the SteamVR
|
||||
# restart
|
||||
# --yes don't ask; installs gaze mode, and our eye tracker if sudo can run without
|
||||
# a password prompt; skips the Bluetooth fixes and the SteamVR restart
|
||||
# --no-bluetooth don't offer the Bluetooth fixes
|
||||
set -euo pipefail
|
||||
|
||||
@@ -19,7 +19,7 @@ for arg in "$@"; do
|
||||
case $arg in
|
||||
--yes) assume_yes=1 ;;
|
||||
--no-bluetooth) bluetooth=0 ;;
|
||||
-h|--help) sed -n '2,11p' "$0"; exit 0 ;;
|
||||
-h|--help) sed -n '2,12p' "$0"; exit 0 ;;
|
||||
*) echo "unknown option: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
@@ -38,6 +38,13 @@ ask() { # ask "question" default(y|n)
|
||||
answer=${answer:-$2}
|
||||
[[ $answer =~ ^[Yy] ]]
|
||||
}
|
||||
# Can sudo run here without a password prompt (SUDO_ASKPASS, the repo's .env, a recent sudo)?
|
||||
# See frame_sudo in scripts/_env.sh.
|
||||
sudo_quiet() {
|
||||
{ [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; } && return 0
|
||||
grep -q '^steamos_root_pwd=.' "$REPO_ROOT/.env" 2>/dev/null && return 0
|
||||
on_frame 'sudo -n true' 2>/dev/null
|
||||
}
|
||||
|
||||
if [ "$FRAME_LOCAL" = 1 ]; then
|
||||
echo "Installing on this Steam Frame from $FRAME_REPO"
|
||||
@@ -50,7 +57,7 @@ if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
|
||||
echo "that need SteamVR start with it if it isn't running now."
|
||||
fi
|
||||
|
||||
step "1/9 distrobox (container tool, installed in your home folder)"
|
||||
step "1/10 distrobox (container tool, installed in your home folder)"
|
||||
if on_frame 'test -x ~/.local/bin/distrobox'; then
|
||||
echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
|
||||
else
|
||||
@@ -60,25 +67,25 @@ else
|
||||
cd ~/dev/src/distrobox && ./install --prefix ~/.local'
|
||||
fi
|
||||
|
||||
step "2/9 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
"$root/setup/dev-container.sh"
|
||||
|
||||
step "3/9 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
"$root/desktops.sh" relay install
|
||||
|
||||
step "4/9 3D mouse: SteamVR driver"
|
||||
step "4/10 3D mouse: SteamVR driver"
|
||||
"$root/pointer/driver/build.sh"
|
||||
"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
|
||||
|
||||
step "5/9 3D mouse: pointer helper service"
|
||||
step "5/10 3D mouse: pointer helper service"
|
||||
"$root/pointer/helper/build.sh"
|
||||
"$root/pointer/helper/run.sh" install
|
||||
|
||||
step "6/9 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
"$root/power/build.sh"
|
||||
"$root/power/run.sh" install
|
||||
|
||||
step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
|
||||
step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
|
||||
"$root/screens/build.sh"
|
||||
"$root/desktops.sh" install >/dev/null
|
||||
"$root/input-settings/install.sh"
|
||||
@@ -87,14 +94,44 @@ step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Framet
|
||||
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
|
||||
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
|
||||
|
||||
step "8/9 gaze mode (optional, experimental: the pointer goes where you look)"
|
||||
step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
|
||||
gaze=0
|
||||
if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
|
||||
"$root/gaze/run.sh" install
|
||||
gaze=1
|
||||
else
|
||||
echo "skipped. Install later with: gaze/run.sh install"
|
||||
fi
|
||||
|
||||
step "9/9 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
# Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
|
||||
# uses SteamVR's eye tracker.
|
||||
step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
|
||||
if [ "$gaze" = 0 ]; then
|
||||
echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
|
||||
elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
|
||||
echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: gaze/tracker/install.sh"
|
||||
elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo), and it downloads about 165 MB (numpy, OpenCV)." y; then
|
||||
if "$root/gaze/tracker/install.sh" install; then
|
||||
# Configs made before GAZE_TRACKER=auto say steam, which keeps SteamVR's.
|
||||
tracker=$(on_frame "sed -n 's/^GAZE_TRACKER=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1")
|
||||
if [ "$tracker" = steam ] &&
|
||||
ask "Your settings pick SteamVR's eye tracker (GAZE_TRACKER=steam, the old default). Use ours instead?" y; then
|
||||
on_frame "sed -i 's~^GAZE_TRACKER=steam\b.*~GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when installed, else SteamVR | own | steam~' ~/.config/frametop.conf"
|
||||
tracker=auto
|
||||
fi
|
||||
if [ "$tracker" = steam ]; then
|
||||
echo "installed; your settings keep SteamVR's eye tracker (pick Own tracker on the Gaze page to use ours)"
|
||||
else
|
||||
echo "gaze mode uses our eye tracker: calibrate it once, with Calibrate on the Gaze page"
|
||||
fi
|
||||
else
|
||||
echo "our eye tracker didn't install, so gaze mode uses SteamVR's. Try again with: gaze/tracker/install.sh"
|
||||
fi
|
||||
else
|
||||
echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
|
||||
fi
|
||||
|
||||
step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
|
||||
"$root/setup/bluetooth/install.sh" install
|
||||
else
|
||||
|
||||
+20
-1
@@ -58,6 +58,7 @@
|
||||
#include <wlr/util/log.h>
|
||||
|
||||
#include "vr.h"
|
||||
#include "controller-click.h"
|
||||
|
||||
#define MAX_SCREENS 24 // screens and spare outputs
|
||||
|
||||
@@ -104,6 +105,7 @@ struct server {
|
||||
struct wl_list buffers; // tracked_buffer
|
||||
struct wl_event_source *tick;
|
||||
struct screen *pointer_focus;
|
||||
struct ft_controller_click controller_click;
|
||||
pid_t child;
|
||||
// Where typing goes: the screens after a click on one, Steam after a click on another
|
||||
// panel. The input relay grabs the keyboards while it's the screens (see keys_update).
|
||||
@@ -292,6 +294,11 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
|
||||
return;
|
||||
}
|
||||
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
|
||||
struct ft_event filtered = *e;
|
||||
if (e->screen < s->n_config &&
|
||||
!ft_controller_click_filter(&s->controller_click, &filtered, s->scale[e->screen])) return;
|
||||
e = &filtered;
|
||||
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
|
||||
struct screen *sc = s->screens[e->screen];
|
||||
struct wlr_surface *surface = sc->toplevel->base->surface;
|
||||
const uint32_t t = now_ms();
|
||||
@@ -514,7 +521,18 @@ static int control_readable(int fd, uint32_t mask, void *data) {
|
||||
unsigned code;
|
||||
int value, index, w, h;
|
||||
double scale;
|
||||
if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
|
||||
char tail;
|
||||
if (strcmp(buf, "controller-click?") == 0) {
|
||||
snprintf(reply, sizeof reply,
|
||||
"{\"supported\":true,\"threshold\":%.3f,\"held\":%s,\"dragging\":%s,\"suppressedMotions\":%lu,\"clicks\":%lu,\"drags\":%lu}",
|
||||
s->controller_click.threshold, s->controller_click.held ? "true" : "false",
|
||||
s->controller_click.dragging ? "true" : "false", s->controller_click.suppressed,
|
||||
s->controller_click.clicks, s->controller_click.drags);
|
||||
} else if (sscanf(buf, "controller-click %lf %c", &scale, &tail) == 1) {
|
||||
if (!isfinite(scale) || scale < 0 || scale > 64 || s->controller_click.buttons)
|
||||
snprintf(reply, sizeof reply, "error threshold or held controller button");
|
||||
else { s->controller_click.threshold = scale; snprintf(reply, sizeof reply, "ok"); }
|
||||
} else if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
|
||||
// A new resolution for a screen, live: KWin resizes the screen to match. (KWin makes
|
||||
// it this size times its scale; ft-floatd sends spares' sizes divided by theirs.)
|
||||
const int min_w = index - 1 < s->n_config ? 320 : 64, min_h = index - 1 < s->n_config ? 200 : 64;
|
||||
@@ -641,6 +659,7 @@ static bool setup_dmabuf(struct server *s) {
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
struct server s = {0};
|
||||
s.controller_click.threshold = 8;
|
||||
for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
|
||||
s.kb_screen = -1;
|
||||
const char *socket_name = "ft-screens-0", *control_name = "ft_screens";
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
#ifndef FT_CONTROLLER_CLICK_H
|
||||
#define FT_CONTROLLER_CLICK_H
|
||||
#include <math.h>
|
||||
#include <linux/input-event-codes.h>
|
||||
#include "vr.h"
|
||||
struct ft_controller_click {
|
||||
bool held, dragging;
|
||||
int screen;
|
||||
uint32_t buttons;
|
||||
double x, y, threshold, radius;
|
||||
unsigned long suppressed, clicks, drags;
|
||||
};
|
||||
// Hold the desktop position at press time until movement exceeds a logical-pixel
|
||||
// radius. No timer or delayed button-down. Only a hand controller's press starts
|
||||
// it: the 3D mouse drives a laser too, and its short drags must stay drags.
|
||||
static inline bool ft_controller_click_filter(struct ft_controller_click *c,
|
||||
struct ft_event *e, double scale) {
|
||||
if (e->type == FT_BUTTON && e->button >= BTN_LEFT && e->button < BTN_LEFT+8) {
|
||||
uint32_t bit = 1u << (e->button-BTN_LEFT);
|
||||
if (e->pressed) c->buttons |= bit; else c->buttons &= ~bit;
|
||||
}
|
||||
if (e->type == FT_BUTTON && e->pressed) {
|
||||
if (e->button == BTN_LEFT && e->controller && !c->held && c->threshold > 0) {
|
||||
c->held = true; c->dragging = false; c->screen = e->screen;
|
||||
c->x = e->x; c->y = e->y; c->radius = c->threshold * scale;
|
||||
} else if (e->button != BTN_LEFT) {
|
||||
c->held = false; // Multi-button gestures retain their usual semantics.
|
||||
}
|
||||
} else if (e->type == FT_MOTION && c->held && !c->dragging) {
|
||||
if (e->screen == c->screen && hypot(e->x-c->x, e->y-c->y) <= c->radius) {
|
||||
++c->suppressed;
|
||||
return false;
|
||||
}
|
||||
c->dragging = true; ++c->drags;
|
||||
} else if (e->type == FT_BUTTON && e->button == BTN_LEFT && !e->pressed && c->held) {
|
||||
if (!c->dragging) {
|
||||
e->screen = c->screen; e->x = c->x; e->y = c->y;
|
||||
++c->clicks;
|
||||
}
|
||||
c->held = false; c->dragging = false;
|
||||
} else if (e->type == FT_LEAVE && c->held) {
|
||||
return false; // Preserve the implicit grab through tiny edge excursions.
|
||||
}
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
@@ -0,0 +1,42 @@
|
||||
#include <assert.h>
|
||||
#include "../controller-click.h"
|
||||
static struct ft_event event(enum ft_event_type t, bool down, double x, double y) {
|
||||
return (struct ft_event){.type=t,.screen=0,.button=BTN_LEFT,.pressed=down,.controller=true,.x=x,.y=y};
|
||||
}
|
||||
int main(void) {
|
||||
struct ft_controller_click c={.threshold=8};
|
||||
struct ft_event e=event(FT_BUTTON,true,100,100);
|
||||
assert(ft_controller_click_filter(&c,&e,2));
|
||||
e=event(FT_MOTION,false,112,108);
|
||||
assert(!ft_controller_click_filter(&c,&e,2));
|
||||
assert(c.held && !c.dragging && c.suppressed==1);
|
||||
e=event(FT_BUTTON,false,116,111);
|
||||
assert(ft_controller_click_filter(&c,&e,2));
|
||||
assert(e.x==100 && e.y==100 && c.clicks==1 && !c.held);
|
||||
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
|
||||
e=event(FT_MOTION,false,109,100);assert(ft_controller_click_filter(&c,&e,1));
|
||||
assert(c.dragging && c.drags==1);
|
||||
e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1));
|
||||
e=event(FT_BUTTON,false,103,100);assert(ft_controller_click_filter(&c,&e,1));
|
||||
assert(e.x==103 && c.clicks==1 && !c.held && !c.dragging);
|
||||
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
|
||||
e=event(FT_LEAVE,false,0,0);assert(!ft_controller_click_filter(&c,&e,1));
|
||||
e=event(FT_BUTTON,false,800,900);e.screen=1;ft_controller_click_filter(&c,&e,1);
|
||||
assert(e.screen==0 && e.x==100 && c.clicks==2);
|
||||
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
|
||||
e=event(FT_MOTION,false,101,100);e.screen=1;
|
||||
assert(ft_controller_click_filter(&c,&e,1) && c.dragging);
|
||||
e=event(FT_BUTTON,false,101,100);ft_controller_click_filter(&c,&e,1);
|
||||
c.threshold=0;
|
||||
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
|
||||
e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1) && !c.held);
|
||||
c.threshold=8;
|
||||
e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
|
||||
e=event(FT_BUTTON,true,100,100);e.button=BTN_RIGHT;ft_controller_click_filter(&c,&e,1);
|
||||
assert(!c.held);
|
||||
// The 3D mouse's laser: a short drag stays a drag, and the release stays where it was.
|
||||
e=event(FT_BUTTON,true,100,100);e.controller=false;assert(ft_controller_click_filter(&c,&e,1) && !c.held);
|
||||
e=event(FT_MOTION,false,104,100);assert(ft_controller_click_filter(&c,&e,1));
|
||||
e=event(FT_BUTTON,false,106,100);e.controller=false;
|
||||
assert(ft_controller_click_filter(&c,&e,1) && e.x==106 && c.clicks==2 && c.suppressed==1);
|
||||
}
|
||||
@@ -1741,6 +1741,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
|
||||
e.type = FT_BUTTON;
|
||||
e.button = LinuxButton(ev.data.mouse.button);
|
||||
e.pressed = ev.eventType == vr::VREvent_MouseButtonDown;
|
||||
e.controller = IsHandController(ev.trackedDeviceIndex);
|
||||
at();
|
||||
if (!e.pressed && s.titleCarry) e.x = s.carryX, e.y = s.carryY, s.titleCarry = false;
|
||||
if (e.pressed) {
|
||||
|
||||
@@ -25,6 +25,7 @@ struct ft_event {
|
||||
double x, y; // FT_MOTION: buffer pixels from the top left
|
||||
uint32_t button; // FT_BUTTON: linux BTN_*
|
||||
bool pressed;
|
||||
bool controller; // FT_BUTTON: from a hand controller's laser (not the 3D mouse's)
|
||||
double dx, dy; // FT_SCROLL: notches (positive dy: scroll down)
|
||||
uint32_t key; // FT_KEY: linux KEY_* from our keyboard (pressed: down or up)
|
||||
};
|
||||
|
||||
+18
-3
@@ -6,7 +6,22 @@
|
||||
# `distrobox enter`.
|
||||
box=${FRAME_BOX:-dev}
|
||||
export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
|
||||
[ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ] && exit 0
|
||||
running() { [ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ]; }
|
||||
running && exit 0
|
||||
podman container exists "$box" 2>/dev/null || exit 0 # not created yet: setup/dev-container.sh does that
|
||||
exec systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \
|
||||
podman start "$box" >/dev/null
|
||||
since=$(date -u +%FT%T)
|
||||
systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \
|
||||
podman start "$box" >/dev/null || exit
|
||||
|
||||
# Every start runs distrobox-init in the container, and `distrobox enter` only waits for it
|
||||
# when it starts the container itself. The first start installs what distrobox needs and sets
|
||||
# up passwordless sudo, which takes a minute or more; until then sudo in the container asks
|
||||
# for a password (issue #9). Later starts take a few seconds.
|
||||
for i in $(seq 600); do
|
||||
podman logs --since "$since" "$box" 2>&1 | grep -q '^container_setup_done' && exit 0
|
||||
running || break
|
||||
[ "$i" = 10 ] && echo "setting up the $box container (the first start takes a few minutes)" >&2
|
||||
sleep 1
|
||||
done
|
||||
echo "the $box container didn't finish starting; see: podman logs $box" >&2
|
||||
exit 1
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
"$root/scripts/frame.sh" -C screens 'mkdir -p build; gcc -std=c11 -Wall -Wextra -Werror tests/controller-click-test.c -lm -o build/controller-click-test && build/controller-click-test'
|
||||
@@ -38,7 +38,7 @@ POINTER_GAZE_SHOW=1 # gaze mode, with POINTER_GAZE_DOT=moving: the dot sh
|
||||
POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
|
||||
POINTER_HEAD_DEADZONE=0.5 # keyboard clicks at the gaze (Meta+J, Meta+K): degrees the head turns before the held dot moves with it
|
||||
POINTER_KEY_TAP=0.25 # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
|
||||
GAZE_TRACKER=steam # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated with Calibrate on Frametop Input Settings' Gaze page)
|
||||
GAZE_TRACKER=auto # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
|
||||
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
|
||||
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
|
||||
HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on
|
||||
|
||||
@@ -45,9 +45,9 @@ fi
|
||||
"$distrobox" enter dev -- bash -c '
|
||||
set -euo pipefail
|
||||
echo "installing ${#@} packages (already-installed ones are skipped)"
|
||||
sudo dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; }
|
||||
sudo -n dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; }
|
||||
# OpenVR programs built here (the pointer helper and probe) look for the runtime at /opt/steamvr.
|
||||
[ -e /opt/steamvr ] || sudo ln -s /run/host/opt/steamvr /opt/steamvr
|
||||
[ -e /opt/steamvr ] || sudo -n ln -s /run/host/opt/steamvr /opt/steamvr
|
||||
echo "dev container ready: $(. /etc/os-release; echo $PRETTY_NAME), glibc $(ldd --version | head -1 | grep -oE "[0-9.]+$")"
|
||||
' dev "$@"
|
||||
EOF
|
||||
Reference in new issue
Block a user