mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-15 00:00:42 +02:00
dfd1bbca0fad72bee86eef7f31eb4534cdbaddcd
32
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DeeJanuzandClaude Opus 5.5
|
82fbc6fc1b |
Docs: the relay's mouse buttons follow the pointer, and media keys reach the desktop
reference.md's ft-screens keys paragraph, the relay's top docstring and hazards.md's key releases section now say that a mouse button passed through as a key in pointer mode goes to the screen the pointer is on, not where typing goes, only while one shows and Frametop isn't paused, that ft-screens lets its release through and releases it itself when the pointer leaves the screens, its screen hides, or Frametop pauses, and that a relay that starts releases mouse buttons too. hazards.md adds that the button still reaches the virtual mouse, which gamescope reads, and that a keyboard's media keys come from its Consumer Control node, which stays ungrabbed, so they reach gamescope as well as the desktop even while typing goes to the desktop. design.md says why the buttons follow the pointer: as keys, a release was lost whenever typing moved, and wlroots counts presses per button. reference.md names scripts/test-relay-buttons.sh, and keys-test's wider scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
8bd2ac4c92 |
Input relay tests: which keys and mouse buttons reach the desktop
keys-test gains two fake nodes with volume keys, a keyboard's Consumer
Control node (USB) and the headset's gpio-keys (BUS_HOST), both as if
their volume keys were remapped. Its virtual devices now record what
they're sent, and the relay's volume handling is a recorder, so no
wpctl runs.
New checks: a relay that starts releases the modifiers and mouse
buttons on the desktop. A media key reaches the desktop and no virtual
device, and one held over the once-a-second check stays down until its
node lets go. A volume stand-in goes to the volume handling only; an
original volume key a remap missed, and the headset's KEY_SELECT, reach
nothing. Clicks with POINTER=0, clicks while paused, and a pass-through
mouse's clicks don't reach the desktop; a plain click in pointer mode
is the helper's alone. A side button mapped to "key" in pointer mode
reaches the desktop and the virtual mouse, and its release still
reaches the desktop after a pause starts.
51 checks, all passing. Against the relay at the PR merge (
|
||
|
DeeJanuzandClaude Opus 5.5
|
030ef2faf1 |
Input relay: a relay that starts releases mouse buttons on the desktop too
A relay that went away with a key-mapped mouse button down left it down in ft-screens, and ft-screens takes no second press of a button it holds. The start's release loop, which let go of only the modifiers, releases BTN_MOUSE..BTN_TASK as well. ft-screens ignores the release of a button it doesn't hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
44c0b2d796 |
Input relay: mouse buttons go to the desktop only when pointer mode passes them through
PR #41 sent ft-screens every key and button a device has. So with POINTER=0 every physical click went to ft-screens as well as to the virtual mouse, clicks were sent while paused, a pass-through mouse's clicks landed wherever a laser last was, and gamepad, joystick, digitizer and BTN_TRIGGER_HAPPY codes became desktop buttons. to_screens() sends keys below BTN_MISC as before, mouse buttons (BTN_MOUSE..BTN_TASK) only when asked to, and nothing else, but the release of anything the desktop has down. The pointer branch asks for it only in pointer mode, where reaching it means the button's map says "key": the PR's case, a side button passed through for Back. Clicks for the virtual controller or the virtual mouse don't go to ft-screens, and a key-mapped button held into a pause still has its release sent. Keys from KEY_OK (0x160) up stay in the relay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
977e9a0950 |
Input relay: only a keyboard's media keys leave a volume node, for the desktop
PR #41 let volume-role nodes fall into the pointer-device branch. Those nodes are never grabbed: the headset's gpio-keys (its KEY_SELECT click button and volume), keyboards' and mice's Consumer Control nodes, and pmic_resin. So media keys were re-emitted on the virtual keyboard and gamescope got them twice, the headset's click button went to ft-screens, the device button maps applied to them, and after a partial volume remap failure an original volume key went out on the virtual keyboard, where gamescope reads it (a volume key with nothing focused aborts it). A volume node's keys stop there again. Only a USB or Bluetooth node's non-volume keys (the media keys) go on, to ft-screens, like a pass-through keyboard's: ft-screens types them only while typing goes to the desktop, and reconcile_desktop_keys releases one whose node goes away while it's held (physically_down reads volume nodes too). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuz
|
82979e4423 | Merge PR #41 from SuperTuxii/allow-more-keys: input: Allow sending keys >= BTN_MISC and allow sending keys from nodes with volume role | ||
|
DeeJanuzandClaude Opus 5.5
|
2ede132b62 |
Input relay tests: what standing down sends, in order
pause-buttons-test checked the pointer's driver_down set and filtered the sent commands down to "btn" lines. On the code before the fix it stopped at a missing attribute, before any behavior was checked, and it couldn't see where "hide" went. The order matters: a release after "hide" wakes a helper that wakes on any btn. It now compares the exact commands stand_down sends (["btn trigger 0", "hide"] for a held left button) and adds the cases that were missing: the pointer already off with a button held (the idle timeout, pointer_toggle), a mapped controller button and a key combination, and gaze holds, which are the helper's to end. Its labels now say what each case is: they called a plain mouse press a gaze drag. Against the relay before the fix, 5 of its 11 checks fail; against the contributor's commit, the 2 pointer-off ones. keys-test gains a pause check through the relay's main(): in pointer mode, a left button held into a pause comes up, then the pointer hides, its release during the pause reaches no one, and clicks go to the helper again after resume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
2a2bca336c |
Input relay: standing down always hides after a release
stand_down sent "hide" only while the relay's pointer was on. It can be off with a button still held: the idle timeout or pointer_toggle during a press. Then the pause sent "btn trigger 0" on its own, the helper woke on it (any btn or scroll wakes it), connected the virtual controller and took the right hand role during the game, and nothing hid it again: the relay doesn't tick a paused pointer. Now every release stand_down sends (held buttons, the system and claim pulses, a scroll pulse) is followed by "hide". The helper also stops waking on releases (next commit), but the relay and the helper update separately, so the relay can't count on that. The comments now say what driver_down covers: the left, right, middle and back actions from a mouse, a mapped controller button or a key combination. Gaze holds go to the helper as gazekey, gazedrag and precision, and it ends them itself when the pointer hides. The docstring also records the one case left: a gaze-mode press held back for aiming clicks once if the helper reads "hide" a loop after the release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
0x1f6
|
8a77364767 |
Input relay: standing down releases the driver buttons the pointer holds
Pausing drops button releases (do_action's works_paused check), and
stand_down only ended the pulses it tracks itself: a gaze drag or gazekey
click held down when the pause started left its virtual controller button
(trigger, b, x, joystick) down until resume, pressing things in Steam.
The pointer now remembers which driver buttons it pressed and hasn't
released, and stand_down releases them before it hides. On pre-fix code the
new test aborts ('Pointer' has no driver_down); with the fix all 8 checks
pass, including two buttons down at once and a stray second stand_down.
input/test/pause-buttons-test.py
|
||
|
SuperTuxii
|
b1b1a16f5b |
input: Allow sending keys >= BTN_MISC and from nodes with volume role
Allow sending keys >= BTN_MISC to screens by handling them as pointer buttons when receiving them from input relay in the compositor. Also allow sending keys other than volume keys from nodes with the volume role. Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com> |
||
|
|
9d7c8a0b91 |
Experimental (#29)
* Input relay: typing with the pointer helper down no longer ends the relay Typing on a pass-through keyboard tells the helper "typing". With the helper not running (SteamVR off), that send raised ConnectionRefusedError and the relay exited, dropping every grab until systemd restarted it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-steam: open Steam's menu through Steam's own UI steam/ft-steam menu opens the SteamVR dashboard on Steam's menu, or closes the dashboard if it's up, without pointer mode: it asks Steam's UI over its debugging port to show its dashboard overlay (ShowVROverlay, what Steam calls itself) and focus the Steam frame's left menu (MenuStore.OpenMainMenu). ft-steam check says whether those calls still exist, and update-check.py runs it, since a Steam client update can rename them. The CDP client moves from display-settings/steam_settings.py to steam/steamui.py, so both use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Shortcuts: Steam menu, commands, and modifier taps Key combinations (and mouse and controller buttons) get two new actions: - steam_menu: Open Steam menu / close dashboard (steam/ft-steam menu). - command:CMD: run CMD with sh -c, as the relay's service, with layout/, float/ and steam/ on its PATH. Input Settings offers it for key combinations as Run a command... Both work without pointer mode. A modifier on its own is now a key combination too: a tap, pressed and released with no other key, mouse button, or scroll in between. A bound tap sends the desktop F24 before the release, so Plasma's launcher stays shut. The defaults gain a Meta tap for the Steam menu; this replaces META_DASHBOARD, which only worked in pointer mode. input/test/keys-test.py runs the relay against fake devices with every outgoing socket renamed, so it's safe next to the live relay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Relay: share a key combination's Meta release with frame-voice A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release from the desktop, and the relay skipped share_key for it too. frame-voice saw Meta go down on @frametop_keys and never come up, so it held all dictated text back, waiting for that release. Keys of grabbed keyboards are now shared as pressed, before key_binding() decides what the desktop gets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: ft-cutouts, the hand cutouts without pinches and grips hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient user units with ft-hands' new --no-gestures: hands are published for ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks or drags and a closing hand doesn't raise the tracking rate. It needs a build and ft-camd's capabilities, not hands/run.sh install. Its units conflict with ft-handsctl's, so each stops the other, and they stop with SteamVR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-cutouts status: only the current run's tracker lines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: say why gaze mode can't work yet, and open the calibration whenever it's missing Turning gaze mode on without a calibration opened Calibrate only on the off-to-on change, and only if it could open right then. With the headset off, the eye tracker silent, or the panel not built, or with gaze mode already on when the gaze service started, nothing opened and nothing said why: the pointer just stayed a mouse. - The gaze service now checks every second: gaze mode on, no calibration for the tracker in use, eyes seen -> the full calibration opens. One that closes unfinished opens again only after the headset comes off and on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that fails retries every 10 s. - Its status says why gaze mode can't work yet (checks.problem): not calibrated and opening, open, closed unfinished, or can't open and why. - Input Settings shows that under the Gaze pointer switch, along with the gaze service not installed or not running and our tracker missing its frame grabber. - ft-gazectl on notes a missing calibration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken A user on a fresh install got "Calibration failed: only 0 of 21 dots" with no reason. The installer never installed our tracker, so gaze used SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an eye for most of the look. The panel just showed a red ring. - install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh) after gaze mode, yes by default; it needs sudo, so --yes runs it only when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker. Configs that still say GAZE_TRACKER=steam (the old template) are asked whether to switch. - GAZE_TRACKER=auto, the new default: ours when it's installed (the frame grabber, its unit, and ft-eyes' Python), else SteamVR's. ft-gazed rechecks every second, so installing it switches over. Input Settings lists Own tracker first as recommended, and says how to install it when it's missing (checking the host's /etc through /run/host from the dev container). - The calibration panel has a note line, orange over the instructions: why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply for ours), what a click is still waiting for after 1.5 s, and a failed calibration's most common reason, which the Gaze page shows too. steady_samples keeps the same samples as before (checked on 2037 windows of recordings); a lost eye is named before a blink, since its openness reads 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * README: link the Frametop Discord Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wait for a new container to finish setting up before entering it container-up.sh starts the dev container in a scope of its own, so distrobox enter finds it running and skips its wait for distrobox-init. On a fresh install, init was still setting up passwordless sudo when dev-container.sh ran sudo dnf install, and sudo asked for a password with no terminal to read it from. container-up.sh now waits for container_setup_done itself, and the container's sudo calls use -n, so a password prompt fails at once with a clear message. Fixes #9 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Prevent small desktop overlay pointer movements from starting a drag * Clear reported drag state on controller release * Click stability: only a hand controller's press starts it The 3D mouse drives SteamVR's laser through the ft_pointer virtual controller, so its events reach the screens the same way a controller's do. The filter held every press, which turned the mouse's short drags (selecting a character or two, nudging a slider) into clicks. Mark button events from hand controllers and start the filter only on those. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input relay: retry a new device until udev gives it to the input group A new /dev/input node is root:root 0600 until udev applies GROUP=input. The scan probed each new node once and marked it seen even when the open failed, so a node caught in that gap was never opened. Behind a KVM, a switch brings back a hub of devices at once: on the Frame, four nodes failed with EACCES in one switch, the keyboard was never grabbed, and its keys went to gamescope instead of the desktop screens. A node that isn't readable yet now waits for the next scan. * Screens: take a screen's overlays from one copy in the catcher While a button pressed on a screen is held, UpdateCatcher checks every tick whether the laser is still on one of the screen's overlays. It built that list from s.All().begin() and s.All().end(), but All() returns a std::array by value: iterators into two different temporaries, which is undefined behaviour. A clang build of ft-screens got a garbage length, threw std::length_error, and aborted on the first click, taking KWin and the desktop with it. * Gaze: leave SteamVR's gaze action alone during VR games From curiousjtuber's PR #13: with the gaze service running, SteamVR restarted its eye tracker every 10 to 13 s of Beat Saber, as if the headset came off, and each restart took input focus from the game. The PR stopped every read in a game. Only the action path reaches SteamVR (UpdateActionState on the gaze set at overlay-global priority, then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are read-only files. So only the action is skipped while a scene app runs, and gaze keeps moving the pointer over the dashboard in a game. The action source is only used with --source action. Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: --record-hz, and the hand recorder's design (hands/rec/DESIGN.md) ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10). DESIGN.md lays out the recorder: the headset panel, the session runner and its script, the files, review and export, consent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: ft-handpanel, the hand recorder's headset panel A head-locked SteamVR overlay for the hand recorder (hands/rec/DESIGN.md): 1.2 m ahead, 12 degrees up, 36 degrees wide, drawn with stb_truetype into three shared DMA-BUFs as ft-gazepanel does. It shows the title, step, wrapped instruction, note, countdown, hand chips, near/far bar and a "Paused" cover, driven over @ft_handpanel. It also places the touch target, a 2 cm dot in its own overlay fixed in the room where the head was at the first command for that point, and logs head and controller poses to poses.jsonl at 250 Hz from a thread of its own. Both threads take one lock around OpenVR calls. --no-vr prints each picture's state to stdout (and --dump writes the pictures), for testing without a headset. hands/rec/build.sh builds it in the dev container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the recorder's worn check goes by the panel's backlight, as frame-job does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the hand recorder's session runner and guided script hands/rec/session.py runs a recording session from script.json: it starts ft-camd and a tracking ft-hands as transient units only if they aren't running, records each section as one take (ft-hands --record-only at 10 sets/s, a new sets-N.bin after each pause), drives ft-handpanel, and writes session.json, calibration.json (identifying fields removed), prompts.jsonl and take.json. Feedback comes from the live hands file and, in the controller sections, from the panel's device poll. It also runs from the command line (--dry-run, --speed, --ring, --no-start). hands/rec/script.json: 11 sections, about 9 minutes without the object and controller sections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the hand recorder's window, review and export hands/rec/ft_handrec.py + main.qml (Kirigami, dev container; host launcher hands/rec/ft-handrec): consent (CONSENT.md, asked again when its version changes; profile.json with a random contributor id), the before-you-start checklist with the lighting and free-space checks, the session controls (Space pauses, Esc stops), review with a frame-set viewer that deletes ranges, takes and sessions, export with progress and cancel (warns while the headset is worn), and the upload page (UPLOAD.md, the huggingface-cli command; HF_DATASET is a placeholder). --dry-run runs sessions without processes, for testing. hands/rec/takes.py (standard library): indexes sets.bin and sets-N.bin without reading pixels, reads one set's cameras, keeps deleted ranges in take.json, and exports: deleted sets left out, zstd -10 -T2 at nice 19, manifest.json and SHA256SUMS, nothing left behind on cancel. CONSENT.md and UPLOAD.md are drafts pending a legal review; the window says contributions aren't open yet. The dev container gains zstd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: upload from the hand recorder's window, export checks, a rehearsal hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+) checks an export before upload and when it's received: SHA256SUMS, an allow-list of files, the manifest's schema and keys, the consent version, a uuid4 contributor, no identifying fields in calibration.json or device.json, every sets.bin.zst decompressed to its end as a stream with each FHSET01 header checked against the manifest, jsonl lines, the total size. It decompresses with compression.zstd, zstandard or the zstd program. validate.py DIR [--json]. hands/rec/hub.py uploads an export with huggingface_hub, as a pull request to contributions/<contributor>/<session>: validate first, refuse a repeat of the same export, check the login (whoami) and access (auth_check), upload_folder(create_pr=True) with the manifest summary as the description, then record the PR under "uploads" in session.json. Errors are explained (terms not accepted, not found, 401/403, network). --dry-run makes no network calls. FT_HANDREC_DATASET overrides HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real upload needs FT_HANDREC_ALLOW_UPLOAD=1. The Upload page shows the login with "Check again" and how to run hf auth login in a terminal (the token never enters the window), then Upload with a phase, progress and Cancel (hub.py as a child process), the PR link, and a warning for an export uploaded before. The manual command stays as the fallback. ft-handrec --hub-dry-run. session.py also saves device.json: cv.cad_from_cal and head from /persist/device_config.json, the labeller's shape, nothing identifying; export copies it. Session ids with a -N suffix are accepted everywhere. hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays 30 s of a capture into a ring, session.py records a short test script with ft-handpanel --no-vr and a tracker, then export, validate and a dry-run upload (--repo ID uploads for real). It runs in one frame-job scope, deletes its data and stops its processes, also on Ctrl+C. hands/rec/tests/test_validate.py covers good and broken exports and hub.py without the network. The dev container gains python3-huggingface-hub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * README: Frametop doesn't work on the SteamOS beta yet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit |
||
|
DeeJanuzandClaude Opus 5.5
|
20e8d6254a |
Merge gaze-calibration into experimental
The gaze checks and calibration in a headset panel (quick, five, full, headset fit; shared-buffer drawing, click-to-capture), keyboard and mouse gaze clicks (Meta+J/K, the mouse's buttons alike, mouse moves only while a button is held, double right/Meta+K tilts a drag), one 55 degree learning limit with a quick check past it, eye presence for "the headset went on", the gaze probe as a development tool, and the relay releasing keys the desktop has down that no keyboard holds. Conflicts with the profiles: the relay keeps known_action/needs_pointer and the gaze defaults (Meta+J/K and the float key); Input Settings keeps the profile actions everywhere and the gaze actions in key combinations only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
2ae10a3b65 |
Input relay: release keys the desktop has down that no keyboard holds
After a gaze calibration, Meta stayed down in KWin: typing opened the overview, and clicks on the desktop did other things. A key can be left down there when its device vanishes with it held (release_held only let go of it on the relay's own devices; the Z3's keyboard dropped out twice right then) or a release goes astray. The relay now remembers which keys it told ft-screens went down, and once a second releases any no device holds (EVIOCGKEY), and the key combinations forget a Meta or modifier no device holds. A relay that starts releases the modifiers on the desktop, for keys an earlier one left down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
bfc044c1b0 |
Try every volume key when one swap fails
remap_volume stopped at the first EVIOCSKEYCODE_V2 that failed, so the volume entries after it were never tried. On a keyboard where swapping volume up failed, volume down stayed a real key that gamescope reads, and one press with nothing focused aborts gamescope and the VR session. Restoring had the same hole: a failed entry left the later stand-ins in place after the relay exited. Now every entry is tried and the first failure is raised at the end. take_volume already marks the device remapped on that error, so the relay routes the stand-ins that did swap and restores them on exit. Checked against a fake keymap with a stubbed fcntl.ioctl: with volume up's swap failing, volume down now swaps (it stayed real before), the same for restore, and full swaps, restores and the no-keymap case come out as before. Found by 0x1f6 in PR #8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
0x1f6andClaude Opus 5.5
|
9c2fccf0a0 |
Survive a malformed control datagram
One bad datagram on the control socket ended the whole relay. Its
dispatch in handle_control ran unguarded in the main loop, so any
exception propagated out of main() and the process exited. The simplest
trigger is "watch abc": float(words[1]) raises ValueError, and the relay
died with
ValueError: could not convert string to float: 'abc'
at handle_control, via the bare handle_control(now) call in the
select loop
The control socket is an abstract socket bound to @frametop_relay.
Abstract sockets carry no permissions, so every local process can send
to it; nothing authenticates the sender. Dying from a datagram was bad
in three ways:
- Every grab is lost. Physical devices go back to gamescope and SteamVR,
which read them themselves, so the mouse types into Steam and the
desktop at once, and Meta+Shift shortcuts fire on both sides.
- The volume-key takeover is lost. gamescope aborts on a volume key when
no window has keyboard focus (wlr_seat_keyboard_notify_enter asserts on
a null focus surface), which ends the whole VR session - the exact
failure the relay exists to prevent.
- systemd restarts the service, but Type=notify with READY only after
the virtual devices exist makes the restart a visible hiccup, and a
crash loop from repeated bad datagrams would flap SteamVR's input.
Reproduced by running the relay with stubbed uinput devices on a
non-Linux host and sending "watch abc" to the control socket: it exited
on the first datagram after answering "devices" correctly. After the
fix, the same exchange gets a log line ("bad control datagram ...") and
the relay keeps answering.
The fix wraps each datagram's dispatch in try/except inside
handle_control's loop, so one malformed message is logged and skipped
while the rest of the queue is still processed. Parsing of the common
helper commands (vrbtn, vrhello, gazeawake, keyboard) keeps its own
guards; the catch-all is only a backstop for anything the guards miss,
including float() on a non-numeric argument to "watch" and "vrcapture".
Adapted for experimental (from PR #8): the guard also wraps the textfield
command, which experimental added to this dispatch after the PR's base.
(cherry picked from commit
|
||
|
DeeJanuzandClaude Opus 5.5
|
858dbe1562 |
Profiles: named layouts that open apps
A profile is a named layout plus the screens it hides and its apps' windows (docs/profiles.md). ft-layout save captures them (ft-floatd's "windows", after the KWin script reports every window as it is now); use opens them: the screens move, open windows of each app go to their places (on a screen, maximized or not, or floating), and missing apps start, once and then again for each window still missing 3 s after the first. Nothing closes. The desktop starts in FT_PROFILE or default_profile (ft-layout start, from the session script). Each profile gets a launcher entry (Frametop: NAME, in SteamVR's Launch a program list) that switches to it or starts the desktop in it. The relay's profile:NAME action and Input Settings' "Open profile" entries put one on a key, mouse button, or controller button. Display Settings' Layout page becomes Layout & profiles: Save as profile, Open profile, the profile's apps, and Start in profile. Plasma's own session restore is off in the Frametop session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuz
|
6ea70f32bc |
Merge branch 'experimental' into gaze-calibration
# Conflicts: # input-settings/ft_input_settings.py # input-settings/main.qml # input/input-relay.py |
||
|
DeeJanuzandClaude Opus 5.5
|
dc3a1b8208 |
Keyboard clicks at the gaze: Meta+J and Meta+K
gaze_left and gaze_right (Meta+J and Meta+K by default) click where you look.
A tap clicks where the dot was at the press and tells the gaze tracker it was
right there. Held, the pointer stays put in your view, so turning your head
carries it onto the target; the release clicks there and the correction is a
lesson (judged by the net correction, not the head's path). Held still for
POINTER_GAZE_HOLD it's a real press that the head drags, and Meta+K while
Meta+J aims presses where the dot is now, to correct and then drag.
A Meta combination now also sends the desktop an F24 press and Meta's release
at once: Meta's release no longer opens Plasma's launcher, and the click isn't
Meta+click (KWin's window move and resize, which swallowed right clicks). The
desktop gets Meta back for the next key if it's still held.
Also fixes
|
||
|
DeeJanuzandClaude Opus 5.5
|
1e83f96b0f |
Float key in the input relay, docking everything, and the plan for profiles
The input relay owns the float key now: float_toggle (Meta+Shift+F unless the rules have their own key combinations) floats the window under the pointer, or the active one over the wallpaper, and docks it if it floats. dock_all puts every floating window back. Both are mappable to mouse and controller buttons, and work without pointer mode. The KWin script no longer registers a shortcut, and ft-floatd drops the old one. Key combinations move to Input Settings' Keyboard page. docs/floating-windows.md records the decisions from 2026-09-30 (the title bar button, Launch as Standalone, profiles), and docs/profiles.md plans profiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
e16b7588e2 |
Gaze mode is a mouse feature: drop its controller parts
Steam reads the Frame controllers itself, outside SteamVR's bindings, and every press and release it sees takes SteamVR out of laser mode, so controller clicks at the gaze can't be done cleanly (docs/gaze-controllers.md, from the gaze-first branch's tests). - Gaze precision and Gaze drag take a mouse button or a key combination only; the controller source, its aim steering, and POINTER_PRECISION_GAIN, POINTER_PRECISION_DEADZONE and POINTER_GAZE_DRAG_GAIN are gone. - The gaze actions (including Gaze pointer on/off) can't be mapped to controller buttons: the relay ignores them there and doesn't ask the helper for those buttons, and Input Settings no longer offers them. - Last used wins in gaze mode too: picking up a controller hands it the laser, as docs/design.md already said. - The gaze dot shows all the time (POINTER_GAZE_DOT=always, the default; moving brings back the old behaviour), with a switch on the Gaze page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
d666fa031f |
Gaze first: precision and drag buttons, key combinations, pointer role
Two new actions for mouse buttons, controller buttons, and key
combinations: gaze_precision (hold: the pointer stops where you look and
the button's device steers it, a controller by where it points at
POINTER_PRECISION_GAIN, the mouse by its moves; release: click there) and
gaze_drag (the same with a real press at once, dragging until the
release). The relay sends "precision|gazedrag <source> 1|0" to the
helper, which runs them through the same holds as pinches and grips.
- Key combinations on any keyboard ("key_bindings" in the rules, e.g.
Ctrl+Alt+G for gaze on/off): the last key isn't typed.
- POINTER_GAZE_MOUSE: in gaze mode the left button is a precision button
(the default, as before) or clicks right away (direct).
- In gaze mode a moving controller no longer takes the pointer away.
- POINTER_ROLE (right, left, stylus): the driver takes a "role" command,
so the pointer can stay off the hand holding the precision controller,
which would otherwise take the role back. Needs the rebuilt driver.
- Input Settings: the actions on the Controllers and Buttons pages; on the
Gaze page the mouse choice, the role, the precision sliders, and key
combinations (captured from any keyboard).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DeeJanuzandClaude Opus 5.5
|
5ca4160a7e |
Hands: fixes from the second headset test (pinches)
- The palm-down filter is off by default: the user's deliberate pinches, hand raised, read 0.90-0.99, like typing. - Typing is told apart by the keyboard instead: the input relay sends the pointer helper "typing" on key presses, and it takes no pinch within POINTER_PINCH_TYPING (1 s) of one. - Grips are still held to hands raised (POINTER_GRIP_BELOW, 0.35 m below the eyes); pinches aren't, since the user's own sat 0.35-0.45 m below, elbow resting. - A grip doesn't begin with the thumb on the index tip: that's a pinch with the other fingers curled, which was taken for a grip. - A pinch's point is the index and middle knuckles: the tips' midpoint moved 1-2 cm as the pinch opened, dragging every release off its press. - ft-hands --gesture-log prints what the detectors measure, 10 times a second. Pinches still aren't reliable enough to use; hand tracking is parked for now in favour of the controllers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
5714978e65 |
Add a keyboard for text fields on the desktop
Fixes #7: SteamVR's keyboard never came up for the desktop's apps, and opening it for our panels doesn't work well on the Frame (it's Steam's own panel, mounted in the dashboard's scene, it follows the laser between panels, and it takes the controllers over to SteamVR's laser). - KWin starts input/ft-textinput as the desktop's input method. It tells the input relay when a text field gains or loses focus, and the relay asks ft-screens to open or close the keyboard. The session drops the QT_IM_MODULE=xim and GTK_IM_MODULE=xim that the gamescope session sets, or Qt and GTK apps never report text fields. - The keyboard is ft-screens' own panel (screens/keyboard.cpp): a US laptop layout, typed with a controller's laser or the 3D mouse. It opens 0.7 m in front of you, below your eyes and facing you. It has a grab bar to move it, a Close key, latching Shift, Ctrl and Alt, and repeat on a held key. It's drawn into shared DMA-BUFs, so it doesn't flicker. Its keys reach the focused screen as key presses, so every app takes them. - It steps aside while the Steam menu or Steam's own keyboard is up and comes back after. A layout reset closes it, and it doesn't open without a head pose. - Frametop Input Settings has a Keyboard page: open it for every text field, only while no keyboard is connected (the default), only from a mapped button (the new Open/close keyboard action, for mice and controllers), or never. A switch keeps it open until you close it. - The pointer helper treats every frametop.* overlay as a real panel. The keyboard's shared texture reports 0x0 like SteamVR's scene-graph controls, and the helper had given it their wide catch radius. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
0d50efec45 |
Map the Frame controllers' buttons; make gaze clicks correctable
The Frame controllers aren't input devices on the host, so the pointer helper reads them with SteamVR input (vrbuttons.h, actions/), one action set per button, and sends presses to the relay, which does the mapped action like for a mouse button. Only mapped buttons are taken, at an overlay-global priority (SteamVR's experimental "Enable global input from overlays"), and only outside games unless In games is on. Frametop Input Settings gets a Controllers page to map them. Gaze mode: a left press while the gaze has the pointer isn't sent at once. The pointer stops, you drag it onto what you meant with the button held, and the release clicks there; a press held still for POINTER_GAZE_HOLD (0.5 s) becomes a real press, for drags. The dot shows only while the mouse moves it (POINTER_GAZE_SHOW), while a press is held, and as a pulse per click. Outside games the pointer stays on while gaze mode is on. ft-gazed takes a third of each lesson's offset instead of all of it: in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. Frametop Input Settings gets a Gaze page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
10f21273a6 |
Add gaze mode to the pointer: it goes where you look
MAGIC pointing (Zhai et al. 1999), off by default: with POINTER_GAZE=1, "gaze on", or a button mapped to the new gaze_toggle action, the cursor ray is the corrected gaze from ft-gazed while the gaze has the pointer. Moving the mouse takes the pointer from where the gaze left it; looking more than POINTER_GAZE_RETAKE (5 deg) away with the mouse still gives it back. The pointer is aimed at the gaze, never steered toward it, and only fresh gaze moves it, so a stopped service or a blink leaves it where it is. A mouse nudge of up to POINTER_GAZE_NUDGE_MAX (8 deg) before a click is sent to ft-gazed as a lesson in the eye tracker's error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
137705cbfd |
Add head follow: the pointer comes along when you turn your head
Off by default. With POINTER_FOLLOW=1 (a switch on the Pointer page of Frametop Input Settings, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction leashed POINTER_LEASH_DEG (10) from where you face. Within the leash it stays put in the room; past it, it turns with your head and keeps its offset. A leash of 0 locks it to your view. Head roll is ignored, the cursor stays within 40 degrees of the reference, and it holds still in the room while the left button is down so the head can't nudge a click or a drag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
8f051db083 |
Send typing to the panel clicked last
An ungrabbed keyboard reached both sides at once: gamescope, which in VR reads every input device itself (the SteamOS build's InputStealer), typed into its focused app, and ft-screens typed into the desktop. Space in the desktop paused Spotify on the dashboard, including every space frame-voice dictated. And while the SteamVR dashboard was open, the desktop got no keys at all. Typing now follows the last click. ft-screens sees clicks on its own screens; the pointer helper reports the panel under the dot on each mouse press, so a click on any other panel sends typing to Steam. While typing goes to the desktop and the screens are showing, ft-screens tells the relay every second, and the relay grabs pass-through keyboards. A grab waits until no key is down, and the relay lets go if ft-screens stops reporting. Controller clicks on other panels aren't visible to overlay apps, so they don't move typing (noted in the README). A program that reads every keyboard for a hotkey loses a grabbed one. Repeating the keys on another input device doesn't work, since gamescope reads that too, so with SHARE_KEYS=1 the relay sends them to @frametop_keys as datagrams with the device name. It's off by default: any local process that binds that name first would get every key typed into the desktop. The docs now say gamescope reads keyboards and the headset's buttons itself; they said SteamVR passed keys on to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
705b434cf3 |
Take over the volume keys so they can't crash gamescope
gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and back. When nothing had focus, it moves focus back to null and wlroots aborts, which ends the whole VR session. One press of the headset's volume button did that while working in the Frametop desktop. The input relay now handles volume keys from every device that has them, the headset's buttons included, and steps the default output with wpctl (5%, repeating while held). SteamVR, which passes keys on to gamescope, never sees a volume key: devices with a keymap (gpio-keys, USB and Bluetooth keyboards) get only their volume entries remapped to unused codes, so the headset's click button keeps working, and pmic_resin, which has only volume down, is grabbed. The keymaps go back when the relay stops, and --no-grab leaves the volume keys alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
de2823c360 |
Turn off the Meta tap dashboard shortcut by default
A Meta tap on a pass-through keyboard toggled the SteamVR dashboard, which got in the way of using Meta on its own. It's now off unless META_DASHBOARD=1 is in ~/.config/frametop.conf. Meta as a modifier (Meta+Shift+R, Meta+Shift+H) is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
fb2d62e1da |
Notice a mouse or keyboard that reconnects between device scans
The relay scanned /dev/input once a second and only probed paths it hadn't seen. A Bluetooth mouse that disconnects and reconnects within that second often gets the same event numbers back, so the relay never grabbed it again and the mouse stopped working. Nodes are now tracked by inode, so a re-created node is probed even in a known place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
983773edbd |
Fix installed service and launcher paths; wait for the pointer helper
The service and menu-entry templates still pointed at the old frametop/ subfolder, so the pointer helper, input relay, and settings apps couldn't start after a clean install. The pointer helper's install step also failed when the service was still starting after 3 s; it now waits up to 20 s and doesn't abort the installer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DeeJanuzandClaude Opus 5.5
|
d439bc3f25 |
Frametop: a multi-screen desktop and universal 3D mouse for the Steam Frame
Several KDE Plasma screens floating in SteamVR, each a real monitor of any resolution and shape, shown by our own compositor (ft-screens), with a layout, wrist pinning, and visibility modes; a Bluetooth mouse that drives all of SteamVR as a room-anchored 3D pointer (input relay, ft-pointer helper, ft_pointer SteamVR driver); two settings apps; and Bluetooth LE fixes. Installs on the headset with ./install.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |