Add opt-in fail-closed Xwayland auto insert

This commit is contained in:
baketnk committed 2026-09-24 16:12:33 -04:00
1 parent 19a3db7667
commit c5e925cb48
27 files changed
+588 -45

No files matched your search

+7 -5
View File
@@ -5,7 +5,8 @@ Standalone project design; see
This document is the full target design, not a blanket implementation claim.
The native POC now implements overlay/actions, bounded SDL3 capture, a persistent
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
Quick typing/focus-generation tracking, polished status-chip UX and full hardware
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md)
and [runtime licensing boundary](third-party.md).
@@ -66,13 +67,14 @@ Recording… 00:04 [ Cancel ]
A click-to-start/stop overlay button provides
a binding-independent alternative. Bound recording to 20 seconds; discard
accidental taps (initial threshold: 200 ms).
- **Quick typing:** insert on completion only when the explicitly armed target
is still valid. **Review mode:** always wait for Insert. Bring up review instead
of silently losing a transcript or typing into a new target.
- **Quick typing (opt-in, locally implemented):** insert on completion only when
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
wait for Insert. Bring up review on uncertainty rather than silently losing a
transcript or typing into a new target. Live Frame acceptance remains open.
- Enter requires its own explicit control activation: insert pending review with
a trailing space, then queue Enter only if the text step succeeds. With no
review, it queues only Enter. Never interpret "submit", "delete" or other speech
as commands. Transcription completion never inserts or auto-submits.
as commands. Transcription completion never auto-submits.
- No generic "undo last dictation" initially: another application's edits/cursor
cannot be reliably rolled back by a guessed number of backspaces.
- Stop/disable releases owned keys and microphone, invalidates pending delivery,
+29
View File
@@ -0,0 +1,29 @@
# Guided Frame focus probe — 2026-09-24
Historical observation, not authorization for later live input or proof of target safety.
The wearer consented to an opt-in, disposable-target focus trial. Two project-owned
`xmessage` windows were created on Xwayland `:0` for 65 seconds, then closed by
the owning finite command. No microphone, transcript, input injection or SteamVR
session cleanup was used. Other SSH/user processes were left untouched.
An earlier read-only snapshot showed `/tmp/.X11-unix/X0` and `X1`, Gamescope
socket `gamescope-0`, and `_NET_ACTIVE_WINDOW` matching
`GAMESCOPE_FOCUSED_WINDOW` on `:0`. Brief snapshots of an owned test window
also showed disagreement between these root properties, so either property
alone is insufficient to authorize automatic typing.
The wearer selected A/B and reported doing several focus changes. Window A was
`0x3e00022`, B was `0x4200022` in that run. A 120 ms sampled observer saw
X keyboard focus, `_NET_ACTIVE_WINDOW` and `GAMESCOPE_FOCUSED_WINDOW` agree at
A, change to B, then return to A several times. A transition to a third window
`0x3c00003` was also observed. At other moments the X keyboard-focus/active
window IDs changed while Gamescope's focused-window property still named A.
The root property is a window ID encoded as CARDINAL, not a generation token.
These samples establish *observable correlation for these two owned Xwayland
windows*, not continuous seat identity across all targets, a focus-loss event
stream, transcript quality, delivered input, native Wayland coverage or headset
acceptance. The offline fail-closed observer subscribes to X property changes
and focus-out on the exact armed window; its own live behavior and actual IME
delivery still require a separate disposable-target validation. There is still
a non-atomic gap between final focus check and compositor input processing.
+21 -1
View File
@@ -63,7 +63,9 @@ tab changes, action-state changes and relocation clear pending presses. Enter is
deliberate action, not inferred from text. Insert appends a trailing space (without
doubling an existing trailing space). Enter inserts any pending review and then
queues Enter; with no pending text it queues Enter only. A failed text step never
proceeds to Enter. Recording never automatically inserts or submits.
proceeds to Enter. Recording never automatically submits. Auto insert, when
explicitly enabled, can queue text + space after transcription only under the
stable Xwayland focus guard described below.
### Bindings button
@@ -95,6 +97,7 @@ installer creates one with defaults on first install. Copy the shipped
"font": "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
"input_priority": "normal",
"advanced_debug": false,
"auto_insert": false,
"wrist": {"x": 0, "y": 0.18, "z": 0.089, "width": 0.30, "roll_degrees": 0},
"theme": {
"background": "#0c101b", "card": "#141c2b", "ink": "#e6f0f9",
@@ -125,6 +128,23 @@ Detailed logs are bounded and owner-private; see [worker diagnostics](worker.md#
config, retaining other fields and formatting; invalid/unwritable configs are
left untouched and return failure. The installer backs up original bytes before
repairing invalid values, while valid `true` and `false` are retained.
`auto_insert` is a separate boolean, default `false`, also available as a
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
`GAMESCOPE_FOCUSED_WINDOW` must agree with the exact X keyboard-focus window.
Both properties and focus are rechecked after IME lease acquisition. A watched
focus-out, root focus-property change (even if the same window returns), window
destruction, held keyboard key, missing X display or any disagreement permanently
disarms that clip. The transcript then remains for explicit review/Insert.
Native Wayland focus and child text-field focus cannot be safely inferred here;
those cases fall back to review. No automatic Enter, speech commands or retry.
The compositor can still change focus in the gap between the final check and
global delivery, and IME commit is not an application receipt. This path has
offline synthetic focus tests; live automatic typing, target coverage and
headset acceptance are still unverified. The setting is preserved on upgrade
and a failed preference write applies only to the current session.
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
`insert`, `enter`) to Frame physical `/user/hand/{left|right}/input/NAME`
button paths. Omitted actions retain their bundled defaults; an empty string
+2 -2
View File
@@ -25,8 +25,8 @@ For the current **external-runtime** POC, `scripts/stage-native-poc.py --help`
documents explicit inputs. It invokes `cmake --install` on an existing native build,
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
not build, download, run the app, or copy a proprietary ASR runtime. The native
POC relies on Frame's system Vulkan loader/driver, Wayland, FreeType, libstdc++
and glibc; audit `ldd` on the installed binary.
POC relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
libstdc++ and glibc; audit `ldd` on the installed binary.
SDL/OpenVR resolve inside its own `lib/`, not a producer
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
+8 -4
View File
@@ -46,10 +46,14 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
## Deliberately not claimed
**Review-first only.** No automatic insertion or inferred commands. A transcript
must be explicitly inserted into the *current* focused destination. We do not yet
implement the proposed Xwayland focus-generation observer or safe quick typing.
There remains a race with focus changes after user approval. Text delivery is
**Review-first by default.** An opt-in Auto insert setting now watches the
Xwayland active window, Gamescope focused-window property and exact keyboard
focus from PTT start to IME lease acquisition. Loss/regain, disagreement,
unavailable focus or a held keyboard key falls back to manual review. No
automatic Enter or inferred speech commands. This focus guard has offline
tests but live microphone → automatic insertion is **not yet accepted** on Frame;
it cannot identify arbitrary native Wayland targets. There remains a race
between the final focus check and global input processing. Text delivery is
reported as **input queued**, not application consumption or message delivery.
A request is consumed once even if transport completion is uncertain; no retries.
Unavailable IME acquisition leaves the preview intact.
+3
View File
@@ -46,6 +46,9 @@ font, places the launcher copy at `fonts/font.ttf`, and includes its license in
retain its LICENSE with redistributed loader binaries.
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
- Wayland client and scanner: retain upstream MIT-style notices.
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
Xwayland focus observer. Include it in native runtime dependency checks;
no X server is started by normal operation.
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
- Optional font override: the earlier device check used system Hack Regular.
A custom release font must include its own license and assessed glyph coverage;