mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-06 04:04:07 +02:00
Add opt-in fail-closed Xwayland auto insert
This commit is contained in:
1 parent
19a3db7667
commit
c5e925cb48
27 files changed
+588
-45
No files matched your search
+7
-5
@@ -5,7 +5,8 @@ Standalone project design; see
|
||||
This document is the full target design, not a blanket implementation claim.
|
||||
The native POC now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
|
||||
Quick typing/focus-generation tracking, polished status-chip UX and full hardware
|
||||
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
|
||||
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
|
||||
acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md)
|
||||
and [runtime licensing boundary](third-party.md).
|
||||
|
||||
@@ -66,13 +67,14 @@ Recording… 00:04 [ Cancel ]
|
||||
A click-to-start/stop overlay button provides
|
||||
a binding-independent alternative. Bound recording to 20 seconds; discard
|
||||
accidental taps (initial threshold: 200 ms).
|
||||
- **Quick typing:** insert on completion only when the explicitly armed target
|
||||
is still valid. **Review mode:** always wait for Insert. Bring up review instead
|
||||
of silently losing a transcript or typing into a new target.
|
||||
- **Quick typing (opt-in, locally implemented):** insert on completion only when
|
||||
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
|
||||
wait for Insert. Bring up review on uncertainty rather than silently losing a
|
||||
transcript or typing into a new target. Live Frame acceptance remains open.
|
||||
- Enter requires its own explicit control activation: insert pending review with
|
||||
a trailing space, then queue Enter only if the text step succeeds. With no
|
||||
review, it queues only Enter. Never interpret "submit", "delete" or other speech
|
||||
as commands. Transcription completion never inserts or auto-submits.
|
||||
as commands. Transcription completion never auto-submits.
|
||||
- No generic "undo last dictation" initially: another application's edits/cursor
|
||||
cannot be reliably rolled back by a guessed number of backspaces.
|
||||
- Stop/disable releases owned keys and microphone, invalidates pending delivery,
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# Guided Frame focus probe — 2026-09-24
|
||||
|
||||
Historical observation, not authorization for later live input or proof of target safety.
|
||||
The wearer consented to an opt-in, disposable-target focus trial. Two project-owned
|
||||
`xmessage` windows were created on Xwayland `:0` for 65 seconds, then closed by
|
||||
the owning finite command. No microphone, transcript, input injection or SteamVR
|
||||
session cleanup was used. Other SSH/user processes were left untouched.
|
||||
|
||||
An earlier read-only snapshot showed `/tmp/.X11-unix/X0` and `X1`, Gamescope
|
||||
socket `gamescope-0`, and `_NET_ACTIVE_WINDOW` matching
|
||||
`GAMESCOPE_FOCUSED_WINDOW` on `:0`. Brief snapshots of an owned test window
|
||||
also showed disagreement between these root properties, so either property
|
||||
alone is insufficient to authorize automatic typing.
|
||||
|
||||
The wearer selected A/B and reported doing several focus changes. Window A was
|
||||
`0x3e00022`, B was `0x4200022` in that run. A 120 ms sampled observer saw
|
||||
X keyboard focus, `_NET_ACTIVE_WINDOW` and `GAMESCOPE_FOCUSED_WINDOW` agree at
|
||||
A, change to B, then return to A several times. A transition to a third window
|
||||
`0x3c00003` was also observed. At other moments the X keyboard-focus/active
|
||||
window IDs changed while Gamescope's focused-window property still named A.
|
||||
The root property is a window ID encoded as CARDINAL, not a generation token.
|
||||
|
||||
These samples establish *observable correlation for these two owned Xwayland
|
||||
windows*, not continuous seat identity across all targets, a focus-loss event
|
||||
stream, transcript quality, delivered input, native Wayland coverage or headset
|
||||
acceptance. The offline fail-closed observer subscribes to X property changes
|
||||
and focus-out on the exact armed window; its own live behavior and actual IME
|
||||
delivery still require a separate disposable-target validation. There is still
|
||||
a non-atomic gap between final focus check and compositor input processing.
|
||||
+21
-1
@@ -63,7 +63,9 @@ tab changes, action-state changes and relocation clear pending presses. Enter is
|
||||
deliberate action, not inferred from text. Insert appends a trailing space (without
|
||||
doubling an existing trailing space). Enter inserts any pending review and then
|
||||
queues Enter; with no pending text it queues Enter only. A failed text step never
|
||||
proceeds to Enter. Recording never automatically inserts or submits.
|
||||
proceeds to Enter. Recording never automatically submits. Auto insert, when
|
||||
explicitly enabled, can queue text + space after transcription only under the
|
||||
stable Xwayland focus guard described below.
|
||||
|
||||
### Bindings button
|
||||
|
||||
@@ -95,6 +97,7 @@ installer creates one with defaults on first install. Copy the shipped
|
||||
"font": "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
|
||||
"input_priority": "normal",
|
||||
"advanced_debug": false,
|
||||
"auto_insert": false,
|
||||
"wrist": {"x": 0, "y": 0.18, "z": 0.089, "width": 0.30, "roll_degrees": 0},
|
||||
"theme": {
|
||||
"background": "#0c101b", "card": "#141c2b", "ink": "#e6f0f9",
|
||||
@@ -125,6 +128,23 @@ Detailed logs are bounded and owner-private; see [worker diagnostics](worker.md#
|
||||
config, retaining other fields and formatting; invalid/unwritable configs are
|
||||
left untouched and return failure. The installer backs up original bytes before
|
||||
repairing invalid values, while valid `true` and `false` are retained.
|
||||
|
||||
`auto_insert` is a separate boolean, default `false`, also available as a
|
||||
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
|
||||
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
||||
`GAMESCOPE_FOCUSED_WINDOW` must agree with the exact X keyboard-focus window.
|
||||
Both properties and focus are rechecked after IME lease acquisition. A watched
|
||||
focus-out, root focus-property change (even if the same window returns), window
|
||||
destruction, held keyboard key, missing X display or any disagreement permanently
|
||||
disarms that clip. The transcript then remains for explicit review/Insert.
|
||||
Native Wayland focus and child text-field focus cannot be safely inferred here;
|
||||
those cases fall back to review. No automatic Enter, speech commands or retry.
|
||||
The compositor can still change focus in the gap between the final check and
|
||||
global delivery, and IME commit is not an application receipt. This path has
|
||||
offline synthetic focus tests; live automatic typing, target coverage and
|
||||
headset acceptance are still unverified. The setting is preserved on upgrade
|
||||
and a failed preference write applies only to the current session.
|
||||
|
||||
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
|
||||
`insert`, `enter`) to Frame physical `/user/hand/{left|right}/input/NAME`
|
||||
button paths. Omitted actions retain their bundled defaults; an empty string
|
||||
|
||||
+2
-2
@@ -25,8 +25,8 @@ For the current **external-runtime** POC, `scripts/stage-native-poc.py --help`
|
||||
documents explicit inputs. It invokes `cmake --install` on an existing native build,
|
||||
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
||||
not build, download, run the app, or copy a proprietary ASR runtime. The native
|
||||
POC relies on Frame's system Vulkan loader/driver, Wayland, FreeType, libstdc++
|
||||
and glibc; audit `ldd` on the installed binary.
|
||||
POC relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
||||
libstdc++ and glibc; audit `ldd` on the installed binary.
|
||||
SDL/OpenVR resolve inside its own `lib/`, not a producer
|
||||
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
|
||||
|
||||
|
||||
+8
-4
@@ -46,10 +46,14 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
||||
|
||||
## Deliberately not claimed
|
||||
|
||||
**Review-first only.** No automatic insertion or inferred commands. A transcript
|
||||
must be explicitly inserted into the *current* focused destination. We do not yet
|
||||
implement the proposed Xwayland focus-generation observer or safe quick typing.
|
||||
There remains a race with focus changes after user approval. Text delivery is
|
||||
**Review-first by default.** An opt-in Auto insert setting now watches the
|
||||
Xwayland active window, Gamescope focused-window property and exact keyboard
|
||||
focus from PTT start to IME lease acquisition. Loss/regain, disagreement,
|
||||
unavailable focus or a held keyboard key falls back to manual review. No
|
||||
automatic Enter or inferred speech commands. This focus guard has offline
|
||||
tests but live microphone → automatic insertion is **not yet accepted** on Frame;
|
||||
it cannot identify arbitrary native Wayland targets. There remains a race
|
||||
between the final focus check and global input processing. Text delivery is
|
||||
reported as **input queued**, not application consumption or message delivery.
|
||||
A request is consumed once even if transport completion is uncertain; no retries.
|
||||
Unavailable IME acquisition leaves the preview intact.
|
||||
|
||||
@@ -46,6 +46,9 @@ font, places the launcher copy at `fonts/font.ttf`, and includes its license in
|
||||
retain its LICENSE with redistributed loader binaries.
|
||||
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
|
||||
- Wayland client and scanner: retain upstream MIT-style notices.
|
||||
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
|
||||
Xwayland focus observer. Include it in native runtime dependency checks;
|
||||
no X server is started by normal operation.
|
||||
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
|
||||
- Optional font override: the earlier device check used system Hack Regular.
|
||||
A custom release font must include its own license and assessed glyph coverage;
|
||||
|
||||
Reference in new issue
Block a user