baketnk--frame-yap/docs/design.md

FrameYap native dictation overlay — proposal

Standalone project design; see provenance and historical Frame probes. This document is the full target design, not a blanket implementation claim. The native POC now implements overlay/actions, bounded SDL3 capture, a persistent Redux adapter, review-first Gamescope insertion and a user-local archive installer. Quick typing/focus-generation tracking, polished status-chip UX and full hardware acceptance remain proposed. See current scope, device observations and runtime licensing boundary.

Recommendation

Build a small native ARM64 OpenVR overlay application, independent of other applications, with local Parakeet Redux CPU inference. Use Gamescope's input-method protocol for literal Unicode text; keep Linux key-injection APIs as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene renderer, avatar, desktop capture or root service is needed in the primary path.

A first-class product goal is a one-command GitHub install without a Steam store AppID. Package a prebuilt native executable and isolated CPU runtime; use a normal OpenVR application key for registration, not Steamworks. Installation must remain user-local with opt-in autolaunch. See installation design.

controller PTT / overlay mic button
          ↓
SDL3 → PipeWire/PulseAudio mic → bounded 16 kHz mono clip
          ↓
one persistent local Parakeet Redux CPU worker
          ↓
correlated literal transcript → focus/delivery policy
          ↓
Gamescope IME set_string + commit → focused Frame application
          ↘ overlay preview / error / explicit Insert when delivery is unsafe

The CPU worker is local process isolation, not remote inference or a service framework. Implement a small transport owned by this repository. Keep the model loaded between utterances; do not spawn Python/load 178 MB for every release.

Minimal interaction

The implemented menu consolidates review, status, controls and settings onto one rendering surface, with world-space placement by default. Settings offer left wrist, right wrist and head mounting. A smaller status-chip presentation while armed remains a possible refinement, not a second implemented overlay. See the current UI and placement behavior.

[ mic ]  Ready · On-device                     [ settings ]
          Target: selected application

Recording…  00:04           [ Cancel ]

"The recognized text appears here."
[ Insert + space ]  [ Discard ]     [ Insert + Enter — explicit ]

• States: disabled, warming, ready, recording, transcribing, review, inserted, unavailable/error. Recording uses visible icon + text, not color alone.
• Default Frame bindings: hold right X to speak, release to finish; B cancels, A inserts with a trailing space, Y inserts pending text then sends Enter. The alternate right-grip tap-then-hold gesture and left-grip double-tap Enter remain available. All are remappable through the Bindings tab's SteamVR editor. A click-to-start/stop overlay button provides a binding-independent alternative. Bound recording to 20 seconds; discard accidental taps (initial threshold: 200 ms).
• Quick typing: insert on completion only when the explicitly armed target is still valid. Review mode: always wait for Insert. Bring up review instead of silently losing a transcript or typing into a new target.
• Enter requires its own explicit control activation: insert pending review with a trailing space, then queue Enter only if the text step succeeds. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech as commands. Transcription completion never inserts or auto-submits.
• No generic "undo last dictation" initially: another application's edits/cursor cannot be reliably rolled back by a guessed number of backspaces.
• Stop/disable releases owned keys and microphone, invalidates pending delivery, and terminates only the owned worker. No Steam/session/SSH cleanup commands.

Host and rendering boundary

Implement the standalone frameyap executable, initialized with VRApplication_Overlay. Frame accepted that application type and IVROverlay_028 in the probe. Use one CreateOverlay handle, an absolute world transform or tracked-device-relative mount, ShowOverlay/HideOverlay, and PollNextOverlayEvent for the panel. Use SteamVR's overlay interaction rather than inventing scene controller rays.

Do not link an external scene host or launch another application behind the panel. A small overlay-specific RAII owner in this repository should manage OpenVR, overlay handles, input manifest and shutdown. No dependency on external application libraries, assets, build trees or Python environments.

Use one small RGBA panel updated only on UI changes and a bounded recording indicator cadence. The native implementation uploads the CPU-rasterized panel to a persistent Vulkan image and uses SetOverlayTexture; the initial SetOverlayRaw proof path has been replaced. No stereo eye targets or per-eye scene rendering. Keep tracking in compositor transforms, not an application-rendered hand-pose animation loop. Do not promise a particular GPU cost until measured.

A scene renderer's canvas/MSDF resources are not an OpenVR overlay backend and are not imported here. The Inconsolata TTF used by kouseki is independently bundled under its retained OFL; the panel renderer is original FrameYap code. The neon HUD frame is a visual reference, not an engine dependency. Further source/asset reuse requires an explicit license-reviewed extraction, never a runtime path into the engine checkout.

Controller bindings

Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the user bind them. Do not assume a scene app's left-bumper mapping works globally or silently takes a game's button away. Check action activity and neutral rearm.

OpenVR 2.15.6 documents experimental overlay action-set priorities 0x01000000..0x01FFFFFF, gated by SteamVR's Experimental overlay input overrides setting. FrameYap now requests the minimum experimental priority when its config has "input_priority": "experimental"; the default is "normal" (priority zero). This can selectively override scene input, but delivered input and dashboard coexistence still need testing on Frame. FrameYap only reads the SteamVR permission setting and never toggles it automatically. Validate bindings with a scene active, dashboard open/closed, lost tracking, and reconnection. Overlay interactivity/input ownership is distinct from OS keyboard focus.

Text delivery: use the proven path first

1. Gamescope IME — primary Frame backend

Connect to the discovered Gamescope socket; bind gamescope_input_method_manager and wl_seat, handle unavailable/done events, then issue set_string(valid_utf8) and commit(last_serial). A v2 binding is sufficient for text and actions even though the current server advertises v3. Use generated bindings from the pinned XML, not a production handwritten wire protocol. Own/destroy the IME object deliberately and release it while disabled.

This delivered an exact mixed-script string into the disposable Xwayland receiver. It does not need clipboard ownership, sudo, uinput, application plugins, or a new input daemon. However it is a private Gamescope extension; isolate it behind a version-gated backend and test after SteamOS updates.

The server implements text through synthetic key events and a temporary keymap, not a guaranteed rich-text/IME edit operation in every app. Verify actual target toolkits and games. Respect singleton/unavailable handling and Steam-keyboard coexistence. Never use the installed gamescope-type CLI as a transcript pipe: its inspected sample loop is byte-oriented and interprets newline as Submit.

2. Explicit fallbacks, not a framework built up front

• X11 clipboard + XTEST paste: useful for apps that reject direct Unicode key events but accept paste. Own the appropriate X selection (CLIPBOARD or PRIMARY), serving the UTF8_STRING target on the destination X server; select the app's paste chord explicitly. Do not assume Ctrl+V in terminals. Do not overwrite/restore arbitrary clipboard history silently; expose Copy as a deliberate fallback. This backend has not yet been exercised on Frame.
• libei: current server accepts a sender and advertises KEYBOARD but not TEXT. Suitable for evdev-style key chords, not automatic Unicode insertion. Bind/resume/device lifecycle must still be tested. Direct EIS socket access is Gamescope-specific here; no portal RemoteDesktop route was exposed.
• uinput: current account can open it without sudo. Reserve for raw virtual keyboard needs; creating/retiring a device is unnecessary for primary dictation. Unicode is not an evdev keycode, so this alone does not solve text delivery.

Do not assume generic labwc virtual-keyboard/data-control protocols are present: those globals are absent. Fail visibly if the primary backend is unavailable; no automatic privilege escalation or OS package/configuration edits.

Target/focus policy

The IME serial is not an established target-generation guard. Input goes to the seat's current focus. No general Linux input API makes insertion into an arbitrary app atomic with a focus check.

For the first supported route, use an explicitly armed Xwayland destination. Track display identity, active top-level, actual X keyboard-focus window and a monotonic focus generation; invalidate on focus loss/regain, window destruction, disconnect or target change. Observe changes throughout capture/inference and recheck immediately before delivery. Do not restore another app's focus behind the user's back. Refuse quick insertion when modifier keys are held or target identity is ambiguous. A matching final window ID alone is insufficient.

Gamescope exposes focus-display/window root properties, but their encoding and relationship to seat focus need implementation-specific validation. Do not infer that X display :0 is always the destination, or that an X focus observation identifies a native Wayland text field. For unobservable native Wayland focus, require explicit review/Insert; do not advertise safe auto-targeting.

If focus changes, keep the result in review. A fresh Insert explicitly approves the current destination and creates a new delivery authorization. Recheck again at insertion. This minimizes stale delivery but does not eliminate a race between the final check and global input processing; do not claim otherwise. A Wayland roundtrip means compositor processing, not application consumption. Report input queued, never message sent.

Validate UTF-8 and enforce the 4096-byte bound; flatten line breaks/tabs, reject NUL, escape and other control characters. No shell evaluation, commands, Enter, terminal escapes or action inference from recognized text. One correlated request may deliver at most once; cancellation invalidates it before any later reply.

On-device Redux

Use the same moondream/parakeet-redux revision from the benchmark: fad622f25f303105c20d70e201bcc477c88b620c (177,774,490-byte weight file), initially moondream 2.4.0 / kestrel 0.8.0. Do not substitute dense Ultra or silently fall back to desktop/cloud inference.

The earlier benchmark used this local API (its source belongs to the originating repository, not this project):

model = md.photon("moondream/parakeet-redux", model_path=local_model_directory,
                  device="cpu", cpu_threads=thread_budget)
text = model.transcribe(audio=mono_float32, sample_rate=16000)["text"]

Linux ARM64 Python 3.12 native wheels and a packaged CPU payload exist. That is sufficient reason to try the native CPU path first, not proof of Frame performance. Its 61/137 ms desktop median/p95 must not be reused as an estimate for the headset. Its historical dictation quality was worse than Small.en on that tiny dataset; keep transcript visibility and a cheap retry.

Implement a small independent audio/worker adapter with explicit capture, single-request bounds, owner-only runtime files, correlated replies and cancellation. The worker should be implemented independently; do not link, vendor or import another application's speech code. Avoid a generic provider framework: one explicit Redux worker is enough for the first version.

Suggested ownership, introduced only as implementation needs it:

• src/overlay.*: OpenVR lifetime, panel presentation and controller actions.
• src/audio.*: explicit SDL capture and bounded mono PCM.
• src/worker.*: one local child process, bounded requests/replies, timeout/reaping.
• src/text_input.*: Gamescope protocol, focus observation and delivery policy.
• python/frameyap/: persistent CPU Redux worker, no external application imports.

The worker reads a fixed private clip and returns an ID-correlated literal string; one request at a time, 64 KiB framed messages, 4096-byte transcript, and a bounded processing timeout. No shell commands in IPC and no input authority in the worker.

• One persistent worker and one loaded model; no queue of utterances. Initially batch at PTT release, not speculative streaming or endpoint/VAD delay.
• Start with two CPU threads, compare against four on Frame; bound Torch and native kernel pools. Choose measured latency versus compositor contention, not the desktop's thread count by habit. No real-time scheduling or permanent CPU pinning initially; inspect runtime affinity behaviour during measurement.
• Explicit local model path and offline loading. Missing runtime/weights produces an actionable error, not an unsolicited download/network fallback.
• Optional explicit enable/warm-up before first PTT; warming must not record audio. Expose that lifecycle explicitly rather than warming on import or construction. Otherwise display first-use loading honestly. Keep model reuse after normal completion; cancellation may restart the owned worker.
• Use a private owner-only directory under $XDG_RUNTIME_DIR for bounded tmpfs-backed clips; remove them on completion, error, cancellation and shutdown. Avoid persistent audio/transcripts by default. Local IPC is not a network hop; do not add shared-memory complexity before measuring it.
• Use CPU-only Torch where supported; test native kernel import/model load with no CUDA device/runtime assumption. Do not copy the desktop's x86 venv.
• Weights are ~178 MB; Torch, kernels, temporary conversion and activations mean install size/RSS will be larger. Measure cold load, peak RSS and package size. Review runtime redistribution licensing separately from model attribution.

Microphone access does not mute VRChat or any other social-voice app. Shared PipeWire capture may let both hear the same utterance; the overlay must not claim private dictation unless the other app's transmission is separately muted. No automatic global microphone mute/reroute in this scope.

Delivery milestones and acceptance gates

These are proposed implementation gates, not completed acceptance:

1. Offline ARM64 CPU spike: isolated user-directory environment, pinned local model, permitted runtime distribution. Run known nonprivate clips, then the consented benchmark clips only if explicitly made available within their data scope. Record cold load, first/warm p50/p95, errors, peak RSS, two/four threads and dependencies. Confirm no required NVIDIA/desktop connection. Stop and report if the packed CPU runtime is incompatible; do not silently expand to dense weights.
2. Minimal overlay: render status/review panel while another scene stays active; confirm dashboard/hand placement, input events, close/reopen and no scene-focus takeover. Validate global PTT separately rather than blocking the clickable prototype on experimental override support.
3. Real dictation path: microphone → local Redux → preview → explicit insert into a disposable target; then enable quick typing after target tracking tests. Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate replies, lost mic, worker crash and missing model without persisting speech.
4. Target matrix: Xwayland terminal/browser and selected native/Proton game text fields; Steam keyboard coexistence; native Wayland targets separately. Test focus changes during capture/inference, rapid loss/regain, held modifiers, explicit Insert retargeting, no hidden Enter, and no second delivery.
5. In-headset acceptance: readable feedback and comfortable PTT; measured release-to-insert latency and compositor timing while an actual scene runs; no noticeable sustained thermal/battery regression. Set numeric budgets after the ARM64 spike, not from an x86 result. Physical success remains human-led.

Native Frame app typing is the first scope. Local keystrokes may or may not be forwarded by the PC-streaming client; that route needs its own test. If necessary, a future optional text-only host bridge could send the completed transcript, without moving recognition off Frame. It is not part of this initial design.

Standalone dependency and test policy

The default hardware-free build needs only CMake and a C++20 compiler (Python runs additional offline tests). FRAMEYAP_NATIVE=ON explicitly selects OpenVR, SDL3, FreeType and Wayland client/generated protocol bindings. A separately authorized Python Redux environment is explicitly supplied at launch. Pin revisions and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery.

Hardware-free tests should cover state transitions, bounded PCM/transcripts, worker framing/timeout/cancellation, duplicate/stale replies and focus generations with fakes. Separate opt-in Frame checks cover protocol availability, owned-window insertion, microphones, inference and overlays. No fixture establishes physical headset acceptance. See provenance for historical source anchors.