baketnk--frame-yap/TODO.md

FrameYap TODO

Compiled from the 2026-09-24 holistic review, with owner decisions applied (see "Decisions" at the end). Items are sized to hand off individually. Size: S ≈ hours, M ≈ a day, L ≈ multi-day.

Guardrails from AGENTS.md apply to every item: offline default build, no implicit downloads, no automatic Enter, small verified commits, docs must state implemented vs proposed behavior honestly. Checked source-work items below mean the local implementation is present, not a shipped or headset-accepted release.

Implemented locally; offline tested; installed/headset validation pending

The current working tree includes the source changes for A3, A5–A7, B1–B4, C1–C3, D1–D2, E2–E3 and F1–F2. Final offline checks passed 24/24 default, 25/25 strict UI, and the fake Wayland input regression suite. An earlier private native ARM64 snapshot passed 28/28; the final source requires a fresh native build before deployment. These checkboxes close the source tasks, not their empirical acceptance gates. C1's second backend is a fake executable fixture, not a second shipped ASR engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's source/attended installer paths still need an audited native archive and an installed clean-account/Frame exercise. D3 (deferred binary archive), D4 (pip runtime), P1 (real-target delivery), G1 (browser), and H (live headset acceptance) remain open. The 0.1.202609250333 build is installed on Frame but not launched or accepted; local code/tests cannot establish a fixed delivery regression.

Priority regression (reported during implementation)

• P1. Repeated delivery loses the beginning of later submissions. Closed 2026-09-25 by owner confirmation: the owner has been dictating into real apps with live voice typing on the paced build and sees no front-prefix loss. History below is kept for context. After the first Type + Enter, later text reportedly loses a dozen to a few dozen leading bytes. Investigate preview versus destination loss, retain the full bounded literal transcript, and add repeated/long/Unicode delivery regression tests. Do not assume a larger buffer fixes it or retry uncertain delivery automatically. An old-code delivery fixture crashed the Gamescope session, not the OS; this is not evidence of a fix. Plain-ASCII prefix corruption was reproduced; retained IME and immediate byte chunking did not fix it. A nonblocking, focus-guarded 24-codepoint/150 ms delivery queue is implemented with offline regressions, including full-length Unicode and all-space batches. Clipboard remains untouched. Real-target confirmation is still required after deployment. Device update (2026-09-24): version 0.1.202609250333 was installed on Frame (version and binary hash checked; not launched; runtime paths preserved; config migration backed up); the deployment peer reported 29/29 native tests. Synthetic repeated text and a full 4,096-byte payload arrived exactly. A later mismatch looks consistent with receiver keymap caching, not Unicode trimming or space substitution, but that is unconfirmed. P1 stays open until real-app delivery is verified; this is not headset acceptance.

A. First release (v0.1) blockers

v0.1.202609251524 released (2026-09-25). Built on the Frame with scripts/build-release.sh, published on GitHub, and installed fresh on the Frame by the owner via curl | sh (install → model → pip runtime), followed by successful live voice typing. The README is now user-facing; technical detail lives in docs/development.md. Still unchecked: the exact missing-runtime panel message and the battery readouts on Frame.

• A8. UI polish batch (shipped in v0.1.202609251524). Removed the fixed "Type adds a space" hint, grew the review card to six lines to close the empty band, added L/HMD/R battery readouts and a Buttons ready/paused dashboard chip, and made Type with nothing to review send Enter. Which batteries Frame reports needs a headset check (--check-controls prints them).

• A1. Remove the runtime-license blocker notes. Upstream's Kestrel README states local inference is free; blocker text removed from README/docs/scripts and replaced with a neutral dependency note in docs/third-party.md.

• A2. Remove references to the unrelated app. (S) Tracked references were removed while preserving Inconsolata/OFL attribution, font SHA-256 and upstream googlefonts/Inconsolata source. A tracked, case-insensitive grep for the former app name must remain empty.

• A3. Commit the pending AGENTS.md rename ("Frame Dictation" → "FrameYap"). (S) Done in baseline checkpoint 07c03ea.

• [~] A4. Runtime dependency pointers. Repos and licenses are listed in docs/third-party.md. No release audit is needed while v0.1 is source-only.

• A5. Drop "POC" from the shipped surface. (S) Public help, README, CMake and installer use the release name. scripts/stage-native-poc.py remains a deprecated compatibility wrapper for scripts/stage-native.py, not the documented or shipped staging entry point. v0.1 is a first small release.

• A6. Rewrite the README front. (M) Pitch, requirements, local-only install, controls table and explicit "Status / not yet validated" section are present.

• A7. Archive docs. Dated evidence (docs/evidence/) and provenance.md moved to the untracked, gitignored docs/archive/; poc.md renamed docs/build.md. Current design/overlay/packaging docs distinguish implemented local behavior from proposed and unaccepted headset/release behavior.

B. Correctness / robustness

• B1. Keep malformed transcripts request-local. (S) Controller::tick() catches a correlated bad UTF-8/control reply and fails the request without stopping the ready worker; hardware-free fakes test retry.
• B2. Use engine-neutral C++ worker errors. (S) F/I messages no longer name Redux's Python engine.
• B3. "Close mic when idle" setting, default OFF. (M) Config/Settings and fake-backed mic-lifetime tests cover default idle draining versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
• B4. Extract/test interaction logic from run(). (L) Controller receives injectable audio/worker/focus/delivery interfaces; offline tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.

C. Backends and model management

• C1. Manifest-driven worker backends (source capability). (L) Redux's pinned manifest and a generic local dispatcher implement the existing Y/T/R/E framing. An offline fake second executable backend works without edits to C++ worker/runtime; only Redux has a shipped inference engine. New backends still require license/runtime and actual inference validation.
• C2. Model/backend state and chooser (local source). (L) Settings lists manifest-backed model status and active/loading/ready/failure states; selection persists and invalidates/restarts worker, clip and review. Install → Confirm Install displays pinned source, size, license, attribution and manifest SHA-256; confirmation launches an owned installer helper with that digest and offline rechecks afterward. Full consent metadata is paginated; the panel shows bounded per-file download/verification events and sanitized errors. Installer output alone never proves success. No implicit downloads or runtime install. Installed chooser/consent behavior on Frame remains unvalidated.
• C3. Offline model status CLI. (S) frameyap --list-models and --check-model ID dispatch pinned local manifest/hash checks, without ASR or downloads. Installed archive CLI still needs clean-account verification.

D. Installer

• D1. Local binary-or-source installer paths. (L) install.sh has a checksummed binary route and explicit local source/toolchain preflight/build/stage/install route, with rollback and no default registration. A native archive was exercised in private HOME/XDG roots on Frame: install, installed model-status CLI, idempotency, wrapper repair and uninstall. This same-user/system-library fixture is not clean-account acceptance.
• D2. Attended-or-unattended model-agnostic installer interface. (M) TTY choices have equivalent flags; --mode binary|source, --backend ID, --model-dir, --yes, --autolaunch/--no-autolaunch, --without-model, --print-plan and --json support offline plans and structured outcomes; explicit model installs use installed pinned manifests. Tested with local fixtures only, not a released archive or an installed Frame UI handoff.
• D3. Publish a prebuilt ARM64 archive. (M) scripts/build-release.sh builds SDL3/OpenVR-pinned archives on Linux ARM64; v0.1.202609251524 is built on the Frame and published with its checksum and install.sh. The archive has no Kestrel/Torch/moondream/model; those are user-side downloads.
• D4. Installer fetches the Python runtime with pip. (M) Implemented 2026-09-25 as explicit install.sh --install-runtime --yes: user-local venv, CPU torch==2.8.0 from PyTorch's CPU index, moondream==2.4.0 from PyPI, import/no-CUDA verification, then python= in paths.conf. Offline tests mock pip; a real run passed on x86-64 Python 3.12. Needs the clean Frame install to confirm on ARM64.

E. Naming and versioning

• E1. Name: keep "FrameYap" for v0.1. Frame (the hardware) + yap (speech) says what it is; no rename churn before the first tag. If a hardware-neutral project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame front end), decide it before the cross-window work in G, not now.
• E2. User-facing Type / Type + Enter / Quick phrases labels. (S) Overlay, diagnostics, README/help and overlay docs align on these controls; insert, enter, quick_chat remain internal binding/API names. Settings explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model from its moondream Python inference package.
• E3. Numeric MAJOR.MINOR.YYYYMMDDHHMM version source work. (S) CMake, CLI/tests, package producer and installer use numeric release versions; dev git info is a separate --version line. SOURCE_DATE_EPOCH can supply the configuration timestamp. A clean release tag/archive still needs D3.

F. Code structure (non-urgent)

• F1. Move --check-* diagnostics out of main.cpp. (S) src/check.cpp owns native checks; src/cli.cpp provides the table-driven parser. Device behavior remains separately gated.
• F2. Split overlay.cpp's Impl. (M) Drag state, persistence/save failures and diagnostics now have separate grouped owners; native snapshot build and offline panel/drag tests cover the refactor.

G. Other windows (post-release)

Text delivery already works into a WezTerm window on Frame (owner-tested; this is not the recorded live acceptance in H). Deeper integration of other windows with this app is future design and out of scope for v0.1.

• G1. Validate browser text fields via the Gamescope input path. (M) Highest-priority target. Record which fields accept Type / Type + Enter (plain inputs, textareas, rich editors, password fields should be expected to differ) and document the results honestly.
• G2. Later, if needed: a KDE desktop-mode backend behind DeliveryLease, and a uinput backend as a last resort. Not scheduled; uinput needs /dev/uinput access and types with no focus check, which conflicts with the project's no-sudo/udev rule and per-window authorization.

H. Carried over from the earlier TODO (hardware validation)

• Basic menu launch on Frame: the user found FrameYap in Steam's Non-Steam section; the panel showed and Quit worked. This does not validate transcription, recording, text delivery, cold startup, or which shortcut discovery mechanism Steam used.
• Verify the exact missing-runtime panel message and Steam's shortcut persistence across a normal restart (without restarting sessions just for the test). Confirm no runtime/model environment override before future live checks.
• Live acceptance on Frame: microphone → reviewed text → real target delivery. Confirmed by the owner's daily live voice typing (2026-09-25). Auto insert with speech and physical resize were not separately reported.

---

Decisions

• Redux runtime: treated as usable for local inference per upstream's Kestrel README; not bundled in our archives (A1 done).
• Multiple backends + a model chooser UI: wanted (C1–C3). The panel can trigger the install on an explicit click, for non-technical users.
• Name: keep FrameYap. v0.1 is a first small release, not a POC.
• Close-mic-when-idle: setting, default off (B3).
• Labels: Type / Type + Enter (E2).
• Installer: binary or source, continues automatically after choices, fully flag- driven for agent use (D1–D2).
• Version: MAJOR.MINOR.YYYYMMDDHHMM, git hash only in dev-build --version output (E3).
• Other windows: browser text fields first (G1); deeper integration is future design.

Open questions

The fresh Frame install passed (2026-09-25). Open follow-ups: the missing-runtime panel message, which batteries Frame reports, browser text fields (G1), and whether the installer's y/n defaults suit first-time users.