baketnk--frame-yap/TODO.md

# FrameYap TODO

Compiled from the 2026-09-24 holistic review, with owner decisions applied (see
"Decisions" at the end). Items are sized to hand off individually. Size: S ≈ hours,
M ≈ a day, L ≈ multi-day.

Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
downloads, no automatic Enter, small verified commits, docs must state implemented
vs proposed behavior honestly. Checked source-work items below mean the local
implementation is present, **not** a shipped or headset-accepted release.

## Implemented locally; offline tested; installed/headset validation pending

The current working tree includes the source changes for A3, A5–A7, B1–B4,
C1–C3, D1–D2, E2–E3 and F1–F2. Final offline checks passed 24/24 default,
25/25 strict UI, and the fake Wayland input regression suite. An earlier private
native ARM64 snapshot passed 28/28; the final source requires a fresh native
build before deployment.
These checkboxes close the *source tasks*, not their empirical acceptance gates.
C1's second backend is a fake executable fixture, **not** a second shipped ASR
engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's
source/attended installer paths still need an audited native archive and an
installed clean-account/Frame exercise. D3 (deferred binary archive), D4 (pip runtime), P1 (real-target delivery), G1
(browser), and H (live headset acceptance) remain open. The 0.1.202609250333 build is installed on Frame but not launched or
accepted; local code/tests cannot establish a fixed delivery regression.

## Priority regression (reported during implementation)

- [x] **P1. Repeated delivery loses the beginning of later submissions.**
  **Closed 2026-09-25 by owner confirmation:** the owner has been dictating
  into real apps with live voice typing on the paced build and sees no
  front-prefix loss. History below is kept for context. After
  the first Type + Enter, later text reportedly loses a dozen to a few dozen leading
  bytes. Investigate preview versus destination loss, retain the full bounded
  literal transcript, and add repeated/long/Unicode delivery regression tests.
  Do not assume a larger buffer fixes it or retry uncertain delivery automatically.
  An old-code delivery fixture crashed the Gamescope session, **not** the OS;
  this is not evidence of a fix. Plain-ASCII prefix corruption was reproduced;
  retained IME and immediate byte chunking did not fix it. A nonblocking,
  focus-guarded 24-codepoint/150 ms delivery queue is implemented with offline
  regressions, including full-length Unicode and all-space batches. Clipboard
  remains untouched. Real-target confirmation is still required after deployment.
  Device update (2026-09-24): version 0.1.202609250333 was installed on Frame
  (version and binary hash checked; not launched; runtime paths preserved; config
  migration backed up); the deployment peer reported 29/29 native tests. Synthetic
  repeated text and a full 4,096-byte payload arrived exactly. A later mismatch
  looks consistent with receiver keymap caching, not Unicode trimming or space
  substitution, but that is unconfirmed. P1 stays open until real-app delivery is
  verified; this is not headset acceptance.

## A. First release (v0.1) blockers

**v0.1.202609251524 released (2026-09-25).** Built on the Frame with
`scripts/build-release.sh`, published on GitHub, and installed fresh on the
Frame by the owner via `curl | sh` (install → model → pip runtime), followed
by successful live voice typing. The README is now user-facing; technical
detail lives in `docs/development.md`. Still unchecked: the exact
missing-runtime panel message and the battery readouts on Frame.

- [x] **A8. UI polish batch (shipped in v0.1.202609251524).** Removed the fixed
  "Type adds a space" hint, grew the review card to six lines to close the empty
  band, added L/HMD/R battery readouts and a Buttons ready/paused dashboard chip,
  and made Type with nothing to review send Enter. Which batteries Frame reports
  needs a headset check (`--check-controls` prints them).

- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
  states local inference is free; blocker text removed from README/docs/scripts and
  replaced with a neutral dependency note in `docs/third-party.md`.
- [x] **A2. Remove references to the unrelated app.** (S)
  Tracked references were removed while preserving Inconsolata/OFL attribution,
  font SHA-256 and upstream googlefonts/Inconsolata source. A tracked,
  case-insensitive grep for the former app name must remain empty.
- [x] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
  "FrameYap"). (S) Done in baseline checkpoint `07c03ea`.
- [~] **A4. Runtime dependency pointers.** Repos and licenses are listed in
  `docs/third-party.md`. Audit the staged native libraries and notices for each published archive;
  speech runtime packages and model files remain separate explicit downloads.
- [x] **A5. Drop "POC" from the shipped surface.** (S) Public help, README,
  CMake and installer use the release name. `scripts/stage-native-poc.py` remains
  a deprecated compatibility wrapper for `scripts/stage-native.py`, not the
  documented or shipped staging entry point. v0.1 is a first small release.
- [x] **A6. Rewrite the README front.** (M) Pitch, requirements, local-only install,
  controls table and explicit "Status / not yet validated" section are present.
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
  to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
  Current design/overlay/packaging docs distinguish implemented local behavior
  from proposed and unaccepted headset/release behavior.

## B. Correctness / robustness

- [x] **B1. Keep malformed transcripts request-local.** (S)
  `Controller::tick()` catches a correlated bad UTF-8/control reply and fails the
  request without stopping the ready worker; hardware-free fakes test retry.
- [x] **B2. Use engine-neutral C++ worker errors.** (S)
  `F`/`I` messages no longer name Redux's Python engine.
- [x] **B3. "Close mic when idle" setting, default OFF.** (M)
  Config/Settings and fake-backed mic-lifetime tests cover default idle draining
  versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
- [x] **B4. Extract/test interaction logic from `run()`.** (L)
  `Controller` receives injectable audio/worker/focus/delivery interfaces; offline
  tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.

## C. Backends and model management

- [x] **C1. Manifest-driven worker backends (source capability).** (L)
  Redux's pinned manifest and a generic local dispatcher implement the existing
  `Y`/`T`/`R`/`E` framing. An offline fake second executable backend works without
  edits to C++ worker/runtime; only Redux has a shipped inference engine. New
  backends still require license/runtime and actual inference validation.
- [x] **C2. Model/backend state and chooser (local source).** (L)
  Settings lists manifest-backed model status and active/loading/ready/failure
  states; selection persists and invalidates/restarts worker, clip and review.
  Install → Confirm Install displays pinned source, size, license, attribution and
  manifest SHA-256; confirmation launches an owned installer helper with that
  digest and offline rechecks afterward. Full consent metadata is paginated;
  the panel shows bounded per-file download/verification events and sanitized
  errors. Installer output alone never proves success. No implicit downloads or
  runtime install. **Installed chooser/consent behavior on Frame remains unvalidated.**
- [x] **C3. Offline model status CLI.** (S) `frameyap --list-models` and
  `--check-model ID` dispatch pinned local manifest/hash checks, without ASR or
  downloads. Installed archive CLI still needs clean-account verification.

## D. Installer

- [x] **D1. Local binary-or-source installer paths.** (L)
  `install.sh` has a checksummed binary route and explicit local source/toolchain
  preflight/build/stage/install route, with rollback and no default registration.
  A native archive was exercised in private HOME/XDG roots on Frame: install,
  installed model-status CLI, idempotency, wrapper repair and uninstall. This
  same-user/system-library fixture is not clean-account acceptance.
- [x] **D2. Attended-or-unattended model-agnostic installer interface.** (M)
  TTY choices have equivalent flags; `--mode binary|source`, `--backend ID`,
  `--model-dir`, `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`,
  `--print-plan` and `--json` support offline plans and structured outcomes;
  explicit model installs use installed pinned manifests. Tested with local
  fixtures only, not a released archive or an installed Frame UI handoff.
- [x] **D3. Publish a prebuilt ARM64 archive.** (M) `scripts/build-release.sh`
  builds SDL3/OpenVR-pinned archives on Linux ARM64; v0.1.202609251524 is built
  on the Frame and published with its checksum and `install.sh`. The archive has
  no Kestrel/Torch/moondream/model; those are user-side downloads.
- [x] **D4. Installer fetches the Python runtime with pip.** (M) Implemented
  2026-09-25 as explicit `install.sh --install-runtime --yes`: user-local venv,
  CPU `torch==2.8.0` from PyTorch's CPU index, `moondream==2.4.0` from PyPI,
  import/no-CUDA verification, then `python=` in `paths.conf`. Offline tests
  mock pip; a real run passed on x86-64 Python 3.12. Needs the clean Frame
  install to confirm on ARM64.

## E. Naming and versioning

- [x] **E1. Name: keep "FrameYap" for v0.1.** Frame (the hardware) + yap (speech)
  says what it is; no rename churn before the first tag. If a hardware-neutral
  project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
  front end), decide it before the cross-window work in G, not now.
- [x] **E2. User-facing Type / Type + Enter / Quick phrases labels.** (S)
  Overlay, diagnostics, README/help and overlay docs align on these controls;
  `insert`, `enter`, `quick_chat` remain internal binding/API names. Settings
  explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model
  from its `moondream` Python inference package.
- [x] **E3. Numeric `MAJOR.MINOR.YYYYMMDDHHMM` version source work.** (S)
  CMake, CLI/tests, package producer and installer use numeric release versions;
  dev git info is a separate `--version` line. `SOURCE_DATE_EPOCH` can supply
  the configuration timestamp. A clean release tag/archive still needs D3.

## F. Code structure (non-urgent)

- [x] **F1. Move `--check-*` diagnostics out of `main.cpp`.** (S)
  `src/check.cpp` owns native checks; `src/cli.cpp` provides the table-driven
  parser. Device behavior remains separately gated.
- [x] **F2. Split `overlay.cpp`'s `Impl`.** (M)
  Drag state, persistence/save failures and diagnostics now have separate grouped
  owners; native snapshot build and offline panel/drag tests cover the refactor.

## G. Other windows (post-release)

Text delivery already works into a WezTerm window on Frame (owner-tested; this is
not the recorded live acceptance in H). Deeper integration of other windows with
this app is future design and out of scope for v0.1.

- [ ] **G1. Validate browser text fields via the Gamescope input path.** (M)
  Highest-priority target. Record which fields accept Type / Type + Enter (plain
  inputs, textareas, rich editors, password fields should be expected to differ) and
  document the results honestly.
- [ ] **G2. Later, if needed:** a KDE desktop-mode backend behind `DeliveryLease`,
  and a uinput backend as a last resort. Not scheduled; uinput needs `/dev/uinput`
  access and types with no focus check, which conflicts with the project's
  no-sudo/udev rule and per-window authorization.

## H. Carried over from the earlier TODO (hardware validation)

- [x] Basic menu launch on Frame: the user found FrameYap in Steam's **Non-Steam**
  section; the panel showed and Quit worked. This does not validate transcription,
  recording, text delivery, cold startup, or which shortcut discovery mechanism Steam
  used.
- [ ] Verify the exact missing-runtime panel message and Steam's shortcut persistence
  across a normal restart (without restarting sessions just for the test). Confirm no
  runtime/model environment override before future live checks.
- [x] Live acceptance on Frame: microphone → reviewed text → real target delivery.
  Confirmed by the owner's daily live voice typing (2026-09-25). Auto insert with
  speech and physical resize were not separately reported.

---

## Decisions

- Redux runtime: treated as usable for local inference per upstream's Kestrel
  README; not bundled in our archives (A1 done).
- Multiple backends + a model chooser UI: wanted (C1–C3). The panel can trigger the
  install on an explicit click, for non-technical users.
- Name: keep FrameYap. v0.1 is a first small release, not a POC.
- Close-mic-when-idle: setting, default off (B3).
- Labels: Type / Type + Enter (E2).
- Installer: binary or source, continues automatically after choices, fully flag-
  driven for agent use (D1–D2).
- Version: `MAJOR.MINOR.YYYYMMDDHHMM`, git hash only in dev-build `--version` output
  (E3).
- Other windows: browser text fields first (G1); deeper integration is future design.

## Open questions

The fresh Frame install passed (2026-09-25). Open follow-ups: the
missing-runtime panel message, which batteries Frame reports, browser text
fields (G1), and whether the installer's y/n defaults suit first-time users.