FrameYap TODO

Compiled from the 2026-09-24 holistic review, with owner decisions applied (see "Decisions" at the end). Items are sized to hand off individually. Size: S ≈ hours, M ≈ a day, L ≈ multi-day.

Guardrails from AGENTS.md apply to every item: offline default build, no implicit downloads, no automatic Enter, small verified commits, docs must state implemented vs proposed behavior honestly. Checked source-work items below mean the local implementation is present, not a shipped or headset-accepted release.

Implemented locally; offline tested; installed/headset validation pending

The current working tree includes the source changes for A3, A5–A7, B1–B4, C1–C3, D1–D2, E2–E3 and F1–F2. Final offline checks passed 24/24 default, 25/25 strict UI, and the fake Wayland input regression suite. An earlier private native ARM64 snapshot passed 28/28; the final source requires a fresh native build before deployment. These checkboxes close the source tasks, not their empirical acceptance gates. C1's second backend is a fake executable fixture, not a second shipped ASR engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's source/attended installer paths still need an audited native archive and an installed clean-account/Frame exercise. A4 (exact artifact ABI/license closure), D3 (publication and clean-account acceptance), P1 (real-target delivery), G1 (browser), and H (live headset acceptance) remain open. No new build is claimed deployed to Frame; local code/tests cannot establish a fixed delivery regression.

Priority regression (reported during implementation)

P1. Repeated delivery loses the beginning of later submissions. After the first Type + Enter, later text reportedly loses a dozen to a few dozen leading bytes. Investigate preview versus destination loss, retain the full bounded literal transcript, and add repeated/long/Unicode delivery regression tests. Do not assume a larger buffer fixes it or retry uncertain delivery automatically. An old-code delivery fixture crashed the Gamescope session, not the OS; this is not evidence of a fix. Plain-ASCII prefix corruption was reproduced; retained IME and immediate byte chunking did not fix it. A nonblocking, focus-guarded 24-codepoint/150 ms delivery queue is implemented with offline regressions, including full-length Unicode and all-space batches. Clipboard remains untouched. Real-target confirmation is still required after deployment.

A. First release (v0.1) blockers

A1. Remove the runtime-license blocker notes. Upstream's Kestrel README states local inference is free; blocker text removed from README/docs/scripts and replaced with a neutral dependency note in docs/third-party.md.
A2. Remove references to the unrelated app. (S) Tracked references were removed while preserving Inconsolata/OFL attribution, font SHA-256 and upstream googlefonts/Inconsolata source. A tracked, case-insensitive grep for the former app name must remain empty.
A3. Commit the pending AGENTS.md rename ("Frame Dictation" → "FrameYap"). (S) Done in baseline checkpoint 07c03ea.
A4. Inventory the remaining runtime dependencies' licenses. (M) Upstream inventory and the FreeType FTL choice are documented, but the exact staged ARM64 native/runtime binaries, transitive wheel/library notices, symbol versions, loader and libc floor still require artifact-specific review before publishing any prebuilt archive.
A5. Drop "POC" from the shipped surface. (S) Public help, README, CMake and installer use the release name. scripts/stage-native-poc.py remains a deprecated compatibility wrapper for scripts/stage-native.py, not the documented or shipped staging entry point. v0.1 is a first small release.
A6. Rewrite the README front. (M) Pitch, requirements, local-only install, controls table and explicit "Status / not yet validated" section are present.
A7. Archive docs. Dated evidence (docs/evidence/) and provenance.md moved to the untracked, gitignored docs/archive/; poc.md renamed docs/build.md. Current design/overlay/packaging docs distinguish implemented local behavior from proposed and unaccepted headset/release behavior.

B. Correctness / robustness

B1. Keep malformed transcripts request-local. (S) Controller::tick() catches a correlated bad UTF-8/control reply and fails the request without stopping the ready worker; hardware-free fakes test retry.
B2. Use engine-neutral C++ worker errors. (S) F/I messages no longer name Redux's Python engine.
B3. "Close mic when idle" setting, default OFF. (M) Config/Settings and fake-backed mic-lifetime tests cover default idle draining versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
B4. Extract/test interaction logic from run(). (L) Controller receives injectable audio/worker/focus/delivery interfaces; offline tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.

C. Backends and model management

C1. Manifest-driven worker backends (source capability). (L) Redux's pinned manifest and a generic local dispatcher implement the existing Y/T/R/E framing. An offline fake second executable backend works without edits to C++ worker/runtime; only Redux has a shipped inference engine. New backends still require license/runtime and actual inference validation.
C2. Model/backend state and chooser (local source). (L) Settings lists manifest-backed model status and active/loading/ready/failure states; selection persists and invalidates/restarts worker, clip and review. Install → Confirm Install displays pinned source, size, license, attribution and manifest SHA-256; confirmation launches an owned installer helper with that digest and offline rechecks afterward. Full consent metadata is paginated; the panel shows bounded per-file download/verification events and sanitized errors. Installer output alone never proves success. No implicit downloads or runtime install. Installed chooser/consent behavior on Frame remains unvalidated.
C3. Offline model status CLI. (S) frameyap --list-models and --check-model ID dispatch pinned local manifest/hash checks, without ASR or downloads. Installed archive CLI still needs clean-account verification.

D. Installer

D1. Local binary-or-source installer paths. (L) install.sh has a checksummed binary route and explicit local source/toolchain preflight/build/stage/install route, with rollback and no default registration. A native archive was exercised in private HOME/XDG roots on Frame: install, installed model-status CLI, idempotency, wrapper repair and uninstall. This same-user/system-library fixture is not clean-account acceptance.
D2. Attended-or-unattended model-agnostic installer interface. (M) TTY choices have equivalent flags; --mode binary|source, --backend ID, --model-dir, --yes, --autolaunch/--no-autolaunch, --without-model, --print-plan and --json support offline plans and structured outcomes; explicit model installs use installed pinned manifests. Tested with local fixtures only, not a released archive or an installed Frame UI handoff.
D3. Publish a first prebuilt ARM64 archive. (M) Depends on A4. Follow the release checklist in docs/packaging.md; do not advertise the one-command route until the archive and its checksum are actually published and tested from a clean account.

E. Naming and versioning

E1. Name: keep "FrameYap" for v0.1. Frame (the hardware) + yap (speech) says what it is; no rename churn before the first tag. If a hardware-neutral project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame front end), decide it before the cross-window work in G, not now.
E2. User-facing Type / Type + Enter / Quick phrases labels. (S) Overlay, diagnostics, README/help and overlay docs align on these controls; insert, enter, quick_chat remain internal binding/API names. Settings explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model from its moondream Python inference package.
E3. Numeric MAJOR.MINOR.YYYYMMDDHHMM version source work. (S) CMake, CLI/tests, package producer and installer use numeric release versions; dev git info is a separate --version line. SOURCE_DATE_EPOCH can supply the configuration timestamp. A clean release tag/archive still needs D3.

F. Code structure (non-urgent)

F1. Move --check-* diagnostics out of main.cpp. (S) src/check.cpp owns native checks; src/cli.cpp provides the table-driven parser. Device behavior remains separately gated.
F2. Split overlay.cpp's Impl. (M) Drag state, persistence/save failures and diagnostics now have separate grouped owners; native snapshot build and offline panel/drag tests cover the refactor.

G. Other windows (post-release)

Text delivery already works into a WezTerm window on Frame (owner-tested; this is not the recorded live acceptance in H). Deeper integration of other windows with this app is future design and out of scope for v0.1.

G1. Validate browser text fields via the Gamescope input path. (M) Highest-priority target. Record which fields accept Type / Type + Enter (plain inputs, textareas, rich editors, password fields should be expected to differ) and document the results honestly.
G2. Later, if needed: a KDE desktop-mode backend behind DeliveryLease, and a uinput backend as a last resort. Not scheduled; uinput needs /dev/uinput access and types with no focus check, which conflicts with the project's no-sudo/udev rule and per-window authorization.

H. Carried over from the earlier TODO (hardware validation)

Basic menu launch on Frame: the user found FrameYap in Steam's Non-Steam section; the panel showed and Quit worked. This does not validate transcription, recording, text delivery, cold startup, or which shortcut discovery mechanism Steam used.
Verify the exact missing-runtime panel message and Steam's shortcut persistence across a normal restart (without restarting sessions just for the test). Confirm no runtime/model environment override before future live checks.
Live acceptance on Frame: microphone → reviewed text → real target delivery, Auto insert with speech, physical resize and a verified deployed version.

Decisions

Redux runtime: treated as usable for local inference per upstream's Kestrel README; not bundled in our archives (A1 done).
Multiple backends + a model chooser UI: wanted (C1–C3). The panel can trigger the install on an explicit click, for non-technical users.
Name: keep FrameYap. v0.1 is a first small release, not a POC.
Close-mic-when-idle: setting, default off (B3).
Labels: Type / Type + Enter (E2).
Installer: binary or source, continues automatically after choices, fully flag- driven for agent use (D1–D2).
Version: MAJOR.MINOR.YYYYMMDDHHMM, git hash only in dev-build --version output (E3).
Other windows: browser text fields first (G1); deeper integration is future design.

Open questions

Release artifact compatibility/license audit, publication, P1 real-target behavior and live headset validation are unresolved gates, not implied by checked source tasks.

11 KiB Raw Blame History Unescape Escape