diff --git a/README.md b/README.md index 20af4c9..feae655 100644 --- a/README.md +++ b/README.md @@ -1,138 +1,47 @@ # FrameYap -Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress). +Voice typing on Steam Frame, with recognition on the headset rather than a desktop or cloud server. +Hold a controller button to record, review the transcript, then deliberately type it into the focused app. +Standalone MIT-licensed OpenVR overlay; no Steam store AppID or sudo. First v0.1 release is in progress. -Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture, -persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope, -and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR -server, cloud fallback or unrelated application dependency. +## Requirements -**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility, -Gamescope discovery and native-only installation have been exercised on Frame. -Live microphone → reviewed text → real target delivery is **not yet accepted**. +- Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format. Binary compatibility must be checked against each actual release artifact, not inferred from a developer build. +- For voice recognition, separately provision a compatible **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0 and dependencies) and the pinned local Parakeet Redux model. Neither is bundled or installed with pip by the current native-only installer. There is no fallback ASR service. +- A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](docs/build.md). -**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the -`moondream` Python package (its Kestrel runtime states that local inference is free -and needs no API key). The build and tests never download it; the installer or you -install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md). -No GitHub release is published yet. +## Install (local artifacts only) -## Controls - -- **Right X (default Frame binding):** hold to record; release to - transcribe. Repeated presses reached the controls-only diagnostic on Frame; - live mic capture through this shortcut still needs guided acceptance. This - PTT action is remappable through SteamVR bindings. -- **Right B:** cancel/discard (or close quick chat). **Right A:** insert reviewed text with a trailing space. - **Right Y:** open quick chat; press again to cycle its highlighted choice. - **Submit** on the overlay (or double-tap left grip when active) sends that choice - verbatim then Enter; outside quick chat it inserts pending review + Enter, or - sends Enter alone when there is no text. Edit the `quick_inputs` list in - `$XDG_CONFIG_HOME/frameyap/config.json` (restart to apply). Nothing submits - automatically. Auto Insert is opt-in and off by default. -- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Submit and Hold Quit - (hold the button for 0.9 seconds before releasing). - Review and settings share one Inconsolata/neon-framed surface. The header - shows local time/date; Settings selects 12/24-hour time and date format/off. - Hold the thin bar below the panel to freely move and rotate it with your - controller; release to leave it at that pose. Drag the external lower-right - bracket to scale it (world, head or wrist), keeping the upper-left anchored. - Both handles use the app's gradient in transparent margins, like Steam's - window handles. Settings → **Lock grab/scale** hides and disables both handles; - the lock is saved. Position/size changes last for this run only. - Wrist mounting fades the panel as its full orientation turns away from an - upright viewer-facing pose (60°–75°), hiding interaction past that angle; - world and head mounting do not fade. - **Bindings** requests SteamVR's binding editor directly. Existing SteamVR overrides may supersede defaults. World-space by - default; settings offer left wrist, right wrist and head mounting, plus recenter. - Dashboard lasers provide clickable controls. Settings → Lasers anytime is an - opt-in, default-off system-wide laser mode while the panel is visible; it may - affect games and is separate from experimental input overrides. -- **Theme and controls:** optional `$XDG_CONFIG_HOME/frameyap/config.json` selects - panel colors, a font path and Frame controller button mappings; missing fonts - fall back to bundled Inconsolata. The installer creates/checks this file and - backs it up before repairs. See [overlay configuration](docs/overlay.md#user-theme-and-controller-configuration). -- **Advanced debugging:** Settings toggle / `"advanced_debug": true` in config. - Off by default. Restarts the worker and discards current work; full exceptions, - worker output and transcripts go to private, bounded local logs. No raw audio - archive. See [diagnostics](docs/worker.md#advanced-debugging). -- **Experimental input priority:** set `"input_priority": "experimental"` in - that config and enable SteamVR's Developer option **Enable global input from - overlays**. FrameYap then requests priority for its bound controller sources. - This may consume controls used by games or the dashboard; coexistence on Frame - is under test. The default is `"normal"`; restart FrameYap after changing it. -- **Review by default:** focus your destination, then press Insert (text + space) or - explicitly Enter (text + space, then Enter). Settings → Auto insert is off by default: - when enabled, it queues text + space only if Xwayland keyboard focus, active - window and Gamescope focus match continuously from recording through delivery. - Any uncertainty leaves a preview for manual Insert; it never sends Enter. - This is not yet validated for live transcription on Frame. Maximum clip 20 seconds; - accidental taps under 200 ms are discarded. -- While the native app is Ready, it keeps the mic device open and discards idle - audio instead of opening/closing on every PTT. Quit releases the device. Other - apps may still hear/transmit your voice; FrameYap does not mute them. - -Physical gesture timing, global bindings during games, mic capture, target-app -compatibility and headset comfort still need coordinated validation. Successful -API initialization is not delivered input or human acceptance. - -## Build and test offline - -CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests. +**No public release is published.** To install a locally vetted, checksummed ARM64 native-only archive without a compiler: ```sh -cmake -S . -B build -cmake --build build -ctest --test-dir build --output-on-failure -./build/frameyap --help -``` - -Default build has no hardware backends. It never downloads packages/models or -initializes SteamVR, a microphone or input injection. Native dependencies and -explicit launch/check commands are documented in [the build guide](docs/build.md). -`--version` uses an ISO-like UTC build timestamp (with Git hash when available), -not a numbered release; use that same tag when packaging the binary. - -## Installation - -The real installer accepts a versioned, checksummed prebuilt ARM64 archive: - -```sh -sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \ +sh install.sh --mode binary --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \ --sha256 ARCHIVE_SHA256 --version VERSION ``` -This is the **local-artifact command shape**, not an available public download. -Installation is user-local, retains rollback, refuses active-app upgrades and -foreign files, and does not launch or register automatically. Registration uses -OpenVR application key `local.frameyap.overlay`, not a Steam store AppID. -Autolaunch is opt-in. An installed desktop entry can be selected manually as a -non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the -panel on one Frame; registration alone did not show an entry in the first checked -dashboard menu. See [packaging and lifecycle](docs/packaging.md). -For a native-only install, menu-driven inference needs a Python runtime you -provide and the pinned model. Configure their absolute paths in -`~/.config/frameyap/paths.conf` as described in the packaging guide; they are -never fetched or bundled implicitly. +`VERSION` is numeric `0.1.YYYYMMDDHHMM` for this release line; a future release tag will be `vVERSION`. The installer has an explicit local source-build path with toolchain/dependency inputs, and a read-only `--print-plan`/machine-readable `--json` mode; the producer stage→archive path still needs a clean-account artifact test and release audit. See [packaging](docs/packaging.md). Installation is user-local with rollback, no automatic launch or registration, and no Steam store AppID. Before voice typing, independently supply a licensed Python CPU environment and pinned weights, and configure their absolute paths in `~/.config/frameyap/paths.conf` (or the XDG config equivalent). Model download is opt-in and separate: `sh install.sh --install-model --backend redux --print-plan --json` inspects pinned metadata, while `--yes` explicitly authorizes installation from the *installed* manifest; `--expected-manifest-sha256 HASH` additionally binds consent to its exact bytes. The locally implemented Settings → Models chooser offers selection/restart and a two-click Install/Confirm Install with source, size, license, attribution and manifest SHA-256; it has not been validated as an installed/headset flow. No chooser action installs the Python runtime. Do not mistake a native-only install for working ASR. The pinned GitHub download route must not be advertised as functional until a vetted release exists. -A pinned GitHub one-command route is implemented in `install.sh --version TAG`, -but **do not advertise or run it as a working public installation until a vetted -release exists**. Native-only artifacts support the overlay/checks without an -end-user compiler; a bundled-ASR distribution is not yet offered. +## Controls (default Frame binding) -## Project map +| Button / control | Action | +| --- | --- | +| Right X, hold / release | Record while held; release to transcribe. | +| Right B | Cancel/discard, or close the Quick phrases picker. | +| Right A | **Type:** queue reviewed text, normally with a trailing space. | +| Right Y | Open **Quick phrases**; press again to cycle the selection. | +| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. | +| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. | +| Overlay Type / Type + Enter | The same deliberate text / Enter actions. | +| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). | -- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests. -- [Design](docs/design.md): full target design; some features remain proposed. -- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md). -- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md), - [dependency/license inventory](docs/third-party.md). -- [TODO](TODO.md): release plan and open work items. +Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Check the focused destination before Type or Type + Enter. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](docs/overlay.md). -Dated device-test records are kept locally (untracked) and are not authority for -current device availability. Live microphone → reviewed text → target delivery has -not been formally accepted; see Status above. +## Status / not yet validated -No recordings, transcripts, private logs, model weights or runtime binaries are -committed. The worker boundary is intentionally small for forks experimenting -with other models/APIs; the default remains local-only Redux. +Native ARM64 build, CPU inference on a public clip, overlay visibility, Gamescope API discovery, a controls-only Right X probe and native-only installation have been exercised on Frame. **Live microphone → reviewed transcript → real target delivery has not been formally accepted.** Focus guard and insertion have offline/owned-target checks, not general app compatibility or human headset acceptance. Global bindings, physical gesture feel, Auto insert with speech, latency, battery/thermal cost and controller coexistence still require opt-in headset testing. Front-prefix loss on repeated submissions (P1) remains under separate investigation; do not treat it as fixed. An old-code fixture crashed the Gamescope session, not the OS, and does not prove the new delivery path. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it. + +The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](docs/overlay.md), [worker](docs/worker.md), [design](docs/design.md). + +Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](docs/third-party.md); [TODO](TODO.md). + +No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection. diff --git a/TODO.md b/TODO.md index f1c03d3..66b985c 100644 --- a/TODO.md +++ b/TODO.md @@ -6,95 +6,110 @@ M ≈ a day, L ≈ multi-day. Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit downloads, no automatic Enter, small verified commits, docs must state implemented -vs proposed behavior honestly. +vs proposed behavior honestly. Checked source-work items below mean the local +implementation is present, **not** a shipped or headset-accepted release. + +## Implemented locally; offline tested; installed/headset validation pending + +The current working tree includes the source changes for A3, A5–A7, B1–B4, +C1–C3, D1–D2, E2–E3 and F1–F2. Offline checks passed 23/23 default, +24/24 strict UI, and 28/28 on a private native ARM64 snapshot. Later review +fixes require a fresh final native build before deployment. +These checkboxes close the *source tasks*, not their empirical acceptance gates. +C1's second backend is a fake executable fixture, **not** a second shipped ASR +engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's +source/attended installer paths still need an audited native archive and an +installed clean-account/Frame exercise. A4 (exact artifact ABI/license closure), +D3 (publication and clean-account acceptance), P1 (real-target delivery), G1 +(browser), and H (live headset acceptance) remain open. No new build is claimed +deployed to Frame; local code/tests cannot establish a fixed delivery regression. + +## Priority regression (reported during implementation) + +- [ ] **P1. Repeated delivery loses the beginning of later submissions.** After + the first Type + Enter, later text reportedly loses a dozen to a few dozen leading + bytes. Investigate preview versus destination loss, retain the full bounded + literal transcript, and add repeated/long/Unicode delivery regression tests. + Do not assume a larger buffer fixes it or retry uncertain delivery automatically. + An old-code delivery fixture crashed the Gamescope session, **not** the OS; + this is not evidence of a fix. Real-target confirmation remains required after + a tested fix is deployed. ## A. First release (v0.1) blockers - [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README states local inference is free; blocker text removed from README/docs/scripts and replaced with a neutral dependency note in `docs/third-party.md`. -- [x] **A2. Remove references to the unrelated app (kouseki).** (S) - Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a - final grep. Keep the Inconsolata/OFL attribution and cite the - upstream font source (googlefonts/Inconsolata) instead. - *Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains. -- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" → - "FrameYap"). (S) +- [x] **A2. Remove references to the unrelated app.** (S) + Tracked references were removed while preserving Inconsolata/OFL attribution, + font SHA-256 and upstream googlefonts/Inconsolata source. A tracked, + case-insensitive grep for the former app name must remain empty. +- [x] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" → + "FrameYap"). (S) Done in baseline checkpoint `07c03ea`. - [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M) - Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL - option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt - archive that includes any of them. -- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README, - `scripts/stage-native-poc.py`, `CMakeLists.txt` messages, - installer strings. v0.1 is a first small release. -- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install, - controls **table** (button → action), then a "Status / not yet validated" section. - Today it reads as a lab notebook and Controls is a wall of text. + Upstream inventory and the FreeType FTL choice are documented, but the exact + staged ARM64 native/runtime binaries, transitive wheel/library notices, symbol + versions, loader and libc floor still require artifact-specific review before + publishing any prebuilt archive. +- [x] **A5. Drop "POC" from the shipped surface.** (S) Public help, README, + CMake and installer use the release name. `scripts/stage-native-poc.py` remains + a deprecated compatibility wrapper for `scripts/stage-native.py`, not the + documented or shipped staging entry point. v0.1 is a first small release. +- [x] **A6. Rewrite the README front.** (M) Pitch, requirements, local-only install, + controls table and explicit "Status / not yet validated" section are present. - [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`. - Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and - hedging language before v0.1. + Current design/overlay/packaging docs distinguish implemented local behavior + from proposed and unaccepted headset/release behavior. ## B. Correctness / robustness -- [ ] **B1. A malformed transcript must not kill the worker.** (S) - `src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control - characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and - `session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a - request-level error (like the `E` path: keep the worker, show "transcription - failed", allow retry). Add a test with a control-character reply asserting the - worker stays ready. -- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes - "moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a - worker-supplied message code. Prerequisite for C1. -- [ ] **B3. "Close mic when idle" setting, default OFF.** (M) - Default keeps the mic open while Ready (opening/closing per PTT causes an audio - spike on the physical hardware, and it avoids first-syllable clipping). The - setting closes it between clips for people who don't want a live device. Document - the tradeoff (spike/latency) next to the toggle, in Settings and in the docs. - Persist in `config.json`; add to the panel Settings tab. -- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L) - `run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no - tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`) - so PTT, cancel, quick phrases, auto-insert and error transitions are testable - without hardware. B1 is the first regression test. +- [x] **B1. Keep malformed transcripts request-local.** (S) + `Controller::tick()` catches a correlated bad UTF-8/control reply and fails the + request without stopping the ready worker; hardware-free fakes test retry. +- [x] **B2. Use engine-neutral C++ worker errors.** (S) + `F`/`I` messages no longer name Redux's Python engine. +- [x] **B3. "Close mic when idle" setting, default OFF.** (M) + Config/Settings and fake-backed mic-lifetime tests cover default idle draining + versus opt-in close/reopen; the physical spike/latency tradeoff is documented. +- [x] **B4. Extract/test interaction logic from `run()`.** (L) + `Controller` receives injectable audio/worker/focus/delivery interfaces; offline + tests cover PTT, cancel, phrases, Auto insert and error/retry transitions. ## C. Backends and model management -- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L) - Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper, - sherpa-onnx Parakeet, …) as separate worker executables speaking the existing - `Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name, - launcher, pinned model files + hashes + attribution, license text, CPU/GPU - requirements) so nothing is hardcoded in C++ or `model_files.py`. - *Done when:* Redux is expressed as a manifest, and a second backend can be added - without touching `worker.cpp`/`runtime.cpp`. -- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4. - Settings page listing backends/models with state (not installed / installed and - verified / loading / ready / failed), the active one marked, and selection that - restarts the worker. Target user is non-technical: an **Install** button on the - panel runs the installer's machine-readable mode (D2) as a child process and shows - progress/errors in the panel, so nobody needs a terminal. The click is the consent; - there are still no implicit or background downloads, and the panel states what - will be downloaded and how large it is before it starts. Persist the choice in - `config.json`. -- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID` - (offline, hash-verifies installed files) so the UI and installer share one - implementation. +- [x] **C1. Manifest-driven worker backends (source capability).** (L) + Redux's pinned manifest and a generic local dispatcher implement the existing + `Y`/`T`/`R`/`E` framing. An offline fake second executable backend works without + edits to C++ worker/runtime; only Redux has a shipped inference engine. New + backends still require license/runtime and actual inference validation. +- [x] **C2. Model/backend state and chooser (local source).** (L) + Settings lists manifest-backed model status and active/loading/ready/failure + states; selection persists and invalidates/restarts worker, clip and review. + Install → Confirm Install displays pinned source, size, license, attribution and + manifest SHA-256; confirmation launches an owned installer helper with that + digest and offline rechecks afterward. Full consent metadata is paginated; + the panel shows bounded per-file download/verification events and sanitized + errors. Installer output alone never proves success. No implicit downloads or + runtime install. **Installed chooser/consent behavior on Frame remains unvalidated.** +- [x] **C3. Offline model status CLI.** (S) `frameyap --list-models` and + `--check-model ID` dispatch pinned local manifest/hash checks, without ASR or + downloads. Installed archive CLI still needs clean-account verification. ## D. Installer -- [ ] **D1. Installer with a binary-or-source choice.** (L) - `install.sh` offers *prebuilt archive* (checksummed) or *build from source* - (checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then - continues automatically through install after the user's choices. Retain rollback, - idempotency, no sudo, no Steam AppID, opt-in autolaunch. -- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M) - Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`, - `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`, - `--json` output) so a model/agent can run it non-interactively; interactive - prompts only run on a TTY and print the equivalent flags they chose. Exit codes - and messages must be machine-readable. +- [x] **D1. Local binary-or-source installer paths.** (L) + `install.sh` has a checksummed binary route and explicit local source/toolchain + preflight/build/stage/install route, with rollback and no default registration. + A native archive was exercised in private HOME/XDG roots on Frame: install, + installed model-status CLI, idempotency, wrapper repair and uninstall. This + same-user/system-library fixture is not clean-account acceptance. +- [x] **D2. Attended-or-unattended model-agnostic installer interface.** (M) + TTY choices have equivalent flags; `--mode binary|source`, `--backend ID`, + `--model-dir`, `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, + `--print-plan` and `--json` support offline plans and structured outcomes; + explicit model installs use installed pinned manifests. Tested with local + fixtures only, not a released archive or an installed Frame UI handoff. - [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the release checklist in `docs/packaging.md`; do not advertise the one-command route until the archive and its checksum are actually published and tested from a clean @@ -106,32 +121,24 @@ vs proposed behavior honestly. says what it is; no rename churn before the first tag. If a hardware-neutral project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame front end), decide it before the cross-window work in G, not now. -- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text + - space) and **Type + Enter**; align overlay buttons, `--check-controls` output, - README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only - if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings - explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux" - vs `moondream` naming once in `docs/worker.md`. -- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S) - e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts - correctly, keeps the build date visible, URL/filename-safe. The version field is - just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted - changes) suffixes were only for telling developer builds apart, so they move out - of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev - build adds a second line like `git abc12345 (uncommitted changes)`. Release - archives are built from a clean tag, so users never see it. Update the CMake - version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`, - installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the - timestamp. Tag releases `v0.1.`. +- [x] **E2. User-facing Type / Type + Enter / Quick phrases labels.** (S) + Overlay, diagnostics, README/help and overlay docs align on these controls; + `insert`, `enter`, `quick_chat` remain internal binding/API names. Settings + explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model + from its `moondream` Python inference package. +- [x] **E3. Numeric `MAJOR.MINOR.YYYYMMDDHHMM` version source work.** (S) + CMake, CLI/tests, package producer and installer use numeric release versions; + dev git info is a separate `--version` line. `SOURCE_DATE_EPOCH` can supply + the configuration timestamp. A clean release tag/archive still needs D3. ## F. Code structure (non-urgent) -- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of - inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a - table-driven option parser. -- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag - state, save-failure flags, counters, pose caches): separate drag, persistence and - diagnostics. +- [x] **F1. Move `--check-*` diagnostics out of `main.cpp`.** (S) + `src/check.cpp` owns native checks; `src/cli.cpp` provides the table-driven + parser. Device behavior remains separately gated. +- [x] **F2. Split `overlay.cpp`'s `Impl`.** (M) + Drag state, persistence/save failures and diagnostics now have separate grouped + owners; native snapshot build and offline panel/drag tests cover the refactor. ## G. Other windows (post-release) @@ -158,7 +165,7 @@ this app is future design and out of scope for v0.1. across a normal restart (without restarting sessions just for the test). Confirm no runtime/model environment override before future live checks. - [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery, - Auto Insert with speech, physical resize. + Auto insert with speech, physical resize and a verified deployed version. --- @@ -179,4 +186,5 @@ this app is future design and out of scope for v0.1. ## Open questions -None currently blocking. +Release artifact compatibility/license audit, publication, P1 real-target behavior +and live headset validation are unresolved gates, not implied by checked source tasks. diff --git a/docs/build.md b/docs/build.md index 0c13fc8..f653e34 100644 --- a/docs/build.md +++ b/docs/build.md @@ -11,8 +11,9 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input. mode while FrameYap is visible; it may affect games, and is not an input override. - Remappable SteamVR actions. The default Steam Frame binding maps right X (hold to record, release to transcribe) to the existing PTT action using the - observed `frame_controller` profile. Right B cancels, A inserts + space, and Y - inserts pending text + Enter (or Enter only with no preview). The Bindings button + observed `frame_controller` profile. Right B cancels, A requests Type (text + + space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay + control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button requests SteamVR's remapping editor directly. Grip bindings remain, but both grip actions were inactive in the observed dashboard check; do not rely on them. If left grip becomes active, two short taps request explicit Enter. @@ -22,27 +23,41 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input. - SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum, 20 second maximum. In the native app the microphone stream opens after model warm-up, stays running between utterances, and discards idle samples; PTT does - not open/pause/close the device. Quit, worker restart, device failure or capture - failure closes it. This avoids repeated capture-device transitions but does not - promise glitch-free playback on every audio stack. Other apps may still + not open/pause/close the device **by default**. Settings → Close mic when idle + (`"close_mic_when_idle": true`, default **false**) closes it between clips and + reopens on PTT; repeated transitions caused an audio spike on Frame and may + add latency or clip the first syllable. Quit, worker restart, device failure + or capture failure also closes it. Keeping it open does not promise glitch-free + playback on every audio stack. Other apps may still transmit your voice: this app does **not** mute VRChat or any other app. +- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed + backend restarts the worker and invalidates pending clip/review/focus authority; + a missing or invalid model disables recording. A separate Install then Confirm + Install exposes pinned source/size/license/attribution and exact manifest-byte + SHA-256 before installer handoff. These are local implementations, not a tested + installed UI or an approved additional inference engine. Only Redux is supplied. - Persistent local Redux worker, correlated bounded pipes, private tmpfs clips, cancellation/reaping and deadlines; exact pinned model SHA-256 verification. - Model imports are lazy and loading is offline. Normal repeats, request-local - transcription failures and microphone failures retain the loaded model; + Model imports are lazy and loading is offline. A correlated bad transcript/ + `E` reply is a request-level failure: Record can retry without dropping the + ready model. Normal repeats and microphone failures retain the loaded model; microphone device failure releases the stream for explicit retry. A cancelled in-flight request or broken worker protocol may require reloading. No cloud/desktop fallback. - Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable - handling, UTF-8/control validation and explicit Submit action for Enter. - Insert ensures a trailing space without doubling an existing one. Enter - first inserts pending review, releases the text lease, then acquires a fresh - lease for Submit. Failed/uncertain text never proceeds to Submit; failed - Submit acquisition never replays text. A full 4096-byte transcript without - room for a space is preserved with an error, never silently truncated. -- Idempotent user-local release-archive installer: SHA-256, safe extraction, - atomic current-version selection, retained rollback, runtime/install lock, - foreign-file refusal and explicit unregister-before-uninstall acknowledgement. + handling, UTF-8/control validation and explicit Type + Enter action. + Type ensures a trailing space without doubling an existing one. Type + Enter + first types pending review, releases the text lease, then acquires a fresh + lease for Enter. Failed/uncertain text never proceeds to Enter; failed + Enter acquisition never replays text. If a validated transcript fills the + 4096-byte bound and lacks a trailing space, Type preserves all its bytes and + queues it **without** the usual space; it does not signal a separate error. + Destination consumption and repeated-delivery behavior remain unaccepted. +- User-local installer with checked binary-archive or explicitly provisioned + source-build mode, safe extraction, atomic current-version selection, retained + rollback, runtime/install lock, foreign-file refusal and explicit + unregister-before-uninstall acknowledgement. Source mode needs a local + compiler, SDK, libraries and license inputs; no runtime is pip-installed. ## Deliberately not claimed @@ -63,15 +78,26 @@ fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip binding are not guaranteed globally active in every scene/dashboard state, and the app never enables SteamVR's experimental overlay overrides on your behalf. -This is a compact prototype panel, not yet the proposed polished miniature status -chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery -impact and target application compatibility require further headset work. +This is one compact panel, not a separate miniature status chip. Front-prefix +loss on repeated submissions (P1) is under separate investigation, **not fixed** +by the chooser/worker documentation or any speculative delivery change. Font +coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact +and target application compatibility require further headset work. ## Inference runtime -Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference -runs through the `moondream` Python package and its Kestrel runtime, which you -install in your own environment; builds/tests never fetch it. See +Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Pinned +file sizes/hashes and attribution live in `assets/backends/redux.json`; offline +`--list-models`/`--check-model redux --model-dir /absolute/model` (or +`scripts/model-status.py`) verify without inference or downloads. Manifest +schema/verification are in `python/frameyap/model_files.py`. The local generic +dispatcher resolves a manifest's in-release Python/executable launcher and +checks request/reply correlation; a new manifest still needs its own licensed, +compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and +`--manifest-dir /absolute/manifests` are wired through +the installed launcher as an explicit override, not a provisioning command. +Inference uses the `moondream` Python package and Kestrel runtime, separately +provisioned in your own environment; builds/tests/installer do not pip-install them. See [third-party notes](third-party.md). ## Developer native build @@ -142,12 +168,16 @@ kestrel 0.8.0) and local weights: ``` Use a disposable text destination first. `--run` loads the model but does not -record until an explicit recording control. Click Insert only after focusing your +record until an explicit recording control. Click Type only after focusing your intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery and terminates only the owned worker. Installer upgrades refuse an active app. -The worker API is intentionally small: a fork can replace the worker implementation -or add its own model/API integration without changing overlay and delivery code. +The `Controller` receives injected `ControllerAudio`, `ControllerWorker`, +`ControllerFocus` and delivery factory interfaces; hardware-free fakes can test +state, retry, authorization and mic lifetime without initializing OpenVR or +recording speech. The worker API is intentionally small: a fork can replace the +worker implementation or add its own model/API integration without changing +overlay and delivery code. The default product remains local-only Redux; extending a fork does not authorize sending existing users' audio to a service. diff --git a/docs/design.md b/docs/design.md index 89a3149..ec53ac6 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,11 +1,16 @@ -# FrameYap native dictation overlay — proposal +# FrameYap native dictation overlay — design and remaining goals Standalone project design. This document is the full target design, not a blanket implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent -Redux adapter, review-first Gamescope insertion and a user-local archive installer. +Redux adapter, review-first Gamescope insertion and a user-local archive installer +with explicit source-build option. Redux's pinned manifest, offline status CLI, +manifest-driven dispatcher and in-panel Models selection/confirmation are +implemented locally. The installed chooser/installer handoff is **not yet +end-to-end verified on Frame**; the native-only archive needs an external CPU +runtime for speech. Opt-in conservative Xwayland focus tracking is implemented locally but not yet -accepted for live automatic typing on Frame. Polished status-chip UX and full hardware -acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md). +accepted for live automatic typing on Frame. A separate status chip and full hardware +acceptance remain possible future work. See [build and scope](build.md) and [third-party notes](third-party.md). Future work is tracked in [TODO.md](../TODO.md). ## Recommendation @@ -17,7 +22,9 @@ as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene renderer, avatar, desktop capture or root service is needed in the primary path. A first-class product goal is a **one-command GitHub install without a Steam store -AppID**. Package a prebuilt native executable and isolated CPU runtime; use a normal +AppID**. Package a prebuilt native executable; the current native-only archive requires a +separately supplied, compatible CPU runtime (no automatic pip install). A future +isolated runtime bundle requires its own license and compatibility audit. Use a normal OpenVR application key for registration, not Steamworks. Installation must remain user-local with opt-in autolaunch. See [installation design](install-design.md). @@ -31,12 +38,14 @@ one persistent local Parakeet Redux CPU worker correlated literal transcript → focus/delivery policy ↓ Gamescope IME set_string + commit → focused Frame application - ↘ overlay preview / error / explicit Insert when delivery is unsafe + ↘ overlay preview / error / explicit Type when delivery is unsafe ``` The CPU worker is local process isolation, not remote inference or a service -framework. Implement a small transport owned by this repository. Keep the model -loaded between utterances; do not spawn Python/load 178 MB for every release. +framework. The locally implemented manifest dispatcher uses a small bounded +transport owned by this repository; only Redux inference is shipped. Keep the +model loaded between utterances; do not spawn Python/load 178 MB for every release. +A correlated bad transcript is a request-level failure, not a model unload. ## Minimal interaction @@ -53,27 +62,29 @@ armed remains a possible refinement, not a second implemented overlay. See Recording… 00:04 [ Cancel ] "The recognized text appears here." -[ Insert + space ] [ Discard ] [ Insert + Enter — explicit ] +[ Type + space ] [ Discard ] [ Type + Enter — explicit ] ``` -- States: disabled, warming, ready, recording, transcribing, review, inserted, - unavailable/error. Recording uses visible icon + text, not color alone. +- States: warming, ready, recording, transcribing, review, input queued, + unavailable/error. Input queued is not an application receipt. Recording uses + visible icon + text, not color alone. - Default Frame bindings: hold right X to speak, release to finish; B cancels, - A inserts with a trailing space, Y opens/cycles the quick-chat selection. - Overlay Submit or left-grip double-tap submits the selected literal + Enter, - or pending review + Enter, or Enter alone if neither is present. - The alternate right-grip tap-then-hold gesture and left-grip double-tap Submit + A types with a trailing space, Y opens/cycles the Quick phrases selection. + Overlay Type + Enter or left-grip double-tap submits the selected literal + Enter, + or pending review (normally + space) then Enter, or Enter alone if neither is present. + The alternate right-grip tap-then-hold gesture and left-grip double-tap Type + Enter remain available. All are remappable through the Bindings button's SteamVR editor. A click-to-start/stop overlay button provides a binding-independent alternative. Bound recording to 20 seconds; discard accidental taps (initial threshold: 200 ms). -- **Quick typing (opt-in, locally implemented):** insert on completion only when +- **Auto insert (opt-in, locally implemented):** type on completion only when uninterrupted Xwayland target observation remains valid. **Review mode (default):** - wait for Insert. Bring up review on uncertainty rather than silently losing a + wait for Type. Bring up review on uncertainty rather than silently losing a transcript or typing into a new target. Live Frame acceptance remains open. -- Submit requires its own explicit control activation: insert pending review with - a trailing space, then queue Enter only if the text step succeeds. With no - review, it queues only Enter. Never interpret "submit", "delete" or other speech +- Type + Enter requires its own explicit control activation: type pending review + (normally with a trailing space), then queue Enter only if the text step succeeds. + At the 4096-byte bound, preserve the entire transcript without a suffix if no + space fits. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech as commands. Transcription completion never auto-submits. - No generic "undo last dictation" initially: another application's edits/cursor cannot be reliably rolled back by a guessed number of backspaces. @@ -109,7 +120,7 @@ license-reviewed extraction, never a runtime path into another project's checkou ### Controller bindings -Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the +Expose PTT, cancel and explicit Type/Type + Enter as named SteamVR actions; let the user bind them. Do not assume a scene app's left-bumper mapping works globally or silently takes a game's button away. Check action activity and neutral rearm. @@ -143,7 +154,7 @@ The server implements text through synthetic key events and a temporary keymap, not a guaranteed rich-text/IME edit operation in every app. Verify actual target toolkits and games. Respect singleton/unavailable handling and Steam-keyboard coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe: -its inspected sample loop is byte-oriented and interprets newline as Submit. +its inspected sample loop is byte-oriented and interprets newline as Enter. ### 2. Explicit fallbacks, not a framework built up front @@ -183,11 +194,11 @@ Gamescope exposes focus-display/window root properties, but their encoding and relationship to seat focus need implementation-specific validation. Do not infer that X display `:0` is always the destination, or that an X focus observation identifies a native Wayland text field. For unobservable native Wayland focus, -require explicit review/Insert; do not advertise safe auto-targeting. +require explicit review/Type; do not advertise safe auto-targeting. -If focus changes, keep the result in review. A fresh Insert explicitly approves +If focus changes, keep the result in review. A fresh Type explicitly approves the current destination and creates a new delivery authorization. Recheck again -at insertion. This minimizes stale delivery but does **not** eliminate a race +at typing. This minimizes stale delivery but does **not** eliminate a race between the final check and global input processing; do not claim otherwise. A Wayland roundtrip means compositor processing, not application consumption. Report `input queued`, never `message sent`. @@ -221,9 +232,11 @@ that tiny dataset; keep transcript visibility and a cheap retry. Implement a small independent audio/worker adapter with explicit capture, single-request bounds, owner-only runtime files, correlated replies and cancellation. -The worker should be implemented independently; do not link, vendor or import -another application's speech code. Avoid a generic provider framework: one -explicit Redux worker is enough for the first version. +The worker is independently implemented; do not link, vendor or import another +application's speech code. A narrow manifest dispatcher now selects a checked +local backend launcher; this is not a general cloud/provider framework and only +Redux is supplied with a runtime implementation. Other manifests require their +own audited offline engine, launcher and tests. Suggested ownership, introduced only as implementation needs it: @@ -243,12 +256,20 @@ processing timeout. No shell commands in IPC and no input authority in the worke native kernel pools. Choose measured latency versus compositor contention, not the desktop's thread count by habit. No real-time scheduling or permanent CPU pinning initially; inspect runtime affinity behaviour during measurement. -- Explicit local model path and offline loading. Missing runtime/weights produces - an actionable error, not an unsolicited download/network fallback. -- Optional explicit enable/warm-up before first PTT; warming must not record audio. - Expose that lifecycle explicitly rather than warming on import or construction. - Otherwise display first-use loading honestly. Keep - model reuse after normal completion; cancellation may restart the owned worker. +- Explicit local model path and offline loading. Pinned files and attribution live + in `assets/backends/redux.json`; `python/frameyap/model_files.py` validates its + schema and hashes. The CLI's offline `--list-models`/`--check-model` checks + never start inference. Settings → Models selects/saves the manifest ID and + restarts the worker, invalidating any audio/review/delivery authorization. + Install requires a separate confirmation showing source, size, license, + attribution and exact-manifest SHA-256; only that click hands off to the + installer for pinned model files. Missing runtime/weights produces an + actionable error, not an unsolicited download/network fallback. +- Native `--run` explicitly starts offline verification then warms a verified + selected local model before PTT; warming must not record audio. Missing or + unverified files disable Record rather than prompting a background download. + Keep model reuse after normal completion and request-local transcript errors; + cancellation may restart the owned worker. - Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded tmpfs-backed clips; remove them on completion, error, cancellation and shutdown. Avoid persistent audio/transcripts by default. Local IPC is not a network hop; @@ -278,14 +299,14 @@ These are proposed implementation gates, **not completed acceptance**: confirm dashboard/hand placement, input events, close/reopen and no scene-focus takeover. Validate global PTT separately rather than blocking the clickable prototype on experimental override support. -3. **Real dictation path:** microphone → local Redux → preview → explicit insert - into a disposable target; then enable quick typing after target tracking tests. +3. **Real dictation path:** microphone → local Redux → preview → explicit Type + into a disposable target; then validate opt-in Auto insert after target tracking tests. Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate replies, lost mic, worker crash and missing model without persisting speech. 4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game text fields; Steam keyboard coexistence; native Wayland targets separately. Test focus changes during capture/inference, rapid loss/regain, held modifiers, - explicit Insert retargeting, no hidden Enter, and no second delivery. + explicit Type retargeting, no hidden Enter, and no second delivery. 5. **In-headset acceptance:** readable feedback and comfortable PTT; measured release-to-insert latency and compositor timing while an actual scene runs; no noticeable sustained thermal/battery regression. Set numeric budgets after @@ -301,8 +322,9 @@ without moving recognition off Frame. It is not part of this initial design. The default hardware-free build needs only CMake and a C++20 compiler (Python runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR, SDL3, FreeType and Wayland client/generated protocol bindings. A separately -authorized Python Redux environment is explicitly supplied at launch. Pin revisions -and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery. +authorized Python Redux environment is explicitly supplied at launch; the +native-only installer neither bundles it nor pip-installs one. Pin revisions +and review licenses for each introduced dependency. No automatic fetch/install in configure or normal tests; no external checkout discovery. Hardware-free tests should cover state transitions, bounded PCM/transcripts, worker framing/timeout/cancellation, duplicate/stale replies and focus generations diff --git a/docs/install-design.md b/docs/install-design.md index b7a9824..0a31a41 100644 --- a/docs/install-design.md +++ b/docs/install-design.md @@ -1,108 +1,82 @@ -# Installation and distribution goal +# Installation goal and current boundary -**User goal:** install from GitHub with a `curl … | bash`-style command, without a -Steam store AppID. An idempotent archive installer and native-only local artifacts -are now implemented/tested; no public release is published. A bundled-ASR -experience is not yet offered. This document retains the target -design; see [current packaging](packaging.md) and [third-party notes](third-party.md). -Planned installer work (binary-or-source choice, flag-driven operation) is in -[TODO.md](../TODO.md). +Goal: a one-command, pinned GitHub release install for Steam Frame without a Steam +store AppID, sudo or end-user compiler. **No public archive or verified clean +install is published. Do not advertise a `curl | sh` command as functional.** +The local installer has binary-archive and explicitly provisioned source-build +modes, machine-readable plans/results and an attended TTY path. The current +native-only artifact does not include or pip-install an ASR runtime; it is not +a one-command voice-typing experience. See [packaging](packaging.md) for exact +flags and [third-party inventory](third-party.md) for open license/ABI audits. -The read-only `scripts/install-preflight.sh` checks whether a host appears suitable -for the **proposed** Linux ARM64 glibc package format and has the expected basic -bootstrap utilities. It reports system Python, Git and uv, but none is required -for the intended bundled release. A read-only check on one Frame observed -Python 3.12.3 and Git, but not uv; availability may change. This script -is not an installer or a model/runtime compatibility test; it has no downloads, -registration, SteamVR initialization or persistent changes. No glibc minimum can -be certified until release artifacts are chosen and tested. +`scripts/install-preflight.sh` is a read-only Linux ARM64/glibc/bootstrap check; +`--source` adds toolchain/library checks. It does not download, install, register, +or certify model/runtime compatibility or a minimum libc version. Its report of +Python/Git/uv availability is not a runtime guarantee: `install.sh` currently +needs Python 3.12+ **for its bootstrap**, while native-only voice inference needs +a separately provisioned compatible CPU Python environment and pinned weights. -## Non-Steam overlay identity +## OpenVR identity -OpenVR overlay applications do not require a Steam store AppID or Steamworks. -The native executable initializes as `VRApplication_Overlay`. For discoverability -and optional autolaunch, register an OpenVR application manifest with a stable, -project-owned **string application key** (proposed: `local.frameyap.overlay`). -That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam -library shortcut should be necessary for the normal route. +`local.frameyap.overlay` is a string OpenVR application key, **not** a Steam +store AppID. The installer creates a manifest/desktop launcher user-locally but +does not register/launch the app by default. Explicit `frameyap --register +/absolute/manifest/path` uses the OpenVR registration API; registration alone +did not reveal a launcher in the first checked dashboard menu. On one Frame the +user opened the panel from the **Non-Steam** section and quit; shortcut discovery +and persistence after a normal restart are still unverified. Optional +`--autolaunch`/`--no-autolaunch` on the installer explicitly request OpenVR +registration/autolaunch choices; no SteamVR settings or sessions are changed +without that request. Unregister explicitly before uninstall. SteamVR must +already be available for registration; never start/restart it for installation. -The manifest identifies the installed executable; the installer/registration -helper should use `IVRApplications::AddApplicationManifest` and the corresponding -remove operation, not hand-edit Steam's internal JSON. Autolaunch uses the OpenVR -application setting only when explicitly requested. Validate the exact manifest, -launch behaviour, registration persistence and uninstall on native Frame before -claiming this route works end to end. Existing probes established overlay client -initialization, not manifest installation. +## Installation workflows -SteamVR/OpenVR must already be installed and usable. If registration needs a -running runtime, defer it to the first explicit launch rather than starting or -restarting SteamVR behind the user's back. +- **Binary mode** (no compiler): verify a locally supplied ARM64 archive digest, + or, after a vetted release actually exists, explicitly approve retrieval of + a pinned `v0.1.YYYYMMDDHHMM` GitHub tag and checksum. No moving `latest` tag. + `--mode binary --archive FILE --sha256 HASH --version 0.1.YYYYMMDDHHMM` + selects the local-artifact path. +- **Source mode**: requires explicit local source, SDK, SDL/OpenVR libraries and + their notices, CMake/C++20, native build dependencies and a version. It builds, + stages, packages and continues through local installation; it does **not** + provision ASR packages or bypass producer license obligations. See + [packaging](packaging.md) for all flags. +- **Model**: only an explicit `--install-model --backend redux --yes` fetches + pinned public files for the *already installed* backend. Inspect the read-only + `--print-plan --json` first; it includes model size, attribution and license + metadata from `assets/backends/redux.json`. `--expected-manifest-sha256 HASH` + binds consent to the exact installed manifest bytes and fails before model + directory creation/network if they changed. The local in-panel chooser shows + source, size, license text, attribution and manifest digest, then requires a + second **Confirm Install** click; the installed/native UI route still needs + clean-target and headset acceptance. `--without-model` permits an + archive install without bundled model files. Neither operation installs Torch, + moondream, Kestrel or an interpreter. Launch paths to an independently + authorized runtime/model can be set in `paths.conf`. +- **Noninteractive**: supply flags and `--yes` for network/model consent. + `--print-plan` is read-only; `--json` provides structured results/errors and + progress events for a model download. No prompt reads stdin in a pipe. An + empty TTY invocation offers a local menu and prints equivalent flags. -## Intended user experience +Installation is user-local under XDG data/config paths with a managed launcher, +retained rollback, SHA-256/path validation, foreign-file refusal and a lock +shared with the app. No OS package changes, udev rule, root service, Steam store +listing, unrelated application dependency, microphone recording, input injection +or automatic update daemon. Hashes detect accidental/unauthorized alteration +of a downloaded artifact but do not authenticate a compromised publisher; +release metadata needs independent trust. No installer operation silently runs +pip or launches inference. Native-only archives support overlay checks but need +an externally provisioned runtime/weights before voice typing. Selecting an +uninstalled backend does not authorize a download or supply its inference engine. -1. Run one documented command from the eventual GitHub repository/release. -2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and - explains/downloads the application, compatible CPU runtime and pinned model. -3. User-local installation provides a simple `frameyap` launcher, desktop - entry where supported, and an OpenVR manifest. No compiler, engine checkout, - Python dependency troubleshooting or separate ASR server for ordinary users. -4. The user explicitly launches/enables dictation. No installation-time microphone - recording, input injection, inference benchmark or overlay takeover. +## Gate before publishing the goal as fulfilled -Illustrative command shape only; `OWNER`, `REPO` and `VERSION` are placeholders: - -```sh -curl --fail --silent --show-error --location \ - https://raw.githubusercontent.com/OWNER/REPO/VERSION/install.sh | bash -``` - -Also document a download-inspect-run path for users who do not want to pipe remote -code into a shell. Pin a release/tag instead of executing a moving branch by default. -Offer explicit version selection and noninteractive flags; do not read interactive -confirmation from stdin while the installer itself is arriving through that pipe. - -## Packaging boundary - -- Prebuilt ARM64 executable plus a known-compatible, isolated CPU inference runtime; - no external application libraries/assets and no system Python modification. -- Model fetched during explicit installation/setup, with pinned revision/hash and - attribution. A documented `--without-model` option can defer the large download. - No surprise first-utterance downloads. Runtime components are installed from - their own package index rather than redistributed in our tarball. -- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`), - configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in - `~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory. -- No sudo, OS read-only-root changes, package-manager installs, udev changes, - `/dev/uinput` permission changes or modifications to unrelated launchers. -- No Steam store AppID, Steamworks SDK, root service or network ASR dependency. -- Third-party notices for everything we redistribute must be included with a release. - -## Installer lifecycle and safety - -- Detect architecture, libc and prerequisites first. Unsupported hosts fail with a - clear explanation; never install an x86 payload silently on ARM64. -- Download to a staging directory, check versioned SHA-256 manifests and archive - paths, then atomically select the completed version. HTTPS/checksums alone do not - authenticate a compromised publisher; use signed release metadata if provided. -- Keep configuration across upgrades; retain the previous version for rollback. - Refuse or defer replacement while this application's process is running rather - than killing arbitrary processes. Never touch SSH or unrelated sessions. -- Autostart is opt-in (`--autostart` or explicit settings); do not enable a systemd - service or SteamVR autolaunch by default. Do not enable overlay input overrides. -- Uninstall removes only owned launcher, manifest registration and install files; - model/config deletion is separately explicit. Preserve other SteamVR apps. -- No update daemon initially. A deliberate rerun/update command is sufficient. - -## Acceptance before advertising one-command installation - -- Clean supported Frame: install without sudo/compiler/engine checkout/store AppID; - launch overlay, load local Redux and type into an owned disposable target. -- Normal use after installation needs no network connection or desktop ASR host. -- Failed download/hash, unsupported architecture, low disk space and interrupted - upgrades leave a usable previous install or a cleanly reported failure. -- Reinstall, rollback and uninstall preserve unrelated data and SteamVR entries. -- Noninteractive piped invocation never hangs on stdin; inspection-first path works. -- Autolaunch remains off unless chosen; uninstall removes only our registration. -- Hardware-free installer tests use temporary homes, mocked runtime registration - and local fixture artifacts. Never exercise a real user's Steam configuration - in ordinary CI/CTest. +Vet the **exact** release closure/licenses, ARM64 symbol versions/loader, +model attribution and compatible CPU Python environment; establish a tested +libc/runtime floor, then publish and authenticate a checksummed archive from a +clean tag. On a clean supported Frame, install without a compiler/sudo/store ID, +load Redux, type into a disposable owned target and validate rollback/uninstall, +foreign-file failures, autolaunch off, and no unintended session changes. Separately +validate microphone → review → delivered input and headset comfort. Offline +installer fixture tests and a local native build are not those acceptance gates. diff --git a/docs/overlay.md b/docs/overlay.md index 0092cdf..47d1155 100644 --- a/docs/overlay.md +++ b/docs/overlay.md @@ -2,10 +2,12 @@ `src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion. -Native `--run` wiring in `src/main.cpp` is implemented behind the explicit -`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful -compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a -normal build or test. +Native `--run` wiring in `src/main.cpp` is behind the explicit +`FRAMEYAP_NATIVE` build option. The Models chooser, offline verification and +explicit installer handoff are implemented locally, **not** an accepted Frame +install/voice-typing path or a published release. Neither hardware-free tests +nor a successful compile establish Frame input, visibility, comfort or text +delivery. Launching the runtime is explicit, never part of a normal build or test. ## Rendering and controls @@ -49,7 +51,9 @@ This replaces the raw-upload rendering path; headset flicker acceptance still requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance. The header shows local time and date instead of the former on-device/review -and current-mount labels. It updates when the displayed minute or date changes, +and current-mount labels. Settings explains Hold Quit (hold 0.9 seconds then +release) and Lasers anytime (system-wide lasers may affect games). The review +tab describes Type and Type + Enter. It updates when the displayed minute or date changes, not every frame. Settings toggles 12/24-hour time and cycles date Off → MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display; mount choices remain in Settings. @@ -58,23 +62,25 @@ The complete transcript preview is paginated by glyph width and four-line height; Previous and Next navigate it without changing the source transcript. Status fits on the single status line; the old bottom detail label is gone. The footer remains available on all tabs: Record (labelled Stop while recording), -Cancel, Insert, Submit, Hold Quit. Hold Quit needs a 900 ms press and release on +Cancel, Type, Type + Enter, Hold Quit. Hold Quit needs a 900 ms press and release on that same button; its thin progress bar shows the hold. Record can retry after an error; it is disabled while warming/transcribing and until an existing review is -inserted or discarded. Cancel can stop worker startup. Insert and Submit are +typed or discarded. Cancel can stop worker startup. Type and Type + Enter are disabled during recording and transcription. A pointer action requires a press/release on the same enabled control from the same cursor; focus loss, -tab changes, action-state changes and relocation clear pending presses. Submit is *always* a separate -deliberate action, not inferred from text. Insert appends a trailing space (without -doubling an existing trailing space). Submit inserts any pending review and then -queues Enter; with no pending text it queues Enter only. Y opens the quick-chat -list over the review area; each further Y press cycles its highlighted choice. -Cancel closes the picker without discarding an existing review. Submit sends the -selected text *without* a trailing space, then Enter. The choices are short +tab changes, action-state changes and relocation clear pending presses. Type + Enter +is *always* a separate deliberate action, not inferred from text. Type normally +appends a trailing space (without doubling an existing one); a full 4096-byte +transcript without room for that suffix is queued unchanged, with no extra error +for the missing space. Type + Enter types any pending review and then queues +Enter; with no pending text it queues Enter only. Y opens the Quick phrases +list over the review area; each further Y press cycles its highlighted choice. Cancel closes the picker without discarding an +existing review. Type + Enter sends the selected phrase *without* a trailing +space, then Enter. The choices are short single-line literals, not speech commands. A failed text step never proceeds to Enter. Recording never automatically submits. Auto insert, when -explicitly enabled, can queue text + space after transcription only under the -stable Xwayland focus guard described below. +explicitly enabled, can queue text (normally + space) after transcription only +under the stable Xwayland focus guard described below. ### Bindings button @@ -107,6 +113,8 @@ installer creates one with defaults on first install. Copy the shipped "input_priority": "normal", "advanced_debug": false, "auto_insert": false, + "close_mic_when_idle": false, + "backend": "redux", "lock_layout": false, "clock_24h": false, "date_format": "mdy", @@ -149,6 +157,18 @@ config, retaining other fields and formatting; invalid/unwritable configs are left untouched and return failure. The installer backs up original bytes before repairing invalid values, while valid `true` and `false` are retained. +`close_mic_when_idle` is a separate boolean, default **false**, also available +as Settings → **Close mic when idle**. Normally the SDL capture device stays +open while Ready and idle samples are discarded. Enabling this toggle closes it +between clips and opens it on PTT: this avoids an open idle capture device but +can cause an audio spike (observed with per-PTT transitions on Frame), startup +latency or first-syllable clipping. Settings persistently displays **OFF: discard +idle audio; ON: spike / start latency** below the toggle, as well as an ON/OFF +indicator; the status/detail line also explains a change when toggled. The +native setting is saved to `config.json`; a failed save applies only for this +session and warns. This is not a mute switch for other applications. Quit and +failure still close the device. + `auto_insert` is a separate boolean, default `false`, also available as a Settings toggle. Only a **new** recording arms it. It observes the Xwayland display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and @@ -156,41 +176,77 @@ display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and Both properties and focus are rechecked after IME lease acquisition. A watched focus-out, root focus-property change (even if the same window returns), window destruction, held keyboard key, missing X display or any disagreement permanently -disarms that clip. The transcript then remains for explicit review/Insert. +disarms that clip. The transcript then remains for explicit review/Type. Native Wayland focus and child text-field focus cannot be safely inferred here; those cases fall back to review. No automatic Enter, speech commands or retry. The compositor can still change focus in the gap between the final check and global delivery, and IME commit is not an application receipt. This path has offline synthetic focus tests and a separate owned-target IME fixture; live -speech-driven Auto Insert, target coverage and headset acceptance remain +speech-driven Auto insert, target coverage and headset acceptance remain unverified. The setting is preserved on upgrade and a failed preference write applies only to the current session. `quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings, each at most 64 characters. Edit the JSON file and restart; there is no headset text editor. Inputs are literal (not expanded or interpreted by FrameYap) and -are sent to the current Gamescope focus, so check the destination before Submit. +are sent to the current Gamescope focus, so check the destination before Type + Enter. `buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`, `insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME` button paths. Omitted actions retain their bundled defaults; an empty string disables a mapping, including after an upgrade. The Frame defaults are right -X = hold-to-talk, B = Cancel, A = Insert + space, Y = quick chat. Enter has no -single-button mapping by default; the left grip double-tap still submits. -An existing config mapping `enter` to right Y is migrated to quick chat in memory +X = hold-to-talk, B = Cancel, A = Type + space, Y = Quick phrases. Enter has no +single-button mapping by default; the left grip double-tap still requests Type + Enter. +An existing config mapping `enter` to right Y is migrated to Quick phrases in memory when `quick_chat` is absent; this does not overwrite custom mappings. Existing configs with empty actions retain those disabled mappings; change them explicitly or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized; SteamVR user overrides may still supersede it. On customized launches a generated action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings` (or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The -config is read once at launch, not hot-reloaded. On install/upgrade the installer -fills missing fields, removes retired keys, and resets invalid entries. It saves -the exact prior bytes under `config.json.backup-*` before a repair and refuses -symlink/oversized config paths; valid customizations remain intact. The installed -launcher no longer pins `--font`, so this selection takes effect. Direct native +config is read once at launch, not hot-reloaded (the in-panel backend selection +is saved separately). On install/upgrade the installer fills known missing fields, +including `close_mic_when_idle` and `backend`, removes retired keys and resets +invalid entries. It saves the exact prior bytes under `config.json.backup-*` +before a repair and refuses symlink/oversized config paths; valid customizations +remain intact. The installed launcher no longer pins `--font`, so this selection takes effect. Direct native launches with bad JSON, colors or button mappings fail startup rather than silently changing input behavior. +### Models / backends (local implementation) + +Settings → **Models / backends** opens a paginated local chooser. The saved +`"backend": "redux"` selects a *manifest ID*, not a model download; +`--backend ID` overrides it for that `--run` invocation. Selecting a listed backend +restarts the owned worker and closes capture, discarding pending audio, review +and any prior focus/delivery authorization. A failed preference write leaves the +choice active only for this session. The currently listed Redux model is the +only inference implementation supplied; adding a manifest alone does not add +an inference runtime. The Models tab labels local checks as checking, missing +(`not_installed`), invalid, or installed/verified; for the selected model it +also reports loading, ready or failed. An install-in-progress note reports +model provisioning, followed by a new offline check; a verified status alone +is not proof the CPU runtime loaded. Recording is disabled until the selected +files verify offline and the worker warms; missing files never trigger a silent +download. Model loading and the default open-while-Ready idle microphone policy +are unchanged. The panel's state and successful helper calls do **not** prove +microphone transcription, input delivery or headset acceptance. + +For a missing/invalid selected model, **Install** first presents a separate +confirmation showing pinned source, approximate download size, license text, +attribution and the exact raw manifest SHA-256 fingerprint. Only **Confirm +Install** launches the local installer with that fingerprint; leaving the view +or changing metadata invalidates consent. The installer rechecks the *installed* +manifest bytes under its model lock before creating a download target or using +the network, downloads only on the explicit click, hashes pinned files and then +the app checks them again offline before enabling recording. Existing invalid +or unsafe model files are refused rather than silently overwritten. This +installs model files, **not** Python, Torch, moondream, Kestrel or other runtime +dependencies. +The source-tree UI needs an installed release for installer-backed provisioning; +a hand-edited manifest is not an approved artifact. The installer handoff and +native UI have not yet been accepted on a clean Frame. See +[packaging](packaging.md#consumer) for an inspection-first CLI path. + ### Experimental controller input priority There are two independent gates: @@ -348,7 +404,7 @@ recentring and readability still require a separately authorized headset check. The opt-in native `--check-controls` probe logs pointer counters and action callbacks to the terminal rather than repainting them on the panel. Its canvas -stays static for Record/Cancel/Insert/Enter clicks so those clicks can be checked +stays static for Record/Cancel/Type/Type + Enter clicks so those clicks can be checked without diagnostic texture uploads. Switching tabs or mount still updates the visible panel. Diagnostics identify `renderer=Vulkan` and count `textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions. @@ -367,8 +423,10 @@ cmake --build build-native --target frameyap_texture_check It does not initialize OpenVR or establish compositor/headset acceptance. -`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert, -Enter. `bindings_frame_controller.json` maps right X click to hold-to-talk PTT; +`assets/actions.json` names seven actions: left/right grip, PTT, cancel, +Type, Type + Enter and Quick phrases. `insert`, `enter`, and `quick_chat` remain +internal binding keys; visible controls read Type, Type + Enter, Quick phrases. +`bindings_frame_controller.json` maps right X click to hold-to-talk PTT; the grip bindings remain for optional remapping/diagnosis. In one dashboard probe grips were inactive; a later controls-only probe delivered repeated right X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions @@ -384,7 +442,7 @@ second squeeze **down** within 350 ms starts capture; hold as long as needed action explicitly begins on down and ends on up. On tracking-pose invalidity, action inactivity or overlay focus loss, a held capture emits Cancel, and reconnection requires a neutral observation before any new press. PTT and left -Enter require an enabled panel; clickable Record remains available for retry +Type + Enter require an enabled panel; clickable Record remains available for retry after an error and Cancel is always available. The action set defaults to normal priority; the experimental config request is described above. Neither priority guarantees delivery while a game or dashboard owns input. diff --git a/docs/packaging.md b/docs/packaging.md index 896c658..e11c355 100644 --- a/docs/packaging.md +++ b/docs/packaging.md @@ -11,53 +11,71 @@ build with `FRAMEYAP_NATIVE=ON`, and stage this layout (regular files, no links) ``` bin/frameyap -lib/* # compatible bundled native libraries -assets/actions.json # and adjacent controller binding JSON +bin/install.sh # managed installer helper when required by layout +lib/* # explicitly supplied compatible native libraries +assets/actions.json # plus binding JSON and backends/redux.json fonts/font.ttf python/frameyap/*.py +scripts/model-status.py # offline CLI verifier; also backend-service.py, fetch-model.py licenses/THIRD_PARTY_NOTICES.txt model/* # optional pinned public weights + attribution runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact ``` -For the current **external-runtime** package, `scripts/stage-native-poc.py --help` -documents explicit inputs. It invokes `cmake --install` on an existing native build, -copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does +The stage copies the self-contained installer into `bin/` and CMake installs +`scripts/model-status.py` alongside the backend manifests. The package allowlist +permits `scripts/`; installed `frameyap --list-models` expects the verifier at +`../scripts/model-status.py`. **No actual staged archive has been audited/tested +from a clean account for publication**; exercise the entire producer pipeline +and the installed CLI before treating the payload layout as release-ready. + +For the current **external-runtime** package, `scripts/stage-native.py --help` +documents explicit inputs. The old `scripts/stage-native-poc.py` is retained as +a deprecated migration wrapper for that command, not the documented or shipped +staging interface. The stage invokes `cmake --install` on an existing native +build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does not build, download, run the app, or copy an ASR runtime. The native app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType, -libstdc++ and glibc; audit `ldd` on the installed binary. +libstdc++ and glibc. Audit the actual staged ARM64 binaries' `NEEDED`, +`GLIBC_*`/`GLIBCXX_*` symbol versions, ELF interpreter and notices; then test +on a clean target. No compatible libc floor is yet established. SDL/OpenVR resolve inside its own `lib/`, not a producer prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux. ```sh python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \ - --version 2026-09-24T162712Z-g417f81c --arch linux-aarch64 \ + --version 0.1.202609241627 --arch linux-aarch64 \ --model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime ``` -Use the actual binary's UTC build stamp (`frameyap --version`) for the archive -tag, not this illustrative timestamp. CMake generates `YYYY-MM-DDTHHMMSSZ` -at configuration time (plus `-gSHORTSHA` for a Git checkout and `-dirty` for -uncommitted tracked changes); producers may pin `-DFRAMEYAP_VERSION=...` to -embed a vetted release stamp. A timestamp distinguishes same-day archives; -the installer still refuses a reused tag whose contents have changed. Historic -`v0.1.0-poc*` local artifacts remain valid for reinstall/rollback. +Use the actual binary's numeric `MAJOR.MINOR.YYYYMMDDHHMM` UTC version +(currently `0.1`), not this illustrative value, for `--version` and filename; +tag a vetted clean tree as `vVERSION`. CMake generates this stamp at +configuration time (`SOURCE_DATE_EPOCH` may supply it); +`-DFRAMEYAP_VERSION=0.1.YYYYMMDDHHMM` can pin it. Development `--version` +may print a *separate* `git HASH` line, with `(uncommitted changes)` only if +dirty: that line is not part of the version, tag or archive name. The installer +rejects a reused tag with different contents; historic local versions may +remain selectable for rollback, not as new releases. `--external-runtime` refuses a runtime directory and records `runtime: external-authorized-python` in `release.json`. The installer explicitly reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required in the archive. Staging validation is not an inference test. -The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` -plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal, +The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing +`HASH FILENAME`. Archive extraction rejects traversal, links/special files, duplicate members, oversized metadata/payloads and invalid layout. Checksums detect corruption, not a malicious/compromised publisher; authenticate release metadata independently. No packaging/installation model fetch. ## Consumer -Bootstrap: Linux ARM64/glibc, Python 3.12+, curl, sha256sum and tar. **No compiler, -sudo, Steam store AppID or engine checkout.** Download/inspect a pinned installer +Bootstrap for a binary archive: Linux ARM64/glibc, Python 3.12+ (installer +bootstrap, **not** the inference runtime), curl for network release downloads, +sha256sum and tar for archive handling. **No compiler, sudo, Steam store AppID +or engine checkout in binary mode.** No minimum glibc floor has been certified; +preflight alone cannot guarantee compatibility. Download/inspect a pinned installer before running it. Current local artifact route: ```sh @@ -65,17 +83,62 @@ sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \ --sha256 64_HEX_DIGIT_HASH --version VERSION ``` -After an actual vetted release exists, `sh install.sh --version TAG` retrieves -that GitHub release and its versioned checksum; no `latest` or moving-branch +After an actual vetted release exists, use a real numeric version (for example, +`sh install.sh --mode binary --version 0.1.202609241627 --yes`); the installer +will retrieve tag `v0.1.202609241627` and its versioned checksum; no `latest` or moving-branch lookup. A pipe invocation is supported, never prompts on stdin, and must also pin a real published tag. **There is no functional public download command yet.** -`--without-model` omits bundled model files from staging, never deletes a current -model on rerun, and records the choice. Same digest/version/choice is idempotent -and repairs missing managed wrappers. Different digest or model choice for the -same version is refused. External model provisioning is always deliberate. +`--without-model` omits any model files in the selected archive, never deletes +an existing current model on rerun, and records the choice. Same +digest/version/choice is idempotent and repairs missing managed wrappers. Different digest or model choice for the +same version is refused. External model provisioning is always deliberate: the +local installer accepts `sh install.sh --install-model --backend redux --yes` +(optional `--model-dir /absolute/path`) to explicitly download and verify files from the +installed pinned manifest, not an ASR runtime. Inspect the model/size first +with `sh install.sh --install-model --backend redux --print-plan --json`. +`--expected-manifest-sha256 HASH` additionally binds consent to the exact raw +installed `redux.json` bytes: under the model lock a mismatch fails **before** +a model directory is created or any network request. In the locally implemented +Models UI, Install displays source, rounded size, license text, attribution and +the fingerprint; only the second Confirm Install click passes that fingerprint +through the backend helper to the installer. Installation alone does not +provision a CPU Python runtime; no in-panel flow has been accepted on Frame. +In the source tree, `python3 scripts/model-status.py --list-models` or +`--check-model redux --model-dir /absolute/model` hash-checks local files; +the native `frameyap --list-models` / `--check-model` entry points use the +adjacent installed verifier script; verify this in a staged archive before release. Manifest schema, +source, size/hash and attribution live in `assets/backends/redux.json` and are +validated by `python/frameyap/model_files.py`. -The installed launcher defaults to `--run`. For a native-only package, supply +For an explicit local **source** install, e.g.: + +```sh +sh install.sh --mode source --source /absolute/source --openvr-root /absolute/sdk \ + --openvr-library /absolute/libopenvr_api.so --openvr-license /absolute/openvr/LICENSE \ + --sdl-library /absolute/libSDL3.so.0 --sdl-license /absolute/sdl/LICENSE \ + --version 0.1.202609241627 +``` + +It checks tools and native dependencies, builds/stages/packages in a private +workspace, then installs the result. Unlike binary mode, this requires a C++ +compiler, CMake, SDK, SDL3, Wayland/scanner, libxcb, FreeType, Vulkan development +files and producer-supplied +licenses; it still does not install Python ASR packages. `--print-plan` performs +a read-only plan, `--json` gives machine-readable results/errors (model installs +also stream file events), and `--yes` authorizes network downloads. Bare TTY +invocation can guide choices and prints equivalent flags; non-TTY runs require +explicit arguments and never prompt. `--autolaunch`/`--no-autolaunch` are explicit +OpenVR registration choices, off by default; do not pass either during an inert +install if a running SteamVR session must remain untouched. + +The installed launcher defaults to `--run` and forwards explicit run flags +(including `--backend ID`, `--model-store /absolute/store` and +`--manifest-dir /absolute/manifests`) to the binary. These overrides select +local metadata/model paths, not a runtime download; the latter two require +absolute paths without dot segments. `--backend` selects for that run unless +changed in the panel; a saved `config.json` backend is otherwise used. For a +native-only package, supply `FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`, or override `--python`/`--model` on an explicit `--run`. For menu launches, create `$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`): @@ -157,7 +220,7 @@ without configured model/runtime cannot transcribe; it should show **Unavailable (possibly after a brief Warming transition), rather than record or infer. Only perform this check on an unconfigured native-only installation: verify that `current/runtime/bin/python3` and `current/model` are absent and no -user-local paths are configured; do not press Record, Insert or Enter. +user-local paths are configured; do not press Record, Type or Type + Enter. The installer also provides a desktop entry; where Steam's UI supports adding a non-Steam app, the user may select that entry or browse to the installed launcher. Shortcut discovery/persistence after a normal restart is not yet verified. Do diff --git a/docs/third-party.md b/docs/third-party.md index db95b37..cc0c1f3 100644 --- a/docs/third-party.md +++ b/docs/third-party.md @@ -1,91 +1,99 @@ # Dependency provenance and release boundary -FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project -owner. This does not relicense external models, fonts, protocols or runtimes. -There is no dependency on another application's checkout, assets or environment. +FrameYap's original code is [MIT licensed](../LICENSE). This does not relicense +models, fonts, protocols, native libraries or Python wheels. There is no dependency +on another application's checkout, assets or environment. **This is an upstream +license inventory, not an audit of a particular release binary. No public release +has been published.** Before shipping any archive, inspect the actual staged +files, their transitive dependencies and notices, and test on a clean supported +host. An external dependency does not make the current archive self-contained. -## Included source +## Included source/assets -`protocol/gamescope-input-method.xml` is the unmodified public Gamescope -**3.16.28** protocol, downloaded from -. -SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`. -Its embedded permissive copyright/license notice is preserved. Generated bindings -are build outputs, not hand-written wire encoding. The private protocol may change -with SteamOS; compatibility must be rechecked. +- `protocol/gamescope-input-method.xml`: unmodified public Gamescope **3.16.28** + protocol, . + SHA-256 `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`; + embedded permissive copyright/license notice preserved. Generated bindings are + build outputs; this private Gamescope extension needs rechecking after updates. +- `assets/fonts/Inconsolata-Regular.ttf`: unchanged Inconsolata Regular from + , copyright 2006 The Inconsolata + Project Authors, **SIL OFL 1.1**. SHA-256 + `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`. + `assets/fonts/OFL-Inconsolata.txt` retains the license. The font remains OFL, + not MIT, and is not sold by itself. An override font requires its own license. +- Frame controller bindings and the mint-to-blue CPU-rasterized panel were + authored here, using public profile names; no SteamVR driver artwork, MSDF + atlas, other app renderer or protected kernels were copied. -Frame controller bindings were authored here using the observed public input -profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code, -images, protected kernels or another application's assets were extracted for these bindings. +Native staging uses `scripts/stage-native.py`: it includes the chosen font/license +and copied SDL3 and OpenVR notices in `licenses/THIRD_PARTY_NOTICES.txt`. +It does **not** bundle ASR. A4 is not closed for release: inspect the final +notice file, especially FreeType attribution, plus the selected native binary +and any bundled wheel/licenses before publishing. -## Bundled font and UI reference +## Native build/runtime inventory -`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular. -SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`. -Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**, -retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace -normalized; license text unchanged). The reviewed OFL permits -bundling and redistribution with its copyright/license notice; the font remains -OFL, not MIT, and is not sold by itself. Upstream: . -The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies -were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph. +| Component | Upstream license / evidence | Current packaging boundary / action | +| --- | --- | --- | +| Valve OpenVR SDK 2.15.6 | BSD-3-Clause-style license, SDK LICENSE. | Staging copies its loader and explicitly supplied license; verify chosen binary and transitive closure. | +| SDL3 (device trial 3.2.16) | zlib, local `/usr/share/licenses/sdl3/LICENSE`. | Staging copies explicit library and license. Confirm exact build options/version and its transitive libraries. | +| Wayland client + scanner | MIT/Expat-style, local `/usr/share/licenses/wayland/COPYING`. | Client is linked from system; scanner is build-time. If shipped, include copyright/license and audit closure. | +| libxcb | MIT-style with name-use restriction, local `/usr/share/licenses/libxcb/COPYING`. | Xwayland focus guard uses client library at runtime; not bundled by current native stage. Audit exact binary. | +| FreeType 2 | **FreeType Project License (FTL) selected** for this project, local `/usr/share/licenses/freetype2/FTL.TXT`; upstream also offers a GPL option. | Current stage uses system library, not bundled. Credit FreeType Team for use; if distributing its binary, meet FTL binary disclaimer/notice obligations and review the precise build. Do not silently substitute GPL terms. | +| Vulkan loader, driver, system graphics dependencies | Loader/driver licenses vary by build and vendor. | Current stage depends on system Vulkan loader/driver; no GPU runtime is bundled. Audit the chosen loader if ever bundled. | +| Compiler runtime (`libstdc++`, `libgcc_s` when used) | GCC libraries: GPL with **GCC Runtime Library Exception** in upstream distribution; local `/usr/share/licenses/libstdc++/RUNTIME.LIBRARY.EXCEPTION` and `libgcc/...` are exception texts, not a full installed release audit. | Current stage relies on system runtime. Audit dynamic linkage, C++ ABI/`GLIBCXX_*` and exception coverage for *any* bundled compiler libraries; include corresponding complete notices/source obligations as applicable. | +| glibc/loader | GNU LGPL-2.1-or-later for core GNU C Library, with component-specific exceptions and other licenses to inspect. | Current stage relies on system libc/loader. Audit exact target binary symbol versions (`GLIBC_*`), ELF interpreter and its transitive closure; no minimum glibc/`GLIBCXX`/kernel floor is certified here. | -The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted -selection) is implemented independently on a single CPU RGBA surface. -CMake installs the font and OFL with assets; native staging defaults to that -font, places the launcher copy at `fonts/font.ttf`, and includes its license in -`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license. - -## Explicit native build inputs (not vendored) - -- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015; - retain its LICENSE with redistributed loader binaries. -- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix. -- Wayland client and scanner: retain upstream MIT-style notices. -- libxcb (X11 protocol client): MIT-style license; used only by the opt-in - Xwayland focus observer. Include it in native runtime dependency checks; - no X server is started by normal operation. -- FreeType: choose and comply with its applicable FTL/GPL licensing option. -- Optional font override: the earlier device check used system Hack Regular. - A custom release font must include its own license and assessed glyph coverage; - new builds default to the bundled Inconsolata described above. -- Compiler runtime, libc minimum and transitive shared libraries require a release - dependency audit. Passing a developer build is not a portable-runtime guarantee. +The historical Frame CPU trial used Torch **2.8.0+cpu** on a host with **glibc +2.39**. Those are *observed trial versions*, **not** minimum compatible versions +for FrameYap, Python wheels or a future released artifact. `ldd` on a developer +machine alone is not sufficient: inspect the staged ARM64 binaries with `readelf` +(`NEEDED`, ELF interpreter, symbol-version requirements), `ldd` on a trusted +clean target, actual bundled libraries and notices, and test the final archive on +a clean supported Frame. Check the chosen compiler, CPU instruction/kernel, +FreeType/Wayland/XCB/SDL/OpenVR/Vulkan ABI, Python/native wheels and licenses. +Do not invent a libc floor from a build host's version. ## Redux weights and inference runtime -Public model: , exact revision -`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution: -Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3. -No modifications to the supplied weights are made. Exact model/config/tokenizer -sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`. -`fetch-model.py` fetches and retains the original model card alongside the files. -No weights are committed to this repository. +Model: , exact revision +`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. +Attribution: Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT +0.6B v3. No modifications to the supplied weights are made. Pinned model/config/ +tokenizer/card sizes and SHA-256 hashes, source and attribution are recorded in +`assets/backends/redux.json` (schema validated by `python/frameyap/model_files.py`), +not hardcoded in `model_files.py`. `fetch-model.py` fetches and retains the model +card alongside the weights on **explicit** request; neither build/tests nor a +normal app launch downloads them. `scripts/model-status.py` and the native +`--list-models` / `--check-model` interface inspect/hashes local files offline. +No weights are committed to Git. -Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels` -dependencies, installed by the user from PyPI into their own environment. Upstream's -Kestrel README states: "Local inference is free and requires no API key" -(); finetuned-model inference needs a Moondream -API key and is not used here. FrameYap's release archives do not vendor or bundle -these packages; they are installed from PyPI onto the user's machine, either by the -user or by the installer at the user's request. +The current Redux worker uses separately provisioned `moondream` Python and its +`kestrel` / `kestrel-kernels` dependencies, not a bundled runtime. Kestrel's +upstream README says “Local inference is free and requires no API key” +(); finetuned-model inference needs an API +key and is **not** this path. The native-only installer **does not run pip**, +provision an interpreter, or make a native-only artifact able to transcribe on +its own. A person supplying a Python environment must review/authorize its +exact dependency closure. No bundled-ASR artifact is licensed/approved by this +inventory. -Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need -their own notice/license inventory before a bundled release. +| Python/native package | Upstream license inventory (not a wheel audit) | Release action | +| --- | --- | --- | +| PyTorch / Torch CPU | PyTorch project: BSD-3-Clause; third-party components/wheels carry additional notices and dependencies. | No Torch wheels bundled. Pin CPU-only ARM64 wheel if building a distribution; audit its `LICENSE`, `NOTICE`, `third_party`/wheel contents and `NEEDED`/symbol versions. | +| NumPy | NumPy core: BSD-3-Clause; dependencies/embedded algorithms carry additional BSD, MIT, 0BSD, zlib, CC0 and other notices depending on wheel. Local `python-numpy` 2.5.3 package metadata (`/usr/lib/python3.14/site-packages/numpy-2.5.3.dist-info/METADATA`) declares `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` with many `License-File` entries (different from the historical trial environment). | Not bundled. Keep *all* license files and inspect the exact target wheel, BLAS/OpenBLAS and runtime closure before redistribution. | +| Hugging Face `tokenizers` | Upstream `huggingface/tokenizers` is Apache-2.0; native/Rust crate dependencies need separate inventory. | Not bundled. Confirm actual installed wheel version, package LICENSE/NOTICE and transitive Rust/native code if ever distributed. | +| `moondream`, Kestrel/kernels/native, Python interpreter | Distinct packages with distinct license files and native transitive code; Kestrel local-use statement is not a blanket redistribution license. | None bundled. Exact versions, permissions, wheel notices, CPython build and native linkage require review before any runtime bundle. | -## CPU trial dependency choice +Development CPU trial *interfaces*, not package-floor promises: moondream +**2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, native **0.1.8**, Python +**3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified moondream install +initially resolved CUDA-enabled Torch and NVIDIA wheels; the owned trial venv +was corrected before measurement (`torch.version.cuda is None`). Do **not** +repeat unconstrained `pip install moondream` as a CPU setup recipe. -Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, -native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified -moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels; -these were replaced/removed from the owned venv before measurement. The measured -Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained -`pip install moondream` as a CPU setup recipe. - -For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution -was downloaded but not bundled with any ASR runtime: -, +A standalone CPython 3.12.14 ARM64 distribution was downloaded for packaging +research but **not bundled**: , `cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`, SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`. -Its included licenses and compatible native dependencies still require review -before any release. No public prebuilt FrameYap release has been published. +Its own licenses and native compatibility need review before any release. diff --git a/docs/worker.md b/docs/worker.md index f3898d0..b17451d 100644 --- a/docs/worker.md +++ b/docs/worker.md @@ -1,8 +1,10 @@ -# Offline Redux worker adapter (component, not an installed product) +# Offline worker adapter and backend dispatcher `src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model, -threads=2, advanced_debug=false)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for -one `WorkerReply` (text or privacy-safe per-request error). One request at a time; +threads=2, advanced_debug=false)` explicitly for the legacy Redux script, or +supply the optional `backend`, `manifest_dir`, `root` arguments for the manifest +dispatcher. Poll until `ready()`, then `submit(id, pcm)` and poll for one +`WorkerReply` (text or privacy-safe per-request error). One request at a time; no queue, no capture and no input injection. `stop()` discards pending audio, terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL), and is safe to repeat. Destruction stops it. `start()` returns without waiting @@ -36,13 +38,19 @@ requests reading the fixed clip; `Y` means ready; `F` means load failure missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8 text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096 bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe -or oversized frame stops the worker. Warmup deadline is 120 s, transcription -deadline 60 s; `poll()` must be called regularly to enforce deadlines. It -never initializes a headset or starts a recording. There is no auto restart. +or oversized frame stops the worker. A correlated `E` reply or an invalid +transcript is a **request-level** failure: the Controller drops that clip, +keeps the ready model loaded and allows Record to retry. A broken protocol, +worker crash or explicit in-flight cancellation is different and can require a +reload. Warmup deadline is 120 s, transcription deadline 60 s; `poll()` must +be called regularly to enforce deadlines. It never initializes a headset or +starts a recording. There is no auto restart after a process failure. -`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI -startup, with HF/Transformers/Datasets offline variables and bounded native -thread-pool variables set before import. It uses +Parakeet Redux is the **model** (`moondream/parakeet-redux`); `moondream` is +its Python inference package, not a second model or cloud endpoint. Kestrel is +a native dependency of that local runtime. `python/frameyap/worker.py` lazily +imports `moondream` only after explicit CLI startup, with HF/Transformers/Datasets +offline variables and bounded native thread-pool variables set before import. It uses `md.photon("moondream/parakeet-redux", model_path=, device="cpu", cpu_threads=threads)` and persistent `transcribe(audio=, sample_rate=16000)["text"]`. @@ -51,16 +59,31 @@ moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide preinstalled local weights from revision `fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer -against `model_files.py` before importing model libraries. Protocol stdout is -isolated at the file-descriptor level from third-party diagnostics. Thread limits -cover Torch interop/native pools and CUDA is not selected. Offline environment +against `assets/backends/redux.json` via the shared, offline +`python/frameyap/model_files.py` schema/verifier before importing model libraries. +`frameyap --list-models` and `frameyap --check-model redux --model-dir /absolute/model` +(or `scripts/model-status.py`) read local manifests and optionally hash local +files; they never load the worker or download weights. For native `--run`, +`--backend ID`, `--model-store /absolute/store`, and +`--manifest-dir /absolute/manifests` are wired through the installed launcher +as explicit overrides; default selection is saved in `config.json` (`redux`). The dispatcher +`python/frameyap/backend_worker.py` validates a selected manifest and hashes +its local model, resolves an in-release relative Python/executable launcher +without a shell, then supervises one child over `frameyap-worker-v1` (ready, +correlated request/reply and failure frames). The launcher must accept +`{model_dir}` and `{clip_dir}` (optional `{threads}`); matching manifest and +pinned weights are **not** a runtime or automatically trusted new backend. +Only Redux is supplied today. Protocol stdout is isolated at the file-descriptor +level from third-party diagnostics. Thread limits cover Torch interop/native +pools and CUDA is not selected. Offline environment flags do not prove every third-party internal is unable to access a network. Runtime/build/tests perform no downloads; the separate explicit setup utility `scripts/fetch-model.py` can provision the public pinned weights. -Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public -runtime bundle has been released. Limited ARM64 measurements were taken during development; -they are not a claim of complete headset acceptance. +Current native-only archives do not bundle or pip-install the runtime; see +[third-party notes](third-party.md). No public runtime bundle has been released. +Limited ARM64 measurements were taken during development; they are not a claim +of complete headset acceptance. ## Advanced debugging