docs: align release scope and track remaining device acceptance gates

This commit is contained in:
baketnk committed 2026-09-24 23:03:50 -04:00
1 parent 4498ef5d04
commit 1321ca8316
9 files changed
+622 -527

No files matched your search

+31 -122
View File
@@ -1,138 +1,47 @@
# FrameYap # FrameYap
Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress). Voice typing on Steam Frame, with recognition on the headset rather than a desktop or cloud server.
Hold a controller button to record, review the transcript, then deliberately type it into the focused app.
Standalone MIT-licensed OpenVR overlay; no Steam store AppID or sudo. First v0.1 release is in progress.
Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture, ## Requirements
persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope,
and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR
server, cloud fallback or unrelated application dependency.
**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility, - Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format. Binary compatibility must be checked against each actual release artifact, not inferred from a developer build.
Gamescope discovery and native-only installation have been exercised on Frame. - For voice recognition, separately provision a compatible **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0 and dependencies) and the pinned local Parakeet Redux model. Neither is bundled or installed with pip by the current native-only installer. There is no fallback ASR service.
Live microphone → reviewed text → real target delivery is **not yet accepted**. - A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](docs/build.md).
**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the ## Install (local artifacts only)
`moondream` Python package (its Kestrel runtime states that local inference is free
and needs no API key). The build and tests never download it; the installer or you
install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md).
No GitHub release is published yet.
## Controls **No public release is published.** To install a locally vetted, checksummed ARM64 native-only archive without a compiler:
- **Right X (default Frame binding):** hold to record; release to
transcribe. Repeated presses reached the controls-only diagnostic on Frame;
live mic capture through this shortcut still needs guided acceptance. This
PTT action is remappable through SteamVR bindings.
- **Right B:** cancel/discard (or close quick chat). **Right A:** insert reviewed text with a trailing space.
**Right Y:** open quick chat; press again to cycle its highlighted choice.
**Submit** on the overlay (or double-tap left grip when active) sends that choice
verbatim then Enter; outside quick chat it inserts pending review + Enter, or
sends Enter alone when there is no text. Edit the `quick_inputs` list in
`$XDG_CONFIG_HOME/frameyap/config.json` (restart to apply). Nothing submits
automatically. Auto Insert is opt-in and off by default.
- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Submit and Hold Quit
(hold the button for 0.9 seconds before releasing).
Review and settings share one Inconsolata/neon-framed surface. The header
shows local time/date; Settings selects 12/24-hour time and date format/off.
Hold the thin bar below the panel to freely move and rotate it with your
controller; release to leave it at that pose. Drag the external lower-right
bracket to scale it (world, head or wrist), keeping the upper-left anchored.
Both handles use the app's gradient in transparent margins, like Steam's
window handles. Settings → **Lock grab/scale** hides and disables both handles;
the lock is saved. Position/size changes last for this run only.
Wrist mounting fades the panel as its full orientation turns away from an
upright viewer-facing pose (60°–75°), hiding interaction past that angle;
world and head mounting do not fade.
**Bindings** requests SteamVR's binding editor directly. Existing SteamVR overrides may supersede defaults. World-space by
default; settings offer left wrist, right wrist and head mounting, plus recenter.
Dashboard lasers provide clickable controls. Settings → Lasers anytime is an
opt-in, default-off system-wide laser mode while the panel is visible; it may
affect games and is separate from experimental input overrides.
- **Theme and controls:** optional `$XDG_CONFIG_HOME/frameyap/config.json` selects
panel colors, a font path and Frame controller button mappings; missing fonts
fall back to bundled Inconsolata. The installer creates/checks this file and
backs it up before repairs. See [overlay configuration](docs/overlay.md#user-theme-and-controller-configuration).
- **Advanced debugging:** Settings toggle / `"advanced_debug": true` in config.
Off by default. Restarts the worker and discards current work; full exceptions,
worker output and transcripts go to private, bounded local logs. No raw audio
archive. See [diagnostics](docs/worker.md#advanced-debugging).
- **Experimental input priority:** set `"input_priority": "experimental"` in
that config and enable SteamVR's Developer option **Enable global input from
overlays**. FrameYap then requests priority for its bound controller sources.
This may consume controls used by games or the dashboard; coexistence on Frame
is under test. The default is `"normal"`; restart FrameYap after changing it.
- **Review by default:** focus your destination, then press Insert (text + space) or
explicitly Enter (text + space, then Enter). Settings → Auto insert is off by default:
when enabled, it queues text + space only if Xwayland keyboard focus, active
window and Gamescope focus match continuously from recording through delivery.
Any uncertainty leaves a preview for manual Insert; it never sends Enter.
This is not yet validated for live transcription on Frame. Maximum clip 20 seconds;
accidental taps under 200 ms are discarded.
- While the native app is Ready, it keeps the mic device open and discards idle
audio instead of opening/closing on every PTT. Quit releases the device. Other
apps may still hear/transmit your voice; FrameYap does not mute them.
Physical gesture timing, global bindings during games, mic capture, target-app
compatibility and headset comfort still need coordinated validation. Successful
API initialization is not delivered input or human acceptance.
## Build and test offline
CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests.
```sh ```sh
cmake -S . -B build sh install.sh --mode binary --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
cmake --build build
ctest --test-dir build --output-on-failure
./build/frameyap --help
```
Default build has no hardware backends. It never downloads packages/models or
initializes SteamVR, a microphone or input injection. Native dependencies and
explicit launch/check commands are documented in [the build guide](docs/build.md).
`--version` uses an ISO-like UTC build timestamp (with Git hash when available),
not a numbered release; use that same tag when packaging the binary.
## Installation
The real installer accepts a versioned, checksummed prebuilt ARM64 archive:
```sh
sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
--sha256 ARCHIVE_SHA256 --version VERSION --sha256 ARCHIVE_SHA256 --version VERSION
``` ```
This is the **local-artifact command shape**, not an available public download. `VERSION` is numeric `0.1.YYYYMMDDHHMM` for this release line; a future release tag will be `vVERSION`. The installer has an explicit local source-build path with toolchain/dependency inputs, and a read-only `--print-plan`/machine-readable `--json` mode; the producer stage→archive path still needs a clean-account artifact test and release audit. See [packaging](docs/packaging.md). Installation is user-local with rollback, no automatic launch or registration, and no Steam store AppID. Before voice typing, independently supply a licensed Python CPU environment and pinned weights, and configure their absolute paths in `~/.config/frameyap/paths.conf` (or the XDG config equivalent). Model download is opt-in and separate: `sh install.sh --install-model --backend redux --print-plan --json` inspects pinned metadata, while `--yes` explicitly authorizes installation from the *installed* manifest; `--expected-manifest-sha256 HASH` additionally binds consent to its exact bytes. The locally implemented Settings → Models chooser offers selection/restart and a two-click Install/Confirm Install with source, size, license, attribution and manifest SHA-256; it has not been validated as an installed/headset flow. No chooser action installs the Python runtime. Do not mistake a native-only install for working ASR. The pinned GitHub download route must not be advertised as functional until a vetted release exists.
Installation is user-local, retains rollback, refuses active-app upgrades and
foreign files, and does not launch or register automatically. Registration uses
OpenVR application key `local.frameyap.overlay`, not a Steam store AppID.
Autolaunch is opt-in. An installed desktop entry can be selected manually as a
non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the
panel on one Frame; registration alone did not show an entry in the first checked
dashboard menu. See [packaging and lifecycle](docs/packaging.md).
For a native-only install, menu-driven inference needs a Python runtime you
provide and the pinned model. Configure their absolute paths in
`~/.config/frameyap/paths.conf` as described in the packaging guide; they are
never fetched or bundled implicitly.
A pinned GitHub one-command route is implemented in `install.sh --version TAG`, ## Controls (default Frame binding)
but **do not advertise or run it as a working public installation until a vetted
release exists**. Native-only artifacts support the overlay/checks without an
end-user compiler; a bundled-ASR distribution is not yet offered.
## Project map | Button / control | Action |
| --- | --- |
| Right X, hold / release | Record while held; release to transcribe. |
| Right B | Cancel/discard, or close the Quick phrases picker. |
| Right A | **Type:** queue reviewed text, normally with a trailing space. |
| Right Y | Open **Quick phrases**; press again to cycle the selection. |
| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. |
| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. |
| Overlay Type / Type + Enter | The same deliberate text / Enter actions. |
| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). |
- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests. Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Check the focused destination before Type or Type + Enter. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](docs/overlay.md).
- [Design](docs/design.md): full target design; some features remain proposed.
- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md).
- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md),
[dependency/license inventory](docs/third-party.md).
- [TODO](TODO.md): release plan and open work items.
Dated device-test records are kept locally (untracked) and are not authority for ## Status / not yet validated
current device availability. Live microphone → reviewed text → target delivery has
not been formally accepted; see Status above.
No recordings, transcripts, private logs, model weights or runtime binaries are Native ARM64 build, CPU inference on a public clip, overlay visibility, Gamescope API discovery, a controls-only Right X probe and native-only installation have been exercised on Frame. **Live microphone → reviewed transcript → real target delivery has not been formally accepted.** Focus guard and insertion have offline/owned-target checks, not general app compatibility or human headset acceptance. Global bindings, physical gesture feel, Auto insert with speech, latency, battery/thermal cost and controller coexistence still require opt-in headset testing. Front-prefix loss on repeated submissions (P1) remains under separate investigation; do not treat it as fixed. An old-code fixture crashed the Gamescope session, not the OS, and does not prove the new delivery path. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it.
committed. The worker boundary is intentionally small for forks experimenting
with other models/APIs; the default remains local-only Redux. The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](docs/overlay.md), [worker](docs/worker.md), [design](docs/design.md).
Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.<timestamp>` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](docs/third-party.md); [TODO](TODO.md).
No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection.
+104 -96
View File
@@ -6,95 +6,110 @@ M ≈ a day, L ≈ multi-day.
Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
downloads, no automatic Enter, small verified commits, docs must state implemented downloads, no automatic Enter, small verified commits, docs must state implemented
vs proposed behavior honestly. vs proposed behavior honestly. Checked source-work items below mean the local
implementation is present, **not** a shipped or headset-accepted release.
## Implemented locally; offline tested; installed/headset validation pending
The current working tree includes the source changes for A3, A5–A7, B1–B4,
C1–C3, D1–D2, E2–E3 and F1–F2. Offline checks passed 23/23 default,
24/24 strict UI, and 28/28 on a private native ARM64 snapshot. Later review
fixes require a fresh final native build before deployment.
These checkboxes close the *source tasks*, not their empirical acceptance gates.
C1's second backend is a fake executable fixture, **not** a second shipped ASR
engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's
source/attended installer paths still need an audited native archive and an
installed clean-account/Frame exercise. A4 (exact artifact ABI/license closure),
D3 (publication and clean-account acceptance), P1 (real-target delivery), G1
(browser), and H (live headset acceptance) remain open. No new build is claimed
deployed to Frame; local code/tests cannot establish a fixed delivery regression.
## Priority regression (reported during implementation)
- [ ] **P1. Repeated delivery loses the beginning of later submissions.** After
the first Type + Enter, later text reportedly loses a dozen to a few dozen leading
bytes. Investigate preview versus destination loss, retain the full bounded
literal transcript, and add repeated/long/Unicode delivery regression tests.
Do not assume a larger buffer fixes it or retry uncertain delivery automatically.
An old-code delivery fixture crashed the Gamescope session, **not** the OS;
this is not evidence of a fix. Real-target confirmation remains required after
a tested fix is deployed.
## A. First release (v0.1) blockers ## A. First release (v0.1) blockers
- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README - [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
states local inference is free; blocker text removed from README/docs/scripts and states local inference is free; blocker text removed from README/docs/scripts and
replaced with a neutral dependency note in `docs/third-party.md`. replaced with a neutral dependency note in `docs/third-party.md`.
- [x] **A2. Remove references to the unrelated app (kouseki).** (S) - [x] **A2. Remove references to the unrelated app.** (S)
Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a Tracked references were removed while preserving Inconsolata/OFL attribution,
final grep. Keep the Inconsolata/OFL attribution and cite the font SHA-256 and upstream googlefonts/Inconsolata source. A tracked,
upstream font source (googlefonts/Inconsolata) instead. case-insensitive grep for the former app name must remain empty.
*Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains. - [x] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" → "FrameYap"). (S) Done in baseline checkpoint `07c03ea`.
"FrameYap"). (S)
- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M) - [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M)
Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL Upstream inventory and the FreeType FTL choice are documented, but the exact
option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt staged ARM64 native/runtime binaries, transitive wheel/library notices, symbol
archive that includes any of them. versions, loader and libc floor still require artifact-specific review before
- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README, publishing any prebuilt archive.
`scripts/stage-native-poc.py`, `CMakeLists.txt` messages, - [x] **A5. Drop "POC" from the shipped surface.** (S) Public help, README,
installer strings. v0.1 is a first small release. CMake and installer use the release name. `scripts/stage-native-poc.py` remains
- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install, a deprecated compatibility wrapper for `scripts/stage-native.py`, not the
controls **table** (button → action), then a "Status / not yet validated" section. documented or shipped staging entry point. v0.1 is a first small release.
Today it reads as a lab notebook and Controls is a wall of text. - [x] **A6. Rewrite the README front.** (M) Pitch, requirements, local-only install,
controls table and explicit "Status / not yet validated" section are present.
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved - [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`. to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and Current design/overlay/packaging docs distinguish implemented local behavior
hedging language before v0.1. from proposed and unaccepted headset/release behavior.
## B. Correctness / robustness ## B. Correctness / robustness
- [ ] **B1. A malformed transcript must not kill the worker.** (S) - [x] **B1. Keep malformed transcripts request-local.** (S)
`src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control `Controller::tick()` catches a correlated bad UTF-8/control reply and fails the
characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and request without stopping the ready worker; hardware-free fakes test retry.
`session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a - [x] **B2. Use engine-neutral C++ worker errors.** (S)
request-level error (like the `E` path: keep the worker, show "transcription `F`/`I` messages no longer name Redux's Python engine.
failed", allow retry). Add a test with a control-character reply asserting the - [x] **B3. "Close mic when idle" setting, default OFF.** (M)
worker stays ready. Config/Settings and fake-backed mic-lifetime tests cover default idle draining
- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
"moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a - [x] **B4. Extract/test interaction logic from `run()`.** (L)
worker-supplied message code. Prerequisite for C1. `Controller` receives injectable audio/worker/focus/delivery interfaces; offline
- [ ] **B3. "Close mic when idle" setting, default OFF.** (M) tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.
Default keeps the mic open while Ready (opening/closing per PTT causes an audio
spike on the physical hardware, and it avoids first-syllable clipping). The
setting closes it between clips for people who don't want a live device. Document
the tradeoff (spike/latency) next to the toggle, in Settings and in the docs.
Persist in `config.json`; add to the panel Settings tab.
- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L)
`run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no
tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`)
so PTT, cancel, quick phrases, auto-insert and error transitions are testable
without hardware. B1 is the first regression test.
## C. Backends and model management ## C. Backends and model management
- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L) - [x] **C1. Manifest-driven worker backends (source capability).** (L)
Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper, Redux's pinned manifest and a generic local dispatcher implement the existing
sherpa-onnx Parakeet, …) as separate worker executables speaking the existing `Y`/`T`/`R`/`E` framing. An offline fake second executable backend works without
`Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name, edits to C++ worker/runtime; only Redux has a shipped inference engine. New
launcher, pinned model files + hashes + attribution, license text, CPU/GPU backends still require license/runtime and actual inference validation.
requirements) so nothing is hardcoded in C++ or `model_files.py`. - [x] **C2. Model/backend state and chooser (local source).** (L)
*Done when:* Redux is expressed as a manifest, and a second backend can be added Settings lists manifest-backed model status and active/loading/ready/failure
without touching `worker.cpp`/`runtime.cpp`. states; selection persists and invalidates/restarts worker, clip and review.
- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4. Install → Confirm Install displays pinned source, size, license, attribution and
Settings page listing backends/models with state (not installed / installed and manifest SHA-256; confirmation launches an owned installer helper with that
verified / loading / ready / failed), the active one marked, and selection that digest and offline rechecks afterward. Full consent metadata is paginated;
restarts the worker. Target user is non-technical: an **Install** button on the the panel shows bounded per-file download/verification events and sanitized
panel runs the installer's machine-readable mode (D2) as a child process and shows errors. Installer output alone never proves success. No implicit downloads or
progress/errors in the panel, so nobody needs a terminal. The click is the consent; runtime install. **Installed chooser/consent behavior on Frame remains unvalidated.**
there are still no implicit or background downloads, and the panel states what - [x] **C3. Offline model status CLI.** (S) `frameyap --list-models` and
will be downloaded and how large it is before it starts. Persist the choice in `--check-model ID` dispatch pinned local manifest/hash checks, without ASR or
`config.json`. downloads. Installed archive CLI still needs clean-account verification.
- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID`
(offline, hash-verifies installed files) so the UI and installer share one
implementation.
## D. Installer ## D. Installer
- [ ] **D1. Installer with a binary-or-source choice.** (L) - [x] **D1. Local binary-or-source installer paths.** (L)
`install.sh` offers *prebuilt archive* (checksummed) or *build from source* `install.sh` has a checksummed binary route and explicit local source/toolchain
(checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then preflight/build/stage/install route, with rollback and no default registration.
continues automatically through install after the user's choices. Retain rollback, A native archive was exercised in private HOME/XDG roots on Frame: install,
idempotency, no sudo, no Steam AppID, opt-in autolaunch. installed model-status CLI, idempotency, wrapper repair and uninstall. This
- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M) same-user/system-library fixture is not clean-account acceptance.
Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`, - [x] **D2. Attended-or-unattended model-agnostic installer interface.** (M)
`--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`, TTY choices have equivalent flags; `--mode binary|source`, `--backend ID`,
`--json` output) so a model/agent can run it non-interactively; interactive `--model-dir`, `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`,
prompts only run on a TTY and print the equivalent flags they chose. Exit codes `--print-plan` and `--json` support offline plans and structured outcomes;
and messages must be machine-readable. explicit model installs use installed pinned manifests. Tested with local
fixtures only, not a released archive or an installed Frame UI handoff.
- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the - [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the
release checklist in `docs/packaging.md`; do not advertise the one-command route release checklist in `docs/packaging.md`; do not advertise the one-command route
until the archive and its checksum are actually published and tested from a clean until the archive and its checksum are actually published and tested from a clean
@@ -106,32 +121,24 @@ vs proposed behavior honestly.
says what it is; no rename churn before the first tag. If a hardware-neutral says what it is; no rename churn before the first tag. If a hardware-neutral
project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
front end), decide it before the cross-window work in G, not now. front end), decide it before the cross-window work in G, not now.
- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text + - [x] **E2. User-facing Type / Type + Enter / Quick phrases labels.** (S)
space) and **Type + Enter**; align overlay buttons, `--check-controls` output, Overlay, diagnostics, README/help and overlay docs align on these controls;
README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only `insert`, `enter`, `quick_chat` remain internal binding/API names. Settings
if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model
explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux" from its `moondream` Python inference package.
vs `moondream` naming once in `docs/worker.md`. - [x] **E3. Numeric `MAJOR.MINOR.YYYYMMDDHHMM` version source work.** (S)
- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S) CMake, CLI/tests, package producer and installer use numeric release versions;
e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts dev git info is a separate `--version` line. `SOURCE_DATE_EPOCH` can supply
correctly, keeps the build date visible, URL/filename-safe. The version field is the configuration timestamp. A clean release tag/archive still needs D3.
just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted
changes) suffixes were only for telling developer builds apart, so they move out
of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev
build adds a second line like `git abc12345 (uncommitted changes)`. Release
archives are built from a clean tag, so users never see it. Update the CMake
version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`,
installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the
timestamp. Tag releases `v0.1.<timestamp>`.
## F. Code structure (non-urgent) ## F. Code structure (non-urgent)
- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of - [x] **F1. Move `--check-*` diagnostics out of `main.cpp`.** (S)
inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a `src/check.cpp` owns native checks; `src/cli.cpp` provides the table-driven
table-driven option parser. parser. Device behavior remains separately gated.
- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag - [x] **F2. Split `overlay.cpp`'s `Impl`.** (M)
state, save-failure flags, counters, pose caches): separate drag, persistence and Drag state, persistence/save failures and diagnostics now have separate grouped
diagnostics. owners; native snapshot build and offline panel/drag tests cover the refactor.
## G. Other windows (post-release) ## G. Other windows (post-release)
@@ -158,7 +165,7 @@ this app is future design and out of scope for v0.1.
across a normal restart (without restarting sessions just for the test). Confirm no across a normal restart (without restarting sessions just for the test). Confirm no
runtime/model environment override before future live checks. runtime/model environment override before future live checks.
- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery, - [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery,
Auto Insert with speech, physical resize. Auto insert with speech, physical resize and a verified deployed version.
--- ---
@@ -179,4 +186,5 @@ this app is future design and out of scope for v0.1.
## Open questions ## Open questions
None currently blocking. Release artifact compatibility/license audit, publication, P1 real-target behavior
and live headset validation are unresolved gates, not implied by checked source tasks.
+55 -25
View File
@@ -11,8 +11,9 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
mode while FrameYap is visible; it may affect games, and is not an input override. mode while FrameYap is visible; it may affect games, and is not an input override.
- Remappable SteamVR actions. The default Steam Frame binding maps right X - Remappable SteamVR actions. The default Steam Frame binding maps right X
(hold to record, release to transcribe) to the existing PTT action using the (hold to record, release to transcribe) to the existing PTT action using the
observed `frame_controller` profile. Right B cancels, A inserts + space, and Y observed `frame_controller` profile. Right B cancels, A requests Type (text +
inserts pending text + Enter (or Enter only with no preview). The Bindings button space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay
control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button
requests SteamVR's remapping editor directly. Grip bindings remain, but both grip requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
actions were inactive in the observed dashboard check; do not rely on them. actions were inactive in the observed dashboard check; do not rely on them.
If left grip becomes active, two short taps request explicit Enter. If left grip becomes active, two short taps request explicit Enter.
@@ -22,27 +23,41 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum, - SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
20 second maximum. In the native app the microphone stream opens after model 20 second maximum. In the native app the microphone stream opens after model
warm-up, stays running between utterances, and discards idle samples; PTT does warm-up, stays running between utterances, and discards idle samples; PTT does
not open/pause/close the device. Quit, worker restart, device failure or capture not open/pause/close the device **by default**. Settings → Close mic when idle
failure closes it. This avoids repeated capture-device transitions but does not (`"close_mic_when_idle": true`, default **false**) closes it between clips and
promise glitch-free playback on every audio stack. Other apps may still reopens on PTT; repeated transitions caused an audio spike on Frame and may
add latency or clip the first syllable. Quit, worker restart, device failure
or capture failure also closes it. Keeping it open does not promise glitch-free
playback on every audio stack. Other apps may still
transmit your voice: this app does **not** mute VRChat or any other app. transmit your voice: this app does **not** mute VRChat or any other app.
- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed
backend restarts the worker and invalidates pending clip/review/focus authority;
a missing or invalid model disables recording. A separate Install then Confirm
Install exposes pinned source/size/license/attribution and exact manifest-byte
SHA-256 before installer handoff. These are local implementations, not a tested
installed UI or an approved additional inference engine. Only Redux is supplied.
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips, - Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
cancellation/reaping and deadlines; exact pinned model SHA-256 verification. cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
Model imports are lazy and loading is offline. Normal repeats, request-local Model imports are lazy and loading is offline. A correlated bad transcript/
transcription failures and microphone failures retain the loaded model; `E` reply is a request-level failure: Record can retry without dropping the
ready model. Normal repeats and microphone failures retain the loaded model;
microphone device failure releases the stream for explicit retry. A cancelled microphone device failure releases the stream for explicit retry. A cancelled
in-flight request or broken worker protocol may require reloading. No in-flight request or broken worker protocol may require reloading. No
cloud/desktop fallback. cloud/desktop fallback.
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable - Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
handling, UTF-8/control validation and explicit Submit action for Enter. handling, UTF-8/control validation and explicit Type + Enter action.
Insert ensures a trailing space without doubling an existing one. Enter Type ensures a trailing space without doubling an existing one. Type + Enter
first inserts pending review, releases the text lease, then acquires a fresh first types pending review, releases the text lease, then acquires a fresh
lease for Submit. Failed/uncertain text never proceeds to Submit; failed lease for Enter. Failed/uncertain text never proceeds to Enter; failed
Submit acquisition never replays text. A full 4096-byte transcript without Enter acquisition never replays text. If a validated transcript fills the
room for a space is preserved with an error, never silently truncated. 4096-byte bound and lacks a trailing space, Type preserves all its bytes and
- Idempotent user-local release-archive installer: SHA-256, safe extraction, queues it **without** the usual space; it does not signal a separate error.
atomic current-version selection, retained rollback, runtime/install lock, Destination consumption and repeated-delivery behavior remain unaccepted.
foreign-file refusal and explicit unregister-before-uninstall acknowledgement. - User-local installer with checked binary-archive or explicitly provisioned
source-build mode, safe extraction, atomic current-version selection, retained
rollback, runtime/install lock, foreign-file refusal and explicit
unregister-before-uninstall acknowledgement. Source mode needs a local
compiler, SDK, libraries and license inputs; no runtime is pip-installed.
## Deliberately not claimed ## Deliberately not claimed
@@ -63,15 +78,26 @@ fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip binding
are not guaranteed globally active in every scene/dashboard state, and the app are not guaranteed globally active in every scene/dashboard state, and the app
never enables SteamVR's experimental overlay overrides on your behalf. never enables SteamVR's experimental overlay overrides on your behalf.
This is a compact prototype panel, not yet the proposed polished miniature status This is one compact panel, not a separate miniature status chip. Front-prefix
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery loss on repeated submissions (P1) is under separate investigation, **not fixed**
impact and target application compatibility require further headset work. by the chooser/worker documentation or any speculative delivery change. Font
coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact
and target application compatibility require further headset work.
## Inference runtime ## Inference runtime
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Pinned
runs through the `moondream` Python package and its Kestrel runtime, which you file sizes/hashes and attribution live in `assets/backends/redux.json`; offline
install in your own environment; builds/tests never fetch it. See `--list-models`/`--check-model redux --model-dir /absolute/model` (or
`scripts/model-status.py`) verify without inference or downloads. Manifest
schema/verification are in `python/frameyap/model_files.py`. The local generic
dispatcher resolves a manifest's in-release Python/executable launcher and
checks request/reply correlation; a new manifest still needs its own licensed,
compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and
`--manifest-dir /absolute/manifests` are wired through
the installed launcher as an explicit override, not a provisioning command.
Inference uses the `moondream` Python package and Kestrel runtime, separately
provisioned in your own environment; builds/tests/installer do not pip-install them. See
[third-party notes](third-party.md). [third-party notes](third-party.md).
## Developer native build ## Developer native build
@@ -142,12 +168,16 @@ kestrel 0.8.0) and local weights:
``` ```
Use a disposable text destination first. `--run` loads the model but does not Use a disposable text destination first. `--run` loads the model but does not
record until an explicit recording control. Click Insert only after focusing your record until an explicit recording control. Click Type only after focusing your
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
and terminates only the owned worker. Installer upgrades refuse an active app. and terminates only the owned worker. Installer upgrades refuse an active app.
The worker API is intentionally small: a fork can replace the worker implementation The `Controller` receives injected `ControllerAudio`, `ControllerWorker`,
or add its own model/API integration without changing overlay and delivery code. `ControllerFocus` and delivery factory interfaces; hardware-free fakes can test
state, retry, authorization and mic lifetime without initializing OpenVR or
recording speech. The worker API is intentionally small: a fork can replace the
worker implementation or add its own model/API integration without changing
overlay and delivery code.
The default product remains local-only Redux; extending a fork does not authorize The default product remains local-only Redux; extending a fork does not authorize
sending existing users' audio to a service. sending existing users' audio to a service.
+61 -39
View File
@@ -1,11 +1,16 @@
# FrameYap native dictation overlay — proposal # FrameYap native dictation overlay — design and remaining goals
Standalone project design. This document is the full target design, not a blanket Standalone project design. This document is the full target design, not a blanket
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
Redux adapter, review-first Gamescope insertion and a user-local archive installer. Redux adapter, review-first Gamescope insertion and a user-local archive installer
with explicit source-build option. Redux's pinned manifest, offline status CLI,
manifest-driven dispatcher and in-panel Models selection/confirmation are
implemented locally. The installed chooser/installer handoff is **not yet
end-to-end verified on Frame**; the native-only archive needs an external CPU
runtime for speech.
Opt-in conservative Xwayland focus tracking is implemented locally but not yet Opt-in conservative Xwayland focus tracking is implemented locally but not yet
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware accepted for live automatic typing on Frame. A separate status chip and full hardware
acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md). acceptance remain possible future work. See [build and scope](build.md) and [third-party notes](third-party.md).
Future work is tracked in [TODO.md](../TODO.md). Future work is tracked in [TODO.md](../TODO.md).
## Recommendation ## Recommendation
@@ -17,7 +22,9 @@ as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene
renderer, avatar, desktop capture or root service is needed in the primary path. renderer, avatar, desktop capture or root service is needed in the primary path.
A first-class product goal is a **one-command GitHub install without a Steam store A first-class product goal is a **one-command GitHub install without a Steam store
AppID**. Package a prebuilt native executable and isolated CPU runtime; use a normal AppID**. Package a prebuilt native executable; the current native-only archive requires a
separately supplied, compatible CPU runtime (no automatic pip install). A future
isolated runtime bundle requires its own license and compatibility audit. Use a normal
OpenVR application key for registration, not Steamworks. Installation must remain OpenVR application key for registration, not Steamworks. Installation must remain
user-local with opt-in autolaunch. See [installation design](install-design.md). user-local with opt-in autolaunch. See [installation design](install-design.md).
@@ -31,12 +38,14 @@ one persistent local Parakeet Redux CPU worker
correlated literal transcript → focus/delivery policy correlated literal transcript → focus/delivery policy
↓ ↓
Gamescope IME set_string + commit → focused Frame application Gamescope IME set_string + commit → focused Frame application
↘ overlay preview / error / explicit Insert when delivery is unsafe ↘ overlay preview / error / explicit Type when delivery is unsafe
``` ```
The CPU worker is local process isolation, not remote inference or a service The CPU worker is local process isolation, not remote inference or a service
framework. Implement a small transport owned by this repository. Keep the model framework. The locally implemented manifest dispatcher uses a small bounded
loaded between utterances; do not spawn Python/load 178 MB for every release. transport owned by this repository; only Redux inference is shipped. Keep the
model loaded between utterances; do not spawn Python/load 178 MB for every release.
A correlated bad transcript is a request-level failure, not a model unload.
## Minimal interaction ## Minimal interaction
@@ -53,27 +62,29 @@ armed remains a possible refinement, not a second implemented overlay. See
Recording… 00:04 [ Cancel ] Recording… 00:04 [ Cancel ]
"The recognized text appears here." "The recognized text appears here."
[ Insert + space ] [ Discard ] [ Insert + Enter — explicit ] [ Type + space ] [ Discard ] [ Type + Enter — explicit ]
``` ```
- States: disabled, warming, ready, recording, transcribing, review, inserted, - States: warming, ready, recording, transcribing, review, input queued,
unavailable/error. Recording uses visible icon + text, not color alone. unavailable/error. Input queued is not an application receipt. Recording uses
visible icon + text, not color alone.
- Default Frame bindings: hold right X to speak, release to finish; B cancels, - Default Frame bindings: hold right X to speak, release to finish; B cancels,
A inserts with a trailing space, Y opens/cycles the quick-chat selection. A types with a trailing space, Y opens/cycles the Quick phrases selection.
Overlay Submit or left-grip double-tap submits the selected literal + Enter, Overlay Type + Enter or left-grip double-tap submits the selected literal + Enter,
or pending review + Enter, or Enter alone if neither is present. or pending review (normally + space) then Enter, or Enter alone if neither is present.
The alternate right-grip tap-then-hold gesture and left-grip double-tap Submit The alternate right-grip tap-then-hold gesture and left-grip double-tap Type + Enter
remain available. All are remappable through the Bindings button's SteamVR editor. remain available. All are remappable through the Bindings button's SteamVR editor.
A click-to-start/stop overlay button provides A click-to-start/stop overlay button provides
a binding-independent alternative. Bound recording to 20 seconds; discard a binding-independent alternative. Bound recording to 20 seconds; discard
accidental taps (initial threshold: 200 ms). accidental taps (initial threshold: 200 ms).
- **Quick typing (opt-in, locally implemented):** insert on completion only when - **Auto insert (opt-in, locally implemented):** type on completion only when
uninterrupted Xwayland target observation remains valid. **Review mode (default):** uninterrupted Xwayland target observation remains valid. **Review mode (default):**
wait for Insert. Bring up review on uncertainty rather than silently losing a wait for Type. Bring up review on uncertainty rather than silently losing a
transcript or typing into a new target. Live Frame acceptance remains open. transcript or typing into a new target. Live Frame acceptance remains open.
- Submit requires its own explicit control activation: insert pending review with - Type + Enter requires its own explicit control activation: type pending review
a trailing space, then queue Enter only if the text step succeeds. With no (normally with a trailing space), then queue Enter only if the text step succeeds.
review, it queues only Enter. Never interpret "submit", "delete" or other speech At the 4096-byte bound, preserve the entire transcript without a suffix if no
space fits. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech
as commands. Transcription completion never auto-submits. as commands. Transcription completion never auto-submits.
- No generic "undo last dictation" initially: another application's edits/cursor - No generic "undo last dictation" initially: another application's edits/cursor
cannot be reliably rolled back by a guessed number of backspaces. cannot be reliably rolled back by a guessed number of backspaces.
@@ -109,7 +120,7 @@ license-reviewed extraction, never a runtime path into another project's checkou
### Controller bindings ### Controller bindings
Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the Expose PTT, cancel and explicit Type/Type + Enter as named SteamVR actions; let the
user bind them. Do not assume a scene app's left-bumper mapping works globally or user bind them. Do not assume a scene app's left-bumper mapping works globally or
silently takes a game's button away. Check action activity and neutral rearm. silently takes a game's button away. Check action activity and neutral rearm.
@@ -143,7 +154,7 @@ The server implements text through synthetic key events and a temporary keymap,
not a guaranteed rich-text/IME edit operation in every app. Verify actual target not a guaranteed rich-text/IME edit operation in every app. Verify actual target
toolkits and games. Respect singleton/unavailable handling and Steam-keyboard toolkits and games. Respect singleton/unavailable handling and Steam-keyboard
coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe: coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe:
its inspected sample loop is byte-oriented and interprets newline as Submit. its inspected sample loop is byte-oriented and interprets newline as Enter.
### 2. Explicit fallbacks, not a framework built up front ### 2. Explicit fallbacks, not a framework built up front
@@ -183,11 +194,11 @@ Gamescope exposes focus-display/window root properties, but their encoding and
relationship to seat focus need implementation-specific validation. Do not infer relationship to seat focus need implementation-specific validation. Do not infer
that X display `:0` is always the destination, or that an X focus observation that X display `:0` is always the destination, or that an X focus observation
identifies a native Wayland text field. For unobservable native Wayland focus, identifies a native Wayland text field. For unobservable native Wayland focus,
require explicit review/Insert; do not advertise safe auto-targeting. require explicit review/Type; do not advertise safe auto-targeting.
If focus changes, keep the result in review. A fresh Insert explicitly approves If focus changes, keep the result in review. A fresh Type explicitly approves
the current destination and creates a new delivery authorization. Recheck again the current destination and creates a new delivery authorization. Recheck again
at insertion. This minimizes stale delivery but does **not** eliminate a race at typing. This minimizes stale delivery but does **not** eliminate a race
between the final check and global input processing; do not claim otherwise. between the final check and global input processing; do not claim otherwise.
A Wayland roundtrip means compositor processing, not application consumption. A Wayland roundtrip means compositor processing, not application consumption.
Report `input queued`, never `message sent`. Report `input queued`, never `message sent`.
@@ -221,9 +232,11 @@ that tiny dataset; keep transcript visibility and a cheap retry.
Implement a small independent audio/worker adapter with explicit capture, Implement a small independent audio/worker adapter with explicit capture,
single-request bounds, owner-only runtime files, correlated replies and cancellation. single-request bounds, owner-only runtime files, correlated replies and cancellation.
The worker should be implemented independently; do not link, vendor or import The worker is independently implemented; do not link, vendor or import another
another application's speech code. Avoid a generic provider framework: one application's speech code. A narrow manifest dispatcher now selects a checked
explicit Redux worker is enough for the first version. local backend launcher; this is not a general cloud/provider framework and only
Redux is supplied with a runtime implementation. Other manifests require their
own audited offline engine, launcher and tests.
Suggested ownership, introduced only as implementation needs it: Suggested ownership, introduced only as implementation needs it:
@@ -243,12 +256,20 @@ processing timeout. No shell commands in IPC and no input authority in the worke
native kernel pools. Choose measured latency versus compositor contention, native kernel pools. Choose measured latency versus compositor contention,
not the desktop's thread count by habit. No real-time scheduling or permanent not the desktop's thread count by habit. No real-time scheduling or permanent
CPU pinning initially; inspect runtime affinity behaviour during measurement. CPU pinning initially; inspect runtime affinity behaviour during measurement.
- Explicit local model path and offline loading. Missing runtime/weights produces - Explicit local model path and offline loading. Pinned files and attribution live
an actionable error, not an unsolicited download/network fallback. in `assets/backends/redux.json`; `python/frameyap/model_files.py` validates its
- Optional explicit enable/warm-up before first PTT; warming must not record audio. schema and hashes. The CLI's offline `--list-models`/`--check-model` checks
Expose that lifecycle explicitly rather than warming on import or construction. never start inference. Settings → Models selects/saves the manifest ID and
Otherwise display first-use loading honestly. Keep restarts the worker, invalidating any audio/review/delivery authorization.
model reuse after normal completion; cancellation may restart the owned worker. Install requires a separate confirmation showing source, size, license,
attribution and exact-manifest SHA-256; only that click hands off to the
installer for pinned model files. Missing runtime/weights produces an
actionable error, not an unsolicited download/network fallback.
- Native `--run` explicitly starts offline verification then warms a verified
selected local model before PTT; warming must not record audio. Missing or
unverified files disable Record rather than prompting a background download.
Keep model reuse after normal completion and request-local transcript errors;
cancellation may restart the owned worker.
- Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded - Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded
tmpfs-backed clips; remove them on completion, error, cancellation and shutdown. tmpfs-backed clips; remove them on completion, error, cancellation and shutdown.
Avoid persistent audio/transcripts by default. Local IPC is not a network hop; Avoid persistent audio/transcripts by default. Local IPC is not a network hop;
@@ -278,14 +299,14 @@ These are proposed implementation gates, **not completed acceptance**:
confirm dashboard/hand placement, input events, close/reopen and no scene-focus confirm dashboard/hand placement, input events, close/reopen and no scene-focus
takeover. Validate global PTT separately rather than blocking the clickable takeover. Validate global PTT separately rather than blocking the clickable
prototype on experimental override support. prototype on experimental override support.
3. **Real dictation path:** microphone → local Redux → preview → explicit insert 3. **Real dictation path:** microphone → local Redux → preview → explicit Type
into a disposable target; then enable quick typing after target tracking tests. into a disposable target; then validate opt-in Auto insert after target tracking tests.
Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate
replies, lost mic, worker crash and missing model without persisting speech. replies, lost mic, worker crash and missing model without persisting speech.
4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game 4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game
text fields; Steam keyboard coexistence; native Wayland targets separately. text fields; Steam keyboard coexistence; native Wayland targets separately.
Test focus changes during capture/inference, rapid loss/regain, held modifiers, Test focus changes during capture/inference, rapid loss/regain, held modifiers,
explicit Insert retargeting, no hidden Enter, and no second delivery. explicit Type retargeting, no hidden Enter, and no second delivery.
5. **In-headset acceptance:** readable feedback and comfortable PTT; measured 5. **In-headset acceptance:** readable feedback and comfortable PTT; measured
release-to-insert latency and compositor timing while an actual scene runs; release-to-insert latency and compositor timing while an actual scene runs;
no noticeable sustained thermal/battery regression. Set numeric budgets after no noticeable sustained thermal/battery regression. Set numeric budgets after
@@ -301,8 +322,9 @@ without moving recognition off Frame. It is not part of this initial design.
The default hardware-free build needs only CMake and a C++20 compiler (Python The default hardware-free build needs only CMake and a C++20 compiler (Python
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR, runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
SDL3, FreeType and Wayland client/generated protocol bindings. A separately SDL3, FreeType and Wayland client/generated protocol bindings. A separately
authorized Python Redux environment is explicitly supplied at launch. Pin revisions authorized Python Redux environment is explicitly supplied at launch; the
and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery. native-only installer neither bundles it nor pip-installs one. Pin revisions
and review licenses for each introduced dependency. No automatic fetch/install in configure or normal tests; no external checkout discovery.
Hardware-free tests should cover state transitions, bounded PCM/transcripts, Hardware-free tests should cover state transitions, bounded PCM/transcripts,
worker framing/timeout/cancellation, duplicate/stale replies and focus generations worker framing/timeout/cancellation, duplicate/stale replies and focus generations
+73 -99
View File
@@ -1,108 +1,82 @@
# Installation and distribution goal # Installation goal and current boundary
**User goal:** install from GitHub with a `curl … | bash`-style command, without a Goal: a one-command, pinned GitHub release install for Steam Frame without a Steam
Steam store AppID. An idempotent archive installer and native-only local artifacts store AppID, sudo or end-user compiler. **No public archive or verified clean
are now implemented/tested; no public release is published. A bundled-ASR install is published. Do not advertise a `curl | sh` command as functional.**
experience is not yet offered. This document retains the target The local installer has binary-archive and explicitly provisioned source-build
design; see [current packaging](packaging.md) and [third-party notes](third-party.md). modes, machine-readable plans/results and an attended TTY path. The current
Planned installer work (binary-or-source choice, flag-driven operation) is in native-only artifact does not include or pip-install an ASR runtime; it is not
[TODO.md](../TODO.md). a one-command voice-typing experience. See [packaging](packaging.md) for exact
flags and [third-party inventory](third-party.md) for open license/ABI audits.
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable `scripts/install-preflight.sh` is a read-only Linux ARM64/glibc/bootstrap check;
for the **proposed** Linux ARM64 glibc package format and has the expected basic `--source` adds toolchain/library checks. It does not download, install, register,
bootstrap utilities. It reports system Python, Git and uv, but none is required or certify model/runtime compatibility or a minimum libc version. Its report of
for the intended bundled release. A read-only check on one Frame observed Python/Git/uv availability is not a runtime guarantee: `install.sh` currently
Python 3.12.3 and Git, but not uv; availability may change. This script needs Python 3.12+ **for its bootstrap**, while native-only voice inference needs
is not an installer or a model/runtime compatibility test; it has no downloads, a separately provisioned compatible CPU Python environment and pinned weights.
registration, SteamVR initialization or persistent changes. No glibc minimum can
be certified until release artifacts are chosen and tested.
## Non-Steam overlay identity ## OpenVR identity
OpenVR overlay applications do not require a Steam store AppID or Steamworks. `local.frameyap.overlay` is a string OpenVR application key, **not** a Steam
The native executable initializes as `VRApplication_Overlay`. For discoverability store AppID. The installer creates a manifest/desktop launcher user-locally but
and optional autolaunch, register an OpenVR application manifest with a stable, does not register/launch the app by default. Explicit `frameyap --register
project-owned **string application key** (proposed: `local.frameyap.overlay`). /absolute/manifest/path` uses the OpenVR registration API; registration alone
That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam did not reveal a launcher in the first checked dashboard menu. On one Frame the
library shortcut should be necessary for the normal route. user opened the panel from the **Non-Steam** section and quit; shortcut discovery
and persistence after a normal restart are still unverified. Optional
`--autolaunch`/`--no-autolaunch` on the installer explicitly request OpenVR
registration/autolaunch choices; no SteamVR settings or sessions are changed
without that request. Unregister explicitly before uninstall. SteamVR must
already be available for registration; never start/restart it for installation.
The manifest identifies the installed executable; the installer/registration ## Installation workflows
helper should use `IVRApplications::AddApplicationManifest` and the corresponding
remove operation, not hand-edit Steam's internal JSON. Autolaunch uses the OpenVR
application setting only when explicitly requested. Validate the exact manifest,
launch behaviour, registration persistence and uninstall on native Frame before
claiming this route works end to end. Existing probes established overlay client
initialization, not manifest installation.
SteamVR/OpenVR must already be installed and usable. If registration needs a - **Binary mode** (no compiler): verify a locally supplied ARM64 archive digest,
running runtime, defer it to the first explicit launch rather than starting or or, after a vetted release actually exists, explicitly approve retrieval of
restarting SteamVR behind the user's back. a pinned `v0.1.YYYYMMDDHHMM` GitHub tag and checksum. No moving `latest` tag.
`--mode binary --archive FILE --sha256 HASH --version 0.1.YYYYMMDDHHMM`
selects the local-artifact path.
- **Source mode**: requires explicit local source, SDK, SDL/OpenVR libraries and
their notices, CMake/C++20, native build dependencies and a version. It builds,
stages, packages and continues through local installation; it does **not**
provision ASR packages or bypass producer license obligations. See
[packaging](packaging.md) for all flags.
- **Model**: only an explicit `--install-model --backend redux --yes` fetches
pinned public files for the *already installed* backend. Inspect the read-only
`--print-plan --json` first; it includes model size, attribution and license
metadata from `assets/backends/redux.json`. `--expected-manifest-sha256 HASH`
binds consent to the exact installed manifest bytes and fails before model
directory creation/network if they changed. The local in-panel chooser shows
source, size, license text, attribution and manifest digest, then requires a
second **Confirm Install** click; the installed/native UI route still needs
clean-target and headset acceptance. `--without-model` permits an
archive install without bundled model files. Neither operation installs Torch,
moondream, Kestrel or an interpreter. Launch paths to an independently
authorized runtime/model can be set in `paths.conf`.
- **Noninteractive**: supply flags and `--yes` for network/model consent.
`--print-plan` is read-only; `--json` provides structured results/errors and
progress events for a model download. No prompt reads stdin in a pipe. An
empty TTY invocation offers a local menu and prints equivalent flags.
## Intended user experience Installation is user-local under XDG data/config paths with a managed launcher,
retained rollback, SHA-256/path validation, foreign-file refusal and a lock
shared with the app. No OS package changes, udev rule, root service, Steam store
listing, unrelated application dependency, microphone recording, input injection
or automatic update daemon. Hashes detect accidental/unauthorized alteration
of a downloaded artifact but do not authenticate a compromised publisher;
release metadata needs independent trust. No installer operation silently runs
pip or launches inference. Native-only archives support overlay checks but need
an externally provisioned runtime/weights before voice typing. Selecting an
uninstalled backend does not authorize a download or supply its inference engine.
1. Run one documented command from the eventual GitHub repository/release. ## Gate before publishing the goal as fulfilled
2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and
explains/downloads the application, compatible CPU runtime and pinned model.
3. User-local installation provides a simple `frameyap` launcher, desktop
entry where supported, and an OpenVR manifest. No compiler, engine checkout,
Python dependency troubleshooting or separate ASR server for ordinary users.
4. The user explicitly launches/enables dictation. No installation-time microphone
recording, input injection, inference benchmark or overlay takeover.
Illustrative command shape only; `OWNER`, `REPO` and `VERSION` are placeholders: Vet the **exact** release closure/licenses, ARM64 symbol versions/loader,
model attribution and compatible CPU Python environment; establish a tested
```sh libc/runtime floor, then publish and authenticate a checksummed archive from a
curl --fail --silent --show-error --location \ clean tag. On a clean supported Frame, install without a compiler/sudo/store ID,
https://raw.githubusercontent.com/OWNER/REPO/VERSION/install.sh | bash load Redux, type into a disposable owned target and validate rollback/uninstall,
``` foreign-file failures, autolaunch off, and no unintended session changes. Separately
validate microphone → review → delivered input and headset comfort. Offline
Also document a download-inspect-run path for users who do not want to pipe remote installer fixture tests and a local native build are not those acceptance gates.
code into a shell. Pin a release/tag instead of executing a moving branch by default.
Offer explicit version selection and noninteractive flags; do not read interactive
confirmation from stdin while the installer itself is arriving through that pipe.
## Packaging boundary
- Prebuilt ARM64 executable plus a known-compatible, isolated CPU inference runtime;
no external application libraries/assets and no system Python modification.
- Model fetched during explicit installation/setup, with pinned revision/hash and
attribution. A documented `--without-model` option can defer the large download.
No surprise first-utterance downloads. Runtime components are installed from
their own package index rather than redistributed in our tarball.
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
`/dev/uinput` permission changes or modifications to unrelated launchers.
- No Steam store AppID, Steamworks SDK, root service or network ASR dependency.
- Third-party notices for everything we redistribute must be included with a release.
## Installer lifecycle and safety
- Detect architecture, libc and prerequisites first. Unsupported hosts fail with a
clear explanation; never install an x86 payload silently on ARM64.
- Download to a staging directory, check versioned SHA-256 manifests and archive
paths, then atomically select the completed version. HTTPS/checksums alone do not
authenticate a compromised publisher; use signed release metadata if provided.
- Keep configuration across upgrades; retain the previous version for rollback.
Refuse or defer replacement while this application's process is running rather
than killing arbitrary processes. Never touch SSH or unrelated sessions.
- Autostart is opt-in (`--autostart` or explicit settings); do not enable a systemd
service or SteamVR autolaunch by default. Do not enable overlay input overrides.
- Uninstall removes only owned launcher, manifest registration and install files;
model/config deletion is separately explicit. Preserve other SteamVR apps.
- No update daemon initially. A deliberate rerun/update command is sufficient.
## Acceptance before advertising one-command installation
- Clean supported Frame: install without sudo/compiler/engine checkout/store AppID;
launch overlay, load local Redux and type into an owned disposable target.
- Normal use after installation needs no network connection or desktop ASR host.
- Failed download/hash, unsupported architecture, low disk space and interrupted
upgrades leave a usable previous install or a cleanly reported failure.
- Reinstall, rollback and uninstall preserve unrelated data and SteamVR entries.
- Noninteractive piped invocation never hangs on stdin; inspection-first path works.
- Autolaunch remains off unless chosen; uninstall removes only our registration.
- Hardware-free installer tests use temporary homes, mocked runtime registration
and local fixture artifacts. Never exercise a real user's Steam configuration
in ordinary CI/CTest.
+89 -31
View File
@@ -2,10 +2,12 @@
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel `src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion. only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
Native `--run` wiring in `src/main.cpp` is implemented behind the explicit Native `--run` wiring in `src/main.cpp` is behind the explicit
`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful `FRAMEYAP_NATIVE` build option. The Models chooser, offline verification and
compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a explicit installer handoff are implemented locally, **not** an accepted Frame
normal build or test. install/voice-typing path or a published release. Neither hardware-free tests
nor a successful compile establish Frame input, visibility, comfort or text
delivery. Launching the runtime is explicit, never part of a normal build or test.
## Rendering and controls ## Rendering and controls
@@ -49,7 +51,9 @@ This replaces the raw-upload rendering path; headset flicker acceptance still
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance. requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
The header shows local time and date instead of the former on-device/review The header shows local time and date instead of the former on-device/review
and current-mount labels. It updates when the displayed minute or date changes, and current-mount labels. Settings explains Hold Quit (hold 0.9 seconds then
release) and Lasers anytime (system-wide lasers may affect games). The review
tab describes Type and Type + Enter. It updates when the displayed minute or date changes,
not every frame. Settings toggles 12/24-hour time and cycles date Off → not every frame. Settings toggles 12/24-hour time and cycles date Off →
MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display; MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display;
mount choices remain in Settings. mount choices remain in Settings.
@@ -58,23 +62,25 @@ The complete transcript preview is paginated by glyph width and four-line
height; Previous and Next navigate it without changing the source transcript. height; Previous and Next navigate it without changing the source transcript.
Status fits on the single status line; the old bottom detail label is gone. Status fits on the single status line; the old bottom detail label is gone.
The footer remains available on all tabs: Record (labelled Stop while recording), The footer remains available on all tabs: Record (labelled Stop while recording),
Cancel, Insert, Submit, Hold Quit. Hold Quit needs a 900 ms press and release on Cancel, Type, Type + Enter, Hold Quit. Hold Quit needs a 900 ms press and release on
that same button; its thin progress bar shows the hold. Record can retry after an that same button; its thin progress bar shows the hold. Record can retry after an
error; it is disabled while warming/transcribing and until an existing review is error; it is disabled while warming/transcribing and until an existing review is
inserted or discarded. Cancel can stop worker startup. Insert and Submit are typed or discarded. Cancel can stop worker startup. Type and Type + Enter are
disabled during recording and transcription. A pointer action requires disabled during recording and transcription. A pointer action requires
a press/release on the same enabled control from the same cursor; focus loss, a press/release on the same enabled control from the same cursor; focus loss,
tab changes, action-state changes and relocation clear pending presses. Submit is *always* a separate tab changes, action-state changes and relocation clear pending presses. Type + Enter
deliberate action, not inferred from text. Insert appends a trailing space (without is *always* a separate deliberate action, not inferred from text. Type normally
doubling an existing trailing space). Submit inserts any pending review and then appends a trailing space (without doubling an existing one); a full 4096-byte
queues Enter; with no pending text it queues Enter only. Y opens the quick-chat transcript without room for that suffix is queued unchanged, with no extra error
list over the review area; each further Y press cycles its highlighted choice. for the missing space. Type + Enter types any pending review and then queues
Cancel closes the picker without discarding an existing review. Submit sends the Enter; with no pending text it queues Enter only. Y opens the Quick phrases
selected text *without* a trailing space, then Enter. The choices are short list over the review area; each further Y press cycles its highlighted choice. Cancel closes the picker without discarding an
existing review. Type + Enter sends the selected phrase *without* a trailing
space, then Enter. The choices are short
single-line literals, not speech commands. A failed text step never proceeds to single-line literals, not speech commands. A failed text step never proceeds to
Enter. Recording never automatically submits. Auto insert, when Enter. Recording never automatically submits. Auto insert, when
explicitly enabled, can queue text + space after transcription only under the explicitly enabled, can queue text (normally + space) after transcription only
stable Xwayland focus guard described below. under the stable Xwayland focus guard described below.
### Bindings button ### Bindings button
@@ -107,6 +113,8 @@ installer creates one with defaults on first install. Copy the shipped
"input_priority": "normal", "input_priority": "normal",
"advanced_debug": false, "advanced_debug": false,
"auto_insert": false, "auto_insert": false,
"close_mic_when_idle": false,
"backend": "redux",
"lock_layout": false, "lock_layout": false,
"clock_24h": false, "clock_24h": false,
"date_format": "mdy", "date_format": "mdy",
@@ -149,6 +157,18 @@ config, retaining other fields and formatting; invalid/unwritable configs are
left untouched and return failure. The installer backs up original bytes before left untouched and return failure. The installer backs up original bytes before
repairing invalid values, while valid `true` and `false` are retained. repairing invalid values, while valid `true` and `false` are retained.
`close_mic_when_idle` is a separate boolean, default **false**, also available
as Settings → **Close mic when idle**. Normally the SDL capture device stays
open while Ready and idle samples are discarded. Enabling this toggle closes it
between clips and opens it on PTT: this avoids an open idle capture device but
can cause an audio spike (observed with per-PTT transitions on Frame), startup
latency or first-syllable clipping. Settings persistently displays **OFF: discard
idle audio; ON: spike / start latency** below the toggle, as well as an ON/OFF
indicator; the status/detail line also explains a change when toggled. The
native setting is saved to `config.json`; a failed save applies only for this
session and warns. This is not a mute switch for other applications. Quit and
failure still close the device.
`auto_insert` is a separate boolean, default `false`, also available as a `auto_insert` is a separate boolean, default `false`, also available as a
Settings toggle. Only a **new** recording arms it. It observes the Xwayland Settings toggle. Only a **new** recording arms it. It observes the Xwayland
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
@@ -156,41 +176,77 @@ display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
Both properties and focus are rechecked after IME lease acquisition. A watched Both properties and focus are rechecked after IME lease acquisition. A watched
focus-out, root focus-property change (even if the same window returns), window focus-out, root focus-property change (even if the same window returns), window
destruction, held keyboard key, missing X display or any disagreement permanently destruction, held keyboard key, missing X display or any disagreement permanently
disarms that clip. The transcript then remains for explicit review/Insert. disarms that clip. The transcript then remains for explicit review/Type.
Native Wayland focus and child text-field focus cannot be safely inferred here; Native Wayland focus and child text-field focus cannot be safely inferred here;
those cases fall back to review. No automatic Enter, speech commands or retry. those cases fall back to review. No automatic Enter, speech commands or retry.
The compositor can still change focus in the gap between the final check and The compositor can still change focus in the gap between the final check and
global delivery, and IME commit is not an application receipt. This path has global delivery, and IME commit is not an application receipt. This path has
offline synthetic focus tests and a separate owned-target IME fixture; live offline synthetic focus tests and a separate owned-target IME fixture; live
speech-driven Auto Insert, target coverage and headset acceptance remain speech-driven Auto insert, target coverage and headset acceptance remain
unverified. The setting is preserved on upgrade unverified. The setting is preserved on upgrade
and a failed preference write applies only to the current session. and a failed preference write applies only to the current session.
`quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings, `quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings,
each at most 64 characters. Edit the JSON file and restart; there is no headset each at most 64 characters. Edit the JSON file and restart; there is no headset
text editor. Inputs are literal (not expanded or interpreted by FrameYap) and text editor. Inputs are literal (not expanded or interpreted by FrameYap) and
are sent to the current Gamescope focus, so check the destination before Submit. are sent to the current Gamescope focus, so check the destination before Type + Enter.
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`, `buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
`insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME` `insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME`
button paths. Omitted actions retain their bundled defaults; an empty string button paths. Omitted actions retain their bundled defaults; an empty string
disables a mapping, including after an upgrade. The Frame defaults are right disables a mapping, including after an upgrade. The Frame defaults are right
X = hold-to-talk, B = Cancel, A = Insert + space, Y = quick chat. Enter has no X = hold-to-talk, B = Cancel, A = Type + space, Y = Quick phrases. Enter has no
single-button mapping by default; the left grip double-tap still submits. single-button mapping by default; the left grip double-tap still requests Type + Enter.
An existing config mapping `enter` to right Y is migrated to quick chat in memory An existing config mapping `enter` to right Y is migrated to Quick phrases in memory
when `quick_chat` is absent; this does not overwrite custom mappings. when `quick_chat` is absent; this does not overwrite custom mappings.
Existing configs with empty actions retain those disabled mappings; change them explicitly Existing configs with empty actions retain those disabled mappings; change them explicitly
or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized; or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized;
SteamVR user overrides may still supersede it. On customized launches a generated SteamVR user overrides may still supersede it. On customized launches a generated
action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings` action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings`
(or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The (or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The
config is read once at launch, not hot-reloaded. On install/upgrade the installer config is read once at launch, not hot-reloaded (the in-panel backend selection
fills missing fields, removes retired keys, and resets invalid entries. It saves is saved separately). On install/upgrade the installer fills known missing fields,
the exact prior bytes under `config.json.backup-*` before a repair and refuses including `close_mic_when_idle` and `backend`, removes retired keys and resets
symlink/oversized config paths; valid customizations remain intact. The installed invalid entries. It saves the exact prior bytes under `config.json.backup-*`
launcher no longer pins `--font`, so this selection takes effect. Direct native before a repair and refuses symlink/oversized config paths; valid customizations
remain intact. The installed launcher no longer pins `--font`, so this selection takes effect. Direct native
launches with bad JSON, colors or button mappings fail startup rather than launches with bad JSON, colors or button mappings fail startup rather than
silently changing input behavior. silently changing input behavior.
### Models / backends (local implementation)
Settings → **Models / backends** opens a paginated local chooser. The saved
`"backend": "redux"` selects a *manifest ID*, not a model download;
`--backend ID` overrides it for that `--run` invocation. Selecting a listed backend
restarts the owned worker and closes capture, discarding pending audio, review
and any prior focus/delivery authorization. A failed preference write leaves the
choice active only for this session. The currently listed Redux model is the
only inference implementation supplied; adding a manifest alone does not add
an inference runtime. The Models tab labels local checks as checking, missing
(`not_installed`), invalid, or installed/verified; for the selected model it
also reports loading, ready or failed. An install-in-progress note reports
model provisioning, followed by a new offline check; a verified status alone
is not proof the CPU runtime loaded. Recording is disabled until the selected
files verify offline and the worker warms; missing files never trigger a silent
download. Model loading and the default open-while-Ready idle microphone policy
are unchanged. The panel's state and successful helper calls do **not** prove
microphone transcription, input delivery or headset acceptance.
For a missing/invalid selected model, **Install** first presents a separate
confirmation showing pinned source, approximate download size, license text,
attribution and the exact raw manifest SHA-256 fingerprint. Only **Confirm
Install** launches the local installer with that fingerprint; leaving the view
or changing metadata invalidates consent. The installer rechecks the *installed*
manifest bytes under its model lock before creating a download target or using
the network, downloads only on the explicit click, hashes pinned files and then
the app checks them again offline before enabling recording. Existing invalid
or unsafe model files are refused rather than silently overwritten. This
installs model files, **not** Python, Torch, moondream, Kestrel or other runtime
dependencies.
The source-tree UI needs an installed release for installer-backed provisioning;
a hand-edited manifest is not an approved artifact. The installer handoff and
native UI have not yet been accepted on a clean Frame. See
[packaging](packaging.md#consumer) for an inspection-first CLI path.
### Experimental controller input priority ### Experimental controller input priority
There are two independent gates: There are two independent gates:
@@ -348,7 +404,7 @@ recentring and readability still require a separately authorized headset check.
The opt-in native `--check-controls` probe logs pointer counters and action The opt-in native `--check-controls` probe logs pointer counters and action
callbacks to the terminal rather than repainting them on the panel. Its canvas callbacks to the terminal rather than repainting them on the panel. Its canvas
stays static for Record/Cancel/Insert/Enter clicks so those clicks can be checked stays static for Record/Cancel/Type/Type + Enter clicks so those clicks can be checked
without diagnostic texture uploads. Switching tabs or mount still updates without diagnostic texture uploads. Switching tabs or mount still updates
the visible panel. Diagnostics identify `renderer=Vulkan` and count the visible panel. Diagnostics identify `renderer=Vulkan` and count
`textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions. `textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions.
@@ -367,8 +423,10 @@ cmake --build build-native --target frameyap_texture_check
It does not initialize OpenVR or establish compositor/headset acceptance. It does not initialize OpenVR or establish compositor/headset acceptance.
`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert, `assets/actions.json` names seven actions: left/right grip, PTT, cancel,
Enter. `bindings_frame_controller.json` maps right X click to hold-to-talk PTT; Type, Type + Enter and Quick phrases. `insert`, `enter`, and `quick_chat` remain
internal binding keys; visible controls read Type, Type + Enter, Quick phrases.
`bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
the grip bindings remain for optional remapping/diagnosis. In one dashboard the grip bindings remain for optional remapping/diagnosis. In one dashboard
probe grips were inactive; a later controls-only probe delivered repeated right probe grips were inactive; a later controls-only probe delivered repeated right
X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions
@@ -384,7 +442,7 @@ second squeeze **down** within 350 ms starts capture; hold as long as needed
action explicitly begins on down and ends on up. On tracking-pose invalidity, action explicitly begins on down and ends on up. On tracking-pose invalidity,
action inactivity or overlay focus loss, a held capture emits Cancel, and action inactivity or overlay focus loss, a held capture emits Cancel, and
reconnection requires a neutral observation before any new press. PTT and left reconnection requires a neutral observation before any new press. PTT and left
Enter require an enabled panel; clickable Record remains available for retry Type + Enter require an enabled panel; clickable Record remains available for retry
after an error and Cancel is always available. The action set defaults to normal after an error and Cancel is always available. The action set defaults to normal
priority; the experimental config request is described above. Neither priority priority; the experimental config request is described above. Neither priority
guarantees delivery while a game or dashboard owns input. guarantees delivery while a game or dashboard owns input.
+89 -26
View File
@@ -11,53 +11,71 @@ build with `FRAMEYAP_NATIVE=ON`, and stage this layout (regular files, no links)
``` ```
bin/frameyap bin/frameyap
lib/* # compatible bundled native libraries bin/install.sh # managed installer helper when required by layout
assets/actions.json # and adjacent controller binding JSON lib/* # explicitly supplied compatible native libraries
assets/actions.json # plus binding JSON and backends/redux.json
fonts/font.ttf fonts/font.ttf
python/frameyap/*.py python/frameyap/*.py
scripts/model-status.py # offline CLI verifier; also backend-service.py, fetch-model.py
licenses/THIRD_PARTY_NOTICES.txt licenses/THIRD_PARTY_NOTICES.txt
model/* # optional pinned public weights + attribution model/* # optional pinned public weights + attribution
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
``` ```
For the current **external-runtime** package, `scripts/stage-native-poc.py --help` The stage copies the self-contained installer into `bin/` and CMake installs
documents explicit inputs. It invokes `cmake --install` on an existing native build, `scripts/model-status.py` alongside the backend manifests. The package allowlist
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does permits `scripts/`; installed `frameyap --list-models` expects the verifier at
`../scripts/model-status.py`. **No actual staged archive has been audited/tested
from a clean account for publication**; exercise the entire producer pipeline
and the installed CLI before treating the payload layout as release-ready.
For the current **external-runtime** package, `scripts/stage-native.py --help`
documents explicit inputs. The old `scripts/stage-native-poc.py` is retained as
a deprecated migration wrapper for that command, not the documented or shipped
staging interface. The stage invokes `cmake --install` on an existing native
build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
not build, download, run the app, or copy an ASR runtime. The native not build, download, run the app, or copy an ASR runtime. The native
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType, app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
libstdc++ and glibc; audit `ldd` on the installed binary. libstdc++ and glibc. Audit the actual staged ARM64 binaries' `NEEDED`,
`GLIBC_*`/`GLIBCXX_*` symbol versions, ELF interpreter and notices; then test
on a clean target. No compatible libc floor is yet established.
SDL/OpenVR resolve inside its own `lib/`, not a producer SDL/OpenVR resolve inside its own `lib/`, not a producer
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux. prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
```sh ```sh
python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \ python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \
--version 2026-09-24T162712Z-g417f81c --arch linux-aarch64 \ --version 0.1.202609241627 --arch linux-aarch64 \
--model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime --model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime
``` ```
Use the actual binary's UTC build stamp (`frameyap --version`) for the archive Use the actual binary's numeric `MAJOR.MINOR.YYYYMMDDHHMM` UTC version
tag, not this illustrative timestamp. CMake generates `YYYY-MM-DDTHHMMSSZ` (currently `0.1`), not this illustrative value, for `--version` and filename;
at configuration time (plus `-gSHORTSHA` for a Git checkout and `-dirty` for tag a vetted clean tree as `vVERSION`. CMake generates this stamp at
uncommitted tracked changes); producers may pin `-DFRAMEYAP_VERSION=...` to configuration time (`SOURCE_DATE_EPOCH` may supply it);
embed a vetted release stamp. A timestamp distinguishes same-day archives; `-DFRAMEYAP_VERSION=0.1.YYYYMMDDHHMM` can pin it. Development `--version`
the installer still refuses a reused tag whose contents have changed. Historic may print a *separate* `git HASH` line, with `(uncommitted changes)` only if
`v0.1.0-poc*` local artifacts remain valid for reinstall/rollback. dirty: that line is not part of the version, tag or archive name. The installer
rejects a reused tag with different contents; historic local versions may
remain selectable for rollback, not as new releases.
`--external-runtime` refuses a runtime directory and records `--external-runtime` refuses a runtime directory and records
`runtime: external-authorized-python` in `release.json`. The installer explicitly `runtime: external-authorized-python` in `release.json`. The installer explicitly
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
in the archive. Staging validation is not an inference test. in the archive. Staging validation is not an inference test.
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing
plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal, `HASH FILENAME`. Archive extraction rejects traversal,
links/special files, duplicate members, oversized metadata/payloads and invalid links/special files, duplicate members, oversized metadata/payloads and invalid
layout. Checksums detect corruption, not a malicious/compromised publisher; layout. Checksums detect corruption, not a malicious/compromised publisher;
authenticate release metadata independently. No packaging/installation model fetch. authenticate release metadata independently. No packaging/installation model fetch.
## Consumer ## Consumer
Bootstrap: Linux ARM64/glibc, Python 3.12+, curl, sha256sum and tar. **No compiler, Bootstrap for a binary archive: Linux ARM64/glibc, Python 3.12+ (installer
sudo, Steam store AppID or engine checkout.** Download/inspect a pinned installer bootstrap, **not** the inference runtime), curl for network release downloads,
sha256sum and tar for archive handling. **No compiler, sudo, Steam store AppID
or engine checkout in binary mode.** No minimum glibc floor has been certified;
preflight alone cannot guarantee compatibility. Download/inspect a pinned installer
before running it. Current local artifact route: before running it. Current local artifact route:
```sh ```sh
@@ -65,17 +83,62 @@ sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
--sha256 64_HEX_DIGIT_HASH --version VERSION --sha256 64_HEX_DIGIT_HASH --version VERSION
``` ```
After an actual vetted release exists, `sh install.sh --version TAG` retrieves After an actual vetted release exists, use a real numeric version (for example,
that GitHub release and its versioned checksum; no `latest` or moving-branch `sh install.sh --mode binary --version 0.1.202609241627 --yes`); the installer
will retrieve tag `v0.1.202609241627` and its versioned checksum; no `latest` or moving-branch
lookup. A pipe invocation is supported, never prompts on stdin, and must also lookup. A pipe invocation is supported, never prompts on stdin, and must also
pin a real published tag. **There is no functional public download command yet.** pin a real published tag. **There is no functional public download command yet.**
`--without-model` omits bundled model files from staging, never deletes a current `--without-model` omits any model files in the selected archive, never deletes
model on rerun, and records the choice. Same digest/version/choice is idempotent an existing current model on rerun, and records the choice. Same
and repairs missing managed wrappers. Different digest or model choice for the digest/version/choice is idempotent and repairs missing managed wrappers. Different digest or model choice for the
same version is refused. External model provisioning is always deliberate. same version is refused. External model provisioning is always deliberate: the
local installer accepts `sh install.sh --install-model --backend redux --yes`
(optional `--model-dir /absolute/path`) to explicitly download and verify files from the
installed pinned manifest, not an ASR runtime. Inspect the model/size first
with `sh install.sh --install-model --backend redux --print-plan --json`.
`--expected-manifest-sha256 HASH` additionally binds consent to the exact raw
installed `redux.json` bytes: under the model lock a mismatch fails **before**
a model directory is created or any network request. In the locally implemented
Models UI, Install displays source, rounded size, license text, attribution and
the fingerprint; only the second Confirm Install click passes that fingerprint
through the backend helper to the installer. Installation alone does not
provision a CPU Python runtime; no in-panel flow has been accepted on Frame.
In the source tree, `python3 scripts/model-status.py --list-models` or
`--check-model redux --model-dir /absolute/model` hash-checks local files;
the native `frameyap --list-models` / `--check-model` entry points use the
adjacent installed verifier script; verify this in a staged archive before release. Manifest schema,
source, size/hash and attribution live in `assets/backends/redux.json` and are
validated by `python/frameyap/model_files.py`.
The installed launcher defaults to `--run`. For a native-only package, supply For an explicit local **source** install, e.g.:
```sh
sh install.sh --mode source --source /absolute/source --openvr-root /absolute/sdk \
--openvr-library /absolute/libopenvr_api.so --openvr-license /absolute/openvr/LICENSE \
--sdl-library /absolute/libSDL3.so.0 --sdl-license /absolute/sdl/LICENSE \
--version 0.1.202609241627
```
It checks tools and native dependencies, builds/stages/packages in a private
workspace, then installs the result. Unlike binary mode, this requires a C++
compiler, CMake, SDK, SDL3, Wayland/scanner, libxcb, FreeType, Vulkan development
files and producer-supplied
licenses; it still does not install Python ASR packages. `--print-plan` performs
a read-only plan, `--json` gives machine-readable results/errors (model installs
also stream file events), and `--yes` authorizes network downloads. Bare TTY
invocation can guide choices and prints equivalent flags; non-TTY runs require
explicit arguments and never prompt. `--autolaunch`/`--no-autolaunch` are explicit
OpenVR registration choices, off by default; do not pass either during an inert
install if a running SteamVR session must remain untouched.
The installed launcher defaults to `--run` and forwards explicit run flags
(including `--backend ID`, `--model-store /absolute/store` and
`--manifest-dir /absolute/manifests`) to the binary. These overrides select
local metadata/model paths, not a runtime download; the latter two require
absolute paths without dot segments. `--backend` selects for that run unless
changed in the panel; a saved `config.json` backend is otherwise used. For a
native-only package, supply
`FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`, `FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`,
or override `--python`/`--model` on an explicit `--run`. For menu launches, create or override `--python`/`--model` on an explicit `--run`. For menu launches, create
`$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`): `$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`):
@@ -157,7 +220,7 @@ without configured model/runtime cannot transcribe; it should show **Unavailable
(possibly after a brief Warming transition), rather than record or infer. (possibly after a brief Warming transition), rather than record or infer.
Only perform this check on an unconfigured native-only installation: verify Only perform this check on an unconfigured native-only installation: verify
that `current/runtime/bin/python3` and `current/model` are absent and no that `current/runtime/bin/python3` and `current/model` are absent and no
user-local paths are configured; do not press Record, Insert or Enter. user-local paths are configured; do not press Record, Type or Type + Enter.
The installer also provides a desktop entry; where Steam's UI supports adding a The installer also provides a desktop entry; where Steam's UI supports adding a
non-Steam app, the user may select that entry or browse to the installed launcher. non-Steam app, the user may select that entry or browse to the installed launcher.
Shortcut discovery/persistence after a normal restart is not yet verified. Do Shortcut discovery/persistence after a normal restart is not yet verified. Do
+82 -74
View File
@@ -1,91 +1,99 @@
# Dependency provenance and release boundary # Dependency provenance and release boundary
FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project FrameYap's original code is [MIT licensed](../LICENSE). This does not relicense
owner. This does not relicense external models, fonts, protocols or runtimes. models, fonts, protocols, native libraries or Python wheels. There is no dependency
There is no dependency on another application's checkout, assets or environment. on another application's checkout, assets or environment. **This is an upstream
license inventory, not an audit of a particular release binary. No public release
has been published.** Before shipping any archive, inspect the actual staged
files, their transitive dependencies and notices, and test on a clean supported
host. An external dependency does not make the current archive self-contained.
## Included source ## Included source/assets
`protocol/gamescope-input-method.xml` is the unmodified public Gamescope - `protocol/gamescope-input-method.xml`: unmodified public Gamescope **3.16.28**
**3.16.28** protocol, downloaded from protocol, <https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
<https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>. SHA-256 `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`;
SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`. embedded permissive copyright/license notice preserved. Generated bindings are
Its embedded permissive copyright/license notice is preserved. Generated bindings build outputs; this private Gamescope extension needs rechecking after updates.
are build outputs, not hand-written wire encoding. The private protocol may change - `assets/fonts/Inconsolata-Regular.ttf`: unchanged Inconsolata Regular from
with SteamOS; compatibility must be rechecked. <https://github.com/googlefonts/Inconsolata>, copyright 2006 The Inconsolata
Project Authors, **SIL OFL 1.1**. SHA-256
`e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
`assets/fonts/OFL-Inconsolata.txt` retains the license. The font remains OFL,
not MIT, and is not sold by itself. An override font requires its own license.
- Frame controller bindings and the mint-to-blue CPU-rasterized panel were
authored here, using public profile names; no SteamVR driver artwork, MSDF
atlas, other app renderer or protected kernels were copied.
Frame controller bindings were authored here using the observed public input Native staging uses `scripts/stage-native.py`: it includes the chosen font/license
profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code, and copied SDL3 and OpenVR notices in `licenses/THIRD_PARTY_NOTICES.txt`.
images, protected kernels or another application's assets were extracted for these bindings. It does **not** bundle ASR. A4 is not closed for release: inspect the final
notice file, especially FreeType attribution, plus the selected native binary
and any bundled wheel/licenses before publishing.
## Bundled font and UI reference ## Native build/runtime inventory
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular. | Component | Upstream license / evidence | Current packaging boundary / action |
SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`. | --- | --- | --- |
Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**, | Valve OpenVR SDK 2.15.6 | BSD-3-Clause-style license, SDK LICENSE. | Staging copies its loader and explicitly supplied license; verify chosen binary and transitive closure. |
retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace | SDL3 (device trial 3.2.16) | zlib, local `/usr/share/licenses/sdl3/LICENSE`. | Staging copies explicit library and license. Confirm exact build options/version and its transitive libraries. |
normalized; license text unchanged). The reviewed OFL permits | Wayland client + scanner | MIT/Expat-style, local `/usr/share/licenses/wayland/COPYING`. | Client is linked from system; scanner is build-time. If shipped, include copyright/license and audit closure. |
bundling and redistribution with its copyright/license notice; the font remains | libxcb | MIT-style with name-use restriction, local `/usr/share/licenses/libxcb/COPYING`. | Xwayland focus guard uses client library at runtime; not bundled by current native stage. Audit exact binary. |
OFL, not MIT, and is not sold by itself. Upstream: <https://github.com/googlefonts/Inconsolata>. | FreeType 2 | **FreeType Project License (FTL) selected** for this project, local `/usr/share/licenses/freetype2/FTL.TXT`; upstream also offers a GPL option. | Current stage uses system library, not bundled. Credit FreeType Team for use; if distributing its binary, meet FTL binary disclaimer/notice obligations and review the precise build. Do not silently substitute GPL terms. |
The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies | Vulkan loader, driver, system graphics dependencies | Loader/driver licenses vary by build and vendor. | Current stage depends on system Vulkan loader/driver; no GPU runtime is bundled. Audit the chosen loader if ever bundled. |
were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph. | Compiler runtime (`libstdc++`, `libgcc_s` when used) | GCC libraries: GPL with **GCC Runtime Library Exception** in upstream distribution; local `/usr/share/licenses/libstdc++/RUNTIME.LIBRARY.EXCEPTION` and `libgcc/...` are exception texts, not a full installed release audit. | Current stage relies on system runtime. Audit dynamic linkage, C++ ABI/`GLIBCXX_*` and exception coverage for *any* bundled compiler libraries; include corresponding complete notices/source obligations as applicable. |
| glibc/loader | GNU LGPL-2.1-or-later for core GNU C Library, with component-specific exceptions and other licenses to inspect. | Current stage relies on system libc/loader. Audit exact target binary symbol versions (`GLIBC_*`), ELF interpreter and its transitive closure; no minimum glibc/`GLIBCXX`/kernel floor is certified here. |
The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted The historical Frame CPU trial used Torch **2.8.0+cpu** on a host with **glibc
selection) is implemented independently on a single CPU RGBA surface. 2.39**. Those are *observed trial versions*, **not** minimum compatible versions
CMake installs the font and OFL with assets; native staging defaults to that for FrameYap, Python wheels or a future released artifact. `ldd` on a developer
font, places the launcher copy at `fonts/font.ttf`, and includes its license in machine alone is not sufficient: inspect the staged ARM64 binaries with `readelf`
`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license. (`NEEDED`, ELF interpreter, symbol-version requirements), `ldd` on a trusted
clean target, actual bundled libraries and notices, and test the final archive on
## Explicit native build inputs (not vendored) a clean supported Frame. Check the chosen compiler, CPU instruction/kernel,
FreeType/Wayland/XCB/SDL/OpenVR/Vulkan ABI, Python/native wheels and licenses.
- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015; Do not invent a libc floor from a build host's version.
retain its LICENSE with redistributed loader binaries.
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
- Wayland client and scanner: retain upstream MIT-style notices.
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
Xwayland focus observer. Include it in native runtime dependency checks;
no X server is started by normal operation.
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
- Optional font override: the earlier device check used system Hack Regular.
A custom release font must include its own license and assessed glyph coverage;
new builds default to the bundled Inconsolata described above.
- Compiler runtime, libc minimum and transitive shared libraries require a release
dependency audit. Passing a developer build is not a portable-runtime guarantee.
## Redux weights and inference runtime ## Redux weights and inference runtime
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision Model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution: `fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**.
Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3. Attribution: Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT
No modifications to the supplied weights are made. Exact model/config/tokenizer 0.6B v3. No modifications to the supplied weights are made. Pinned model/config/
sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`. tokenizer/card sizes and SHA-256 hashes, source and attribution are recorded in
`fetch-model.py` fetches and retains the original model card alongside the files. `assets/backends/redux.json` (schema validated by `python/frameyap/model_files.py`),
No weights are committed to this repository. not hardcoded in `model_files.py`. `fetch-model.py` fetches and retains the model
card alongside the weights on **explicit** request; neither build/tests nor a
normal app launch downloads them. `scripts/model-status.py` and the native
`--list-models` / `--check-model` interface inspect/hashes local files offline.
No weights are committed to Git.
Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels` The current Redux worker uses separately provisioned `moondream` Python and its
dependencies, installed by the user from PyPI into their own environment. Upstream's `kestrel` / `kestrel-kernels` dependencies, not a bundled runtime. Kestrel's
Kestrel README states: "Local inference is free and requires no API key" upstream README says “Local inference is free and requires no API key”
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs a Moondream (<https://github.com/m87-labs/kestrel>); finetuned-model inference needs an API
API key and is not used here. FrameYap's release archives do not vendor or bundle key and is **not** this path. The native-only installer **does not run pip**,
these packages; they are installed from PyPI onto the user's machine, either by the provision an interpreter, or make a native-only artifact able to transcribe on
user or by the installer at the user's request. its own. A person supplying a Python environment must review/authorize its
exact dependency closure. No bundled-ASR artifact is licensed/approved by this
inventory.
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need | Python/native package | Upstream license inventory (not a wheel audit) | Release action |
their own notice/license inventory before a bundled release. | --- | --- | --- |
| PyTorch / Torch CPU | PyTorch project: BSD-3-Clause; third-party components/wheels carry additional notices and dependencies. | No Torch wheels bundled. Pin CPU-only ARM64 wheel if building a distribution; audit its `LICENSE`, `NOTICE`, `third_party`/wheel contents and `NEEDED`/symbol versions. |
| NumPy | NumPy core: BSD-3-Clause; dependencies/embedded algorithms carry additional BSD, MIT, 0BSD, zlib, CC0 and other notices depending on wheel. Local `python-numpy` 2.5.3 package metadata (`/usr/lib/python3.14/site-packages/numpy-2.5.3.dist-info/METADATA`) declares `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` with many `License-File` entries (different from the historical trial environment). | Not bundled. Keep *all* license files and inspect the exact target wheel, BLAS/OpenBLAS and runtime closure before redistribution. |
| Hugging Face `tokenizers` | Upstream `huggingface/tokenizers` is Apache-2.0; native/Rust crate dependencies need separate inventory. | Not bundled. Confirm actual installed wheel version, package LICENSE/NOTICE and transitive Rust/native code if ever distributed. |
| `moondream`, Kestrel/kernels/native, Python interpreter | Distinct packages with distinct license files and native transitive code; Kestrel local-use statement is not a blanket redistribution license. | None bundled. Exact versions, permissions, wheel notices, CPython build and native linkage require review before any runtime bundle. |
## CPU trial dependency choice Development CPU trial *interfaces*, not package-floor promises: moondream
**2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, native **0.1.8**, Python
**3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified moondream install
initially resolved CUDA-enabled Torch and NVIDIA wheels; the owned trial venv
was corrected before measurement (`torch.version.cuda is None`). Do **not**
repeat unconstrained `pip install moondream` as a CPU setup recipe.
Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, A standalone CPython 3.12.14 ARM64 distribution was downloaded for packaging
native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified research but **not bundled**: <https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels;
these were replaced/removed from the owned venv before measurement. The measured
Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
`pip install moondream` as a CPU setup recipe.
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
was downloaded but not bundled with any ASR runtime:
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`, `cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`. SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
Its included licenses and compatible native dependencies still require review Its own licenses and native compatibility need review before any release.
before any release. No public prebuilt FrameYap release has been published.
+38 -15
View File
@@ -1,8 +1,10 @@
# Offline Redux worker adapter (component, not an installed product) # Offline worker adapter and backend dispatcher
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model, `src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
threads=2, advanced_debug=false)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for threads=2, advanced_debug=false)` explicitly for the legacy Redux script, or
one `WorkerReply` (text or privacy-safe per-request error). One request at a time; supply the optional `backend`, `manifest_dir`, `root` arguments for the manifest
dispatcher. Poll until `ready()`, then `submit(id, pcm)` and poll for one
`WorkerReply` (text or privacy-safe per-request error). One request at a time;
no queue, no capture and no input injection. `stop()` discards pending audio, no queue, no capture and no input injection. `stop()` discards pending audio,
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL), terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
and is safe to repeat. Destruction stops it. `start()` returns without waiting and is safe to repeat. Destruction stops it. `start()` returns without waiting
@@ -36,13 +38,19 @@ requests reading the fixed clip; `Y` means ready; `F` means load failure
missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8 missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8
text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096 text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
or oversized frame stops the worker. Warmup deadline is 120 s, transcription or oversized frame stops the worker. A correlated `E` reply or an invalid
deadline 60 s; `poll()` must be called regularly to enforce deadlines. It transcript is a **request-level** failure: the Controller drops that clip,
never initializes a headset or starts a recording. There is no auto restart. keeps the ready model loaded and allows Record to retry. A broken protocol,
worker crash or explicit in-flight cancellation is different and can require a
reload. Warmup deadline is 120 s, transcription deadline 60 s; `poll()` must
be called regularly to enforce deadlines. It never initializes a headset or
starts a recording. There is no auto restart after a process failure.
`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI Parakeet Redux is the **model** (`moondream/parakeet-redux`); `moondream` is
startup, with HF/Transformers/Datasets offline variables and bounded native its Python inference package, not a second model or cloud endpoint. Kestrel is
thread-pool variables set before import. It uses a native dependency of that local runtime. `python/frameyap/worker.py` lazily
imports `moondream` only after explicit CLI startup, with HF/Transformers/Datasets
offline variables and bounded native thread-pool variables set before import. It uses
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>, `md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
device="cpu", cpu_threads=threads)` and persistent device="cpu", cpu_threads=threads)` and persistent
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`. `transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
@@ -51,16 +59,31 @@ moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
preinstalled local weights from revision preinstalled local weights from revision
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory `fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
against `model_files.py` before importing model libraries. Protocol stdout is against `assets/backends/redux.json` via the shared, offline
isolated at the file-descriptor level from third-party diagnostics. Thread limits `python/frameyap/model_files.py` schema/verifier before importing model libraries.
cover Torch interop/native pools and CUDA is not selected. Offline environment `frameyap --list-models` and `frameyap --check-model redux --model-dir /absolute/model`
(or `scripts/model-status.py`) read local manifests and optionally hash local
files; they never load the worker or download weights. For native `--run`,
`--backend ID`, `--model-store /absolute/store`, and
`--manifest-dir /absolute/manifests` are wired through the installed launcher
as explicit overrides; default selection is saved in `config.json` (`redux`). The dispatcher
`python/frameyap/backend_worker.py` validates a selected manifest and hashes
its local model, resolves an in-release relative Python/executable launcher
without a shell, then supervises one child over `frameyap-worker-v1` (ready,
correlated request/reply and failure frames). The launcher must accept
`{model_dir}` and `{clip_dir}` (optional `{threads}`); matching manifest and
pinned weights are **not** a runtime or automatically trusted new backend.
Only Redux is supplied today. Protocol stdout is isolated at the file-descriptor
level from third-party diagnostics. Thread limits cover Torch interop/native
pools and CUDA is not selected. Offline environment
flags do not prove every third-party internal is unable to access a network. flags do not prove every third-party internal is unable to access a network.
Runtime/build/tests perform no downloads; the separate explicit setup utility Runtime/build/tests perform no downloads; the separate explicit setup utility
`scripts/fetch-model.py` can provision the public pinned weights. `scripts/fetch-model.py` can provision the public pinned weights.
Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public Current native-only archives do not bundle or pip-install the runtime; see
runtime bundle has been released. Limited ARM64 measurements were taken during development; [third-party notes](third-party.md). No public runtime bundle has been released.
they are not a claim of complete headset acceptance. Limited ARM64 measurements were taken during development; they are not a claim
of complete headset acceptance.
## Advanced debugging ## Advanced debugging