docs: align release scope and track remaining device acceptance gates

This commit is contained in:
baketnk committed 2026-09-24 23:03:50 -04:00
1 parent 4498ef5d04
commit 1321ca8316
9 files changed
+622 -527

No files matched your search

+55 -25
View File
@@ -11,8 +11,9 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
mode while FrameYap is visible; it may affect games, and is not an input override.
- Remappable SteamVR actions. The default Steam Frame binding maps right X
(hold to record, release to transcribe) to the existing PTT action using the
observed `frame_controller` profile. Right B cancels, A inserts + space, and Y
inserts pending text + Enter (or Enter only with no preview). The Bindings button
observed `frame_controller` profile. Right B cancels, A requests Type (text +
space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay
control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button
requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
actions were inactive in the observed dashboard check; do not rely on them.
If left grip becomes active, two short taps request explicit Enter.
@@ -22,27 +23,41 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
20 second maximum. In the native app the microphone stream opens after model
warm-up, stays running between utterances, and discards idle samples; PTT does
not open/pause/close the device. Quit, worker restart, device failure or capture
failure closes it. This avoids repeated capture-device transitions but does not
promise glitch-free playback on every audio stack. Other apps may still
not open/pause/close the device **by default**. Settings → Close mic when idle
(`"close_mic_when_idle": true`, default **false**) closes it between clips and
reopens on PTT; repeated transitions caused an audio spike on Frame and may
add latency or clip the first syllable. Quit, worker restart, device failure
or capture failure also closes it. Keeping it open does not promise glitch-free
playback on every audio stack. Other apps may still
transmit your voice: this app does **not** mute VRChat or any other app.
- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed
backend restarts the worker and invalidates pending clip/review/focus authority;
a missing or invalid model disables recording. A separate Install then Confirm
Install exposes pinned source/size/license/attribution and exact manifest-byte
SHA-256 before installer handoff. These are local implementations, not a tested
installed UI or an approved additional inference engine. Only Redux is supplied.
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
Model imports are lazy and loading is offline. Normal repeats, request-local
transcription failures and microphone failures retain the loaded model;
Model imports are lazy and loading is offline. A correlated bad transcript/
`E` reply is a request-level failure: Record can retry without dropping the
ready model. Normal repeats and microphone failures retain the loaded model;
microphone device failure releases the stream for explicit retry. A cancelled
in-flight request or broken worker protocol may require reloading. No
cloud/desktop fallback.
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
handling, UTF-8/control validation and explicit Submit action for Enter.
Insert ensures a trailing space without doubling an existing one. Enter
first inserts pending review, releases the text lease, then acquires a fresh
lease for Submit. Failed/uncertain text never proceeds to Submit; failed
Submit acquisition never replays text. A full 4096-byte transcript without
room for a space is preserved with an error, never silently truncated.
- Idempotent user-local release-archive installer: SHA-256, safe extraction,
atomic current-version selection, retained rollback, runtime/install lock,
foreign-file refusal and explicit unregister-before-uninstall acknowledgement.
handling, UTF-8/control validation and explicit Type + Enter action.
Type ensures a trailing space without doubling an existing one. Type + Enter
first types pending review, releases the text lease, then acquires a fresh
lease for Enter. Failed/uncertain text never proceeds to Enter; failed
Enter acquisition never replays text. If a validated transcript fills the
4096-byte bound and lacks a trailing space, Type preserves all its bytes and
queues it **without** the usual space; it does not signal a separate error.
Destination consumption and repeated-delivery behavior remain unaccepted.
- User-local installer with checked binary-archive or explicitly provisioned
source-build mode, safe extraction, atomic current-version selection, retained
rollback, runtime/install lock, foreign-file refusal and explicit
unregister-before-uninstall acknowledgement. Source mode needs a local
compiler, SDK, libraries and license inputs; no runtime is pip-installed.
## Deliberately not claimed
@@ -63,15 +78,26 @@ fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip binding
are not guaranteed globally active in every scene/dashboard state, and the app
never enables SteamVR's experimental overlay overrides on your behalf.
This is a compact prototype panel, not yet the proposed polished miniature status
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
impact and target application compatibility require further headset work.
This is one compact panel, not a separate miniature status chip. Front-prefix
loss on repeated submissions (P1) is under separate investigation, **not fixed**
by the chooser/worker documentation or any speculative delivery change. Font
coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact
and target application compatibility require further headset work.
## Inference runtime
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference
runs through the `moondream` Python package and its Kestrel runtime, which you
install in your own environment; builds/tests never fetch it. See
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Pinned
file sizes/hashes and attribution live in `assets/backends/redux.json`; offline
`--list-models`/`--check-model redux --model-dir /absolute/model` (or
`scripts/model-status.py`) verify without inference or downloads. Manifest
schema/verification are in `python/frameyap/model_files.py`. The local generic
dispatcher resolves a manifest's in-release Python/executable launcher and
checks request/reply correlation; a new manifest still needs its own licensed,
compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and
`--manifest-dir /absolute/manifests` are wired through
the installed launcher as an explicit override, not a provisioning command.
Inference uses the `moondream` Python package and Kestrel runtime, separately
provisioned in your own environment; builds/tests/installer do not pip-install them. See
[third-party notes](third-party.md).
## Developer native build
@@ -142,12 +168,16 @@ kestrel 0.8.0) and local weights:
```
Use a disposable text destination first. `--run` loads the model but does not
record until an explicit recording control. Click Insert only after focusing your
record until an explicit recording control. Click Type only after focusing your
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
and terminates only the owned worker. Installer upgrades refuse an active app.
The worker API is intentionally small: a fork can replace the worker implementation
or add its own model/API integration without changing overlay and delivery code.
The `Controller` receives injected `ControllerAudio`, `ControllerWorker`,
`ControllerFocus` and delivery factory interfaces; hardware-free fakes can test
state, retry, authorization and mic lifetime without initializing OpenVR or
recording speech. The worker API is intentionally small: a fork can replace the
worker implementation or add its own model/API integration without changing
overlay and delivery code.
The default product remains local-only Redux; extending a fork does not authorize
sending existing users' audio to a service.
+61 -39
View File
@@ -1,11 +1,16 @@
# FrameYap native dictation overlay — proposal
# FrameYap native dictation overlay — design and remaining goals
Standalone project design. This document is the full target design, not a blanket
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
Redux adapter, review-first Gamescope insertion and a user-local archive installer
with explicit source-build option. Redux's pinned manifest, offline status CLI,
manifest-driven dispatcher and in-panel Models selection/confirmation are
implemented locally. The installed chooser/installer handoff is **not yet
end-to-end verified on Frame**; the native-only archive needs an external CPU
runtime for speech.
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md).
accepted for live automatic typing on Frame. A separate status chip and full hardware
acceptance remain possible future work. See [build and scope](build.md) and [third-party notes](third-party.md).
Future work is tracked in [TODO.md](../TODO.md).
## Recommendation
@@ -17,7 +22,9 @@ as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene
renderer, avatar, desktop capture or root service is needed in the primary path.
A first-class product goal is a **one-command GitHub install without a Steam store
AppID**. Package a prebuilt native executable and isolated CPU runtime; use a normal
AppID**. Package a prebuilt native executable; the current native-only archive requires a
separately supplied, compatible CPU runtime (no automatic pip install). A future
isolated runtime bundle requires its own license and compatibility audit. Use a normal
OpenVR application key for registration, not Steamworks. Installation must remain
user-local with opt-in autolaunch. See [installation design](install-design.md).
@@ -31,12 +38,14 @@ one persistent local Parakeet Redux CPU worker
correlated literal transcript → focus/delivery policy
↓
Gamescope IME set_string + commit → focused Frame application
↘ overlay preview / error / explicit Insert when delivery is unsafe
↘ overlay preview / error / explicit Type when delivery is unsafe
```
The CPU worker is local process isolation, not remote inference or a service
framework. Implement a small transport owned by this repository. Keep the model
loaded between utterances; do not spawn Python/load 178 MB for every release.
framework. The locally implemented manifest dispatcher uses a small bounded
transport owned by this repository; only Redux inference is shipped. Keep the
model loaded between utterances; do not spawn Python/load 178 MB for every release.
A correlated bad transcript is a request-level failure, not a model unload.
## Minimal interaction
@@ -53,27 +62,29 @@ armed remains a possible refinement, not a second implemented overlay. See
Recording… 00:04 [ Cancel ]
"The recognized text appears here."
[ Insert + space ] [ Discard ] [ Insert + Enter — explicit ]
[ Type + space ] [ Discard ] [ Type + Enter — explicit ]
```
- States: disabled, warming, ready, recording, transcribing, review, inserted,
unavailable/error. Recording uses visible icon + text, not color alone.
- States: warming, ready, recording, transcribing, review, input queued,
unavailable/error. Input queued is not an application receipt. Recording uses
visible icon + text, not color alone.
- Default Frame bindings: hold right X to speak, release to finish; B cancels,
A inserts with a trailing space, Y opens/cycles the quick-chat selection.
Overlay Submit or left-grip double-tap submits the selected literal + Enter,
or pending review + Enter, or Enter alone if neither is present.
The alternate right-grip tap-then-hold gesture and left-grip double-tap Submit
A types with a trailing space, Y opens/cycles the Quick phrases selection.
Overlay Type + Enter or left-grip double-tap submits the selected literal + Enter,
or pending review (normally + space) then Enter, or Enter alone if neither is present.
The alternate right-grip tap-then-hold gesture and left-grip double-tap Type + Enter
remain available. All are remappable through the Bindings button's SteamVR editor.
A click-to-start/stop overlay button provides
a binding-independent alternative. Bound recording to 20 seconds; discard
accidental taps (initial threshold: 200 ms).
- **Quick typing (opt-in, locally implemented):** insert on completion only when
- **Auto insert (opt-in, locally implemented):** type on completion only when
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
wait for Insert. Bring up review on uncertainty rather than silently losing a
wait for Type. Bring up review on uncertainty rather than silently losing a
transcript or typing into a new target. Live Frame acceptance remains open.
- Submit requires its own explicit control activation: insert pending review with
a trailing space, then queue Enter only if the text step succeeds. With no
review, it queues only Enter. Never interpret "submit", "delete" or other speech
- Type + Enter requires its own explicit control activation: type pending review
(normally with a trailing space), then queue Enter only if the text step succeeds.
At the 4096-byte bound, preserve the entire transcript without a suffix if no
space fits. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech
as commands. Transcription completion never auto-submits.
- No generic "undo last dictation" initially: another application's edits/cursor
cannot be reliably rolled back by a guessed number of backspaces.
@@ -109,7 +120,7 @@ license-reviewed extraction, never a runtime path into another project's checkou
### Controller bindings
Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the
Expose PTT, cancel and explicit Type/Type + Enter as named SteamVR actions; let the
user bind them. Do not assume a scene app's left-bumper mapping works globally or
silently takes a game's button away. Check action activity and neutral rearm.
@@ -143,7 +154,7 @@ The server implements text through synthetic key events and a temporary keymap,
not a guaranteed rich-text/IME edit operation in every app. Verify actual target
toolkits and games. Respect singleton/unavailable handling and Steam-keyboard
coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe:
its inspected sample loop is byte-oriented and interprets newline as Submit.
its inspected sample loop is byte-oriented and interprets newline as Enter.
### 2. Explicit fallbacks, not a framework built up front
@@ -183,11 +194,11 @@ Gamescope exposes focus-display/window root properties, but their encoding and
relationship to seat focus need implementation-specific validation. Do not infer
that X display `:0` is always the destination, or that an X focus observation
identifies a native Wayland text field. For unobservable native Wayland focus,
require explicit review/Insert; do not advertise safe auto-targeting.
require explicit review/Type; do not advertise safe auto-targeting.
If focus changes, keep the result in review. A fresh Insert explicitly approves
If focus changes, keep the result in review. A fresh Type explicitly approves
the current destination and creates a new delivery authorization. Recheck again
at insertion. This minimizes stale delivery but does **not** eliminate a race
at typing. This minimizes stale delivery but does **not** eliminate a race
between the final check and global input processing; do not claim otherwise.
A Wayland roundtrip means compositor processing, not application consumption.
Report `input queued`, never `message sent`.
@@ -221,9 +232,11 @@ that tiny dataset; keep transcript visibility and a cheap retry.
Implement a small independent audio/worker adapter with explicit capture,
single-request bounds, owner-only runtime files, correlated replies and cancellation.
The worker should be implemented independently; do not link, vendor or import
another application's speech code. Avoid a generic provider framework: one
explicit Redux worker is enough for the first version.
The worker is independently implemented; do not link, vendor or import another
application's speech code. A narrow manifest dispatcher now selects a checked
local backend launcher; this is not a general cloud/provider framework and only
Redux is supplied with a runtime implementation. Other manifests require their
own audited offline engine, launcher and tests.
Suggested ownership, introduced only as implementation needs it:
@@ -243,12 +256,20 @@ processing timeout. No shell commands in IPC and no input authority in the worke
native kernel pools. Choose measured latency versus compositor contention,
not the desktop's thread count by habit. No real-time scheduling or permanent
CPU pinning initially; inspect runtime affinity behaviour during measurement.
- Explicit local model path and offline loading. Missing runtime/weights produces
an actionable error, not an unsolicited download/network fallback.
- Optional explicit enable/warm-up before first PTT; warming must not record audio.
Expose that lifecycle explicitly rather than warming on import or construction.
Otherwise display first-use loading honestly. Keep
model reuse after normal completion; cancellation may restart the owned worker.
- Explicit local model path and offline loading. Pinned files and attribution live
in `assets/backends/redux.json`; `python/frameyap/model_files.py` validates its
schema and hashes. The CLI's offline `--list-models`/`--check-model` checks
never start inference. Settings → Models selects/saves the manifest ID and
restarts the worker, invalidating any audio/review/delivery authorization.
Install requires a separate confirmation showing source, size, license,
attribution and exact-manifest SHA-256; only that click hands off to the
installer for pinned model files. Missing runtime/weights produces an
actionable error, not an unsolicited download/network fallback.
- Native `--run` explicitly starts offline verification then warms a verified
selected local model before PTT; warming must not record audio. Missing or
unverified files disable Record rather than prompting a background download.
Keep model reuse after normal completion and request-local transcript errors;
cancellation may restart the owned worker.
- Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded
tmpfs-backed clips; remove them on completion, error, cancellation and shutdown.
Avoid persistent audio/transcripts by default. Local IPC is not a network hop;
@@ -278,14 +299,14 @@ These are proposed implementation gates, **not completed acceptance**:
confirm dashboard/hand placement, input events, close/reopen and no scene-focus
takeover. Validate global PTT separately rather than blocking the clickable
prototype on experimental override support.
3. **Real dictation path:** microphone → local Redux → preview → explicit insert
into a disposable target; then enable quick typing after target tracking tests.
3. **Real dictation path:** microphone → local Redux → preview → explicit Type
into a disposable target; then validate opt-in Auto insert after target tracking tests.
Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate
replies, lost mic, worker crash and missing model without persisting speech.
4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game
text fields; Steam keyboard coexistence; native Wayland targets separately.
Test focus changes during capture/inference, rapid loss/regain, held modifiers,
explicit Insert retargeting, no hidden Enter, and no second delivery.
explicit Type retargeting, no hidden Enter, and no second delivery.
5. **In-headset acceptance:** readable feedback and comfortable PTT; measured
release-to-insert latency and compositor timing while an actual scene runs;
no noticeable sustained thermal/battery regression. Set numeric budgets after
@@ -301,8 +322,9 @@ without moving recognition off Frame. It is not part of this initial design.
The default hardware-free build needs only CMake and a C++20 compiler (Python
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
SDL3, FreeType and Wayland client/generated protocol bindings. A separately
authorized Python Redux environment is explicitly supplied at launch. Pin revisions
and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery.
authorized Python Redux environment is explicitly supplied at launch; the
native-only installer neither bundles it nor pip-installs one. Pin revisions
and review licenses for each introduced dependency. No automatic fetch/install in configure or normal tests; no external checkout discovery.
Hardware-free tests should cover state transitions, bounded PCM/transcripts,
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
+73 -99
View File
@@ -1,108 +1,82 @@
# Installation and distribution goal
# Installation goal and current boundary
**User goal:** install from GitHub with a `curl … | bash`-style command, without a
Steam store AppID. An idempotent archive installer and native-only local artifacts
are now implemented/tested; no public release is published. A bundled-ASR
experience is not yet offered. This document retains the target
design; see [current packaging](packaging.md) and [third-party notes](third-party.md).
Planned installer work (binary-or-source choice, flag-driven operation) is in
[TODO.md](../TODO.md).
Goal: a one-command, pinned GitHub release install for Steam Frame without a Steam
store AppID, sudo or end-user compiler. **No public archive or verified clean
install is published. Do not advertise a `curl | sh` command as functional.**
The local installer has binary-archive and explicitly provisioned source-build
modes, machine-readable plans/results and an attended TTY path. The current
native-only artifact does not include or pip-install an ASR runtime; it is not
a one-command voice-typing experience. See [packaging](packaging.md) for exact
flags and [third-party inventory](third-party.md) for open license/ABI audits.
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable
for the **proposed** Linux ARM64 glibc package format and has the expected basic
bootstrap utilities. It reports system Python, Git and uv, but none is required
for the intended bundled release. A read-only check on one Frame observed
Python 3.12.3 and Git, but not uv; availability may change. This script
is not an installer or a model/runtime compatibility test; it has no downloads,
registration, SteamVR initialization or persistent changes. No glibc minimum can
be certified until release artifacts are chosen and tested.
`scripts/install-preflight.sh` is a read-only Linux ARM64/glibc/bootstrap check;
`--source` adds toolchain/library checks. It does not download, install, register,
or certify model/runtime compatibility or a minimum libc version. Its report of
Python/Git/uv availability is not a runtime guarantee: `install.sh` currently
needs Python 3.12+ **for its bootstrap**, while native-only voice inference needs
a separately provisioned compatible CPU Python environment and pinned weights.
## Non-Steam overlay identity
## OpenVR identity
OpenVR overlay applications do not require a Steam store AppID or Steamworks.
The native executable initializes as `VRApplication_Overlay`. For discoverability
and optional autolaunch, register an OpenVR application manifest with a stable,
project-owned **string application key** (proposed: `local.frameyap.overlay`).
That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam
library shortcut should be necessary for the normal route.
`local.frameyap.overlay` is a string OpenVR application key, **not** a Steam
store AppID. The installer creates a manifest/desktop launcher user-locally but
does not register/launch the app by default. Explicit `frameyap --register
/absolute/manifest/path` uses the OpenVR registration API; registration alone
did not reveal a launcher in the first checked dashboard menu. On one Frame the
user opened the panel from the **Non-Steam** section and quit; shortcut discovery
and persistence after a normal restart are still unverified. Optional
`--autolaunch`/`--no-autolaunch` on the installer explicitly request OpenVR
registration/autolaunch choices; no SteamVR settings or sessions are changed
without that request. Unregister explicitly before uninstall. SteamVR must
already be available for registration; never start/restart it for installation.
The manifest identifies the installed executable; the installer/registration
helper should use `IVRApplications::AddApplicationManifest` and the corresponding
remove operation, not hand-edit Steam's internal JSON. Autolaunch uses the OpenVR
application setting only when explicitly requested. Validate the exact manifest,
launch behaviour, registration persistence and uninstall on native Frame before
claiming this route works end to end. Existing probes established overlay client
initialization, not manifest installation.
## Installation workflows
SteamVR/OpenVR must already be installed and usable. If registration needs a
running runtime, defer it to the first explicit launch rather than starting or
restarting SteamVR behind the user's back.
- **Binary mode** (no compiler): verify a locally supplied ARM64 archive digest,
or, after a vetted release actually exists, explicitly approve retrieval of
a pinned `v0.1.YYYYMMDDHHMM` GitHub tag and checksum. No moving `latest` tag.
`--mode binary --archive FILE --sha256 HASH --version 0.1.YYYYMMDDHHMM`
selects the local-artifact path.
- **Source mode**: requires explicit local source, SDK, SDL/OpenVR libraries and
their notices, CMake/C++20, native build dependencies and a version. It builds,
stages, packages and continues through local installation; it does **not**
provision ASR packages or bypass producer license obligations. See
[packaging](packaging.md) for all flags.
- **Model**: only an explicit `--install-model --backend redux --yes` fetches
pinned public files for the *already installed* backend. Inspect the read-only
`--print-plan --json` first; it includes model size, attribution and license
metadata from `assets/backends/redux.json`. `--expected-manifest-sha256 HASH`
binds consent to the exact installed manifest bytes and fails before model
directory creation/network if they changed. The local in-panel chooser shows
source, size, license text, attribution and manifest digest, then requires a
second **Confirm Install** click; the installed/native UI route still needs
clean-target and headset acceptance. `--without-model` permits an
archive install without bundled model files. Neither operation installs Torch,
moondream, Kestrel or an interpreter. Launch paths to an independently
authorized runtime/model can be set in `paths.conf`.
- **Noninteractive**: supply flags and `--yes` for network/model consent.
`--print-plan` is read-only; `--json` provides structured results/errors and
progress events for a model download. No prompt reads stdin in a pipe. An
empty TTY invocation offers a local menu and prints equivalent flags.
## Intended user experience
Installation is user-local under XDG data/config paths with a managed launcher,
retained rollback, SHA-256/path validation, foreign-file refusal and a lock
shared with the app. No OS package changes, udev rule, root service, Steam store
listing, unrelated application dependency, microphone recording, input injection
or automatic update daemon. Hashes detect accidental/unauthorized alteration
of a downloaded artifact but do not authenticate a compromised publisher;
release metadata needs independent trust. No installer operation silently runs
pip or launches inference. Native-only archives support overlay checks but need
an externally provisioned runtime/weights before voice typing. Selecting an
uninstalled backend does not authorize a download or supply its inference engine.
1. Run one documented command from the eventual GitHub repository/release.
2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and
explains/downloads the application, compatible CPU runtime and pinned model.
3. User-local installation provides a simple `frameyap` launcher, desktop
entry where supported, and an OpenVR manifest. No compiler, engine checkout,
Python dependency troubleshooting or separate ASR server for ordinary users.
4. The user explicitly launches/enables dictation. No installation-time microphone
recording, input injection, inference benchmark or overlay takeover.
## Gate before publishing the goal as fulfilled
Illustrative command shape only; `OWNER`, `REPO` and `VERSION` are placeholders:
```sh
curl --fail --silent --show-error --location \
https://raw.githubusercontent.com/OWNER/REPO/VERSION/install.sh | bash
```
Also document a download-inspect-run path for users who do not want to pipe remote
code into a shell. Pin a release/tag instead of executing a moving branch by default.
Offer explicit version selection and noninteractive flags; do not read interactive
confirmation from stdin while the installer itself is arriving through that pipe.
## Packaging boundary
- Prebuilt ARM64 executable plus a known-compatible, isolated CPU inference runtime;
no external application libraries/assets and no system Python modification.
- Model fetched during explicit installation/setup, with pinned revision/hash and
attribution. A documented `--without-model` option can defer the large download.
No surprise first-utterance downloads. Runtime components are installed from
their own package index rather than redistributed in our tarball.
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
`/dev/uinput` permission changes or modifications to unrelated launchers.
- No Steam store AppID, Steamworks SDK, root service or network ASR dependency.
- Third-party notices for everything we redistribute must be included with a release.
## Installer lifecycle and safety
- Detect architecture, libc and prerequisites first. Unsupported hosts fail with a
clear explanation; never install an x86 payload silently on ARM64.
- Download to a staging directory, check versioned SHA-256 manifests and archive
paths, then atomically select the completed version. HTTPS/checksums alone do not
authenticate a compromised publisher; use signed release metadata if provided.
- Keep configuration across upgrades; retain the previous version for rollback.
Refuse or defer replacement while this application's process is running rather
than killing arbitrary processes. Never touch SSH or unrelated sessions.
- Autostart is opt-in (`--autostart` or explicit settings); do not enable a systemd
service or SteamVR autolaunch by default. Do not enable overlay input overrides.
- Uninstall removes only owned launcher, manifest registration and install files;
model/config deletion is separately explicit. Preserve other SteamVR apps.
- No update daemon initially. A deliberate rerun/update command is sufficient.
## Acceptance before advertising one-command installation
- Clean supported Frame: install without sudo/compiler/engine checkout/store AppID;
launch overlay, load local Redux and type into an owned disposable target.
- Normal use after installation needs no network connection or desktop ASR host.
- Failed download/hash, unsupported architecture, low disk space and interrupted
upgrades leave a usable previous install or a cleanly reported failure.
- Reinstall, rollback and uninstall preserve unrelated data and SteamVR entries.
- Noninteractive piped invocation never hangs on stdin; inspection-first path works.
- Autolaunch remains off unless chosen; uninstall removes only our registration.
- Hardware-free installer tests use temporary homes, mocked runtime registration
and local fixture artifacts. Never exercise a real user's Steam configuration
in ordinary CI/CTest.
Vet the **exact** release closure/licenses, ARM64 symbol versions/loader,
model attribution and compatible CPU Python environment; establish a tested
libc/runtime floor, then publish and authenticate a checksummed archive from a
clean tag. On a clean supported Frame, install without a compiler/sudo/store ID,
load Redux, type into a disposable owned target and validate rollback/uninstall,
foreign-file failures, autolaunch off, and no unintended session changes. Separately
validate microphone → review → delivered input and headset comfort. Offline
installer fixture tests and a local native build are not those acceptance gates.
+89 -31
View File
@@ -2,10 +2,12 @@
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
Native `--run` wiring in `src/main.cpp` is implemented behind the explicit
`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful
compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a
normal build or test.
Native `--run` wiring in `src/main.cpp` is behind the explicit
`FRAMEYAP_NATIVE` build option. The Models chooser, offline verification and
explicit installer handoff are implemented locally, **not** an accepted Frame
install/voice-typing path or a published release. Neither hardware-free tests
nor a successful compile establish Frame input, visibility, comfort or text
delivery. Launching the runtime is explicit, never part of a normal build or test.
## Rendering and controls
@@ -49,7 +51,9 @@ This replaces the raw-upload rendering path; headset flicker acceptance still
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
The header shows local time and date instead of the former on-device/review
and current-mount labels. It updates when the displayed minute or date changes,
and current-mount labels. Settings explains Hold Quit (hold 0.9 seconds then
release) and Lasers anytime (system-wide lasers may affect games). The review
tab describes Type and Type + Enter. It updates when the displayed minute or date changes,
not every frame. Settings toggles 12/24-hour time and cycles date Off →
MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display;
mount choices remain in Settings.
@@ -58,23 +62,25 @@ The complete transcript preview is paginated by glyph width and four-line
height; Previous and Next navigate it without changing the source transcript.
Status fits on the single status line; the old bottom detail label is gone.
The footer remains available on all tabs: Record (labelled Stop while recording),
Cancel, Insert, Submit, Hold Quit. Hold Quit needs a 900 ms press and release on
Cancel, Type, Type + Enter, Hold Quit. Hold Quit needs a 900 ms press and release on
that same button; its thin progress bar shows the hold. Record can retry after an
error; it is disabled while warming/transcribing and until an existing review is
inserted or discarded. Cancel can stop worker startup. Insert and Submit are
typed or discarded. Cancel can stop worker startup. Type and Type + Enter are
disabled during recording and transcription. A pointer action requires
a press/release on the same enabled control from the same cursor; focus loss,
tab changes, action-state changes and relocation clear pending presses. Submit is *always* a separate
deliberate action, not inferred from text. Insert appends a trailing space (without
doubling an existing trailing space). Submit inserts any pending review and then
queues Enter; with no pending text it queues Enter only. Y opens the quick-chat
list over the review area; each further Y press cycles its highlighted choice.
Cancel closes the picker without discarding an existing review. Submit sends the
selected text *without* a trailing space, then Enter. The choices are short
tab changes, action-state changes and relocation clear pending presses. Type + Enter
is *always* a separate deliberate action, not inferred from text. Type normally
appends a trailing space (without doubling an existing one); a full 4096-byte
transcript without room for that suffix is queued unchanged, with no extra error
for the missing space. Type + Enter types any pending review and then queues
Enter; with no pending text it queues Enter only. Y opens the Quick phrases
list over the review area; each further Y press cycles its highlighted choice. Cancel closes the picker without discarding an
existing review. Type + Enter sends the selected phrase *without* a trailing
space, then Enter. The choices are short
single-line literals, not speech commands. A failed text step never proceeds to
Enter. Recording never automatically submits. Auto insert, when
explicitly enabled, can queue text + space after transcription only under the
stable Xwayland focus guard described below.
explicitly enabled, can queue text (normally + space) after transcription only
under the stable Xwayland focus guard described below.
### Bindings button
@@ -107,6 +113,8 @@ installer creates one with defaults on first install. Copy the shipped
"input_priority": "normal",
"advanced_debug": false,
"auto_insert": false,
"close_mic_when_idle": false,
"backend": "redux",
"lock_layout": false,
"clock_24h": false,
"date_format": "mdy",
@@ -149,6 +157,18 @@ config, retaining other fields and formatting; invalid/unwritable configs are
left untouched and return failure. The installer backs up original bytes before
repairing invalid values, while valid `true` and `false` are retained.
`close_mic_when_idle` is a separate boolean, default **false**, also available
as Settings → **Close mic when idle**. Normally the SDL capture device stays
open while Ready and idle samples are discarded. Enabling this toggle closes it
between clips and opens it on PTT: this avoids an open idle capture device but
can cause an audio spike (observed with per-PTT transitions on Frame), startup
latency or first-syllable clipping. Settings persistently displays **OFF: discard
idle audio; ON: spike / start latency** below the toggle, as well as an ON/OFF
indicator; the status/detail line also explains a change when toggled. The
native setting is saved to `config.json`; a failed save applies only for this
session and warns. This is not a mute switch for other applications. Quit and
failure still close the device.
`auto_insert` is a separate boolean, default `false`, also available as a
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
@@ -156,41 +176,77 @@ display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
Both properties and focus are rechecked after IME lease acquisition. A watched
focus-out, root focus-property change (even if the same window returns), window
destruction, held keyboard key, missing X display or any disagreement permanently
disarms that clip. The transcript then remains for explicit review/Insert.
disarms that clip. The transcript then remains for explicit review/Type.
Native Wayland focus and child text-field focus cannot be safely inferred here;
those cases fall back to review. No automatic Enter, speech commands or retry.
The compositor can still change focus in the gap between the final check and
global delivery, and IME commit is not an application receipt. This path has
offline synthetic focus tests and a separate owned-target IME fixture; live
speech-driven Auto Insert, target coverage and headset acceptance remain
speech-driven Auto insert, target coverage and headset acceptance remain
unverified. The setting is preserved on upgrade
and a failed preference write applies only to the current session.
`quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings,
each at most 64 characters. Edit the JSON file and restart; there is no headset
text editor. Inputs are literal (not expanded or interpreted by FrameYap) and
are sent to the current Gamescope focus, so check the destination before Submit.
are sent to the current Gamescope focus, so check the destination before Type + Enter.
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
`insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME`
button paths. Omitted actions retain their bundled defaults; an empty string
disables a mapping, including after an upgrade. The Frame defaults are right
X = hold-to-talk, B = Cancel, A = Insert + space, Y = quick chat. Enter has no
single-button mapping by default; the left grip double-tap still submits.
An existing config mapping `enter` to right Y is migrated to quick chat in memory
X = hold-to-talk, B = Cancel, A = Type + space, Y = Quick phrases. Enter has no
single-button mapping by default; the left grip double-tap still requests Type + Enter.
An existing config mapping `enter` to right Y is migrated to Quick phrases in memory
when `quick_chat` is absent; this does not overwrite custom mappings.
Existing configs with empty actions retain those disabled mappings; change them explicitly
or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized;
SteamVR user overrides may still supersede it. On customized launches a generated
action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings`
(or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The
config is read once at launch, not hot-reloaded. On install/upgrade the installer
fills missing fields, removes retired keys, and resets invalid entries. It saves
the exact prior bytes under `config.json.backup-*` before a repair and refuses
symlink/oversized config paths; valid customizations remain intact. The installed
launcher no longer pins `--font`, so this selection takes effect. Direct native
config is read once at launch, not hot-reloaded (the in-panel backend selection
is saved separately). On install/upgrade the installer fills known missing fields,
including `close_mic_when_idle` and `backend`, removes retired keys and resets
invalid entries. It saves the exact prior bytes under `config.json.backup-*`
before a repair and refuses symlink/oversized config paths; valid customizations
remain intact. The installed launcher no longer pins `--font`, so this selection takes effect. Direct native
launches with bad JSON, colors or button mappings fail startup rather than
silently changing input behavior.
### Models / backends (local implementation)
Settings → **Models / backends** opens a paginated local chooser. The saved
`"backend": "redux"` selects a *manifest ID*, not a model download;
`--backend ID` overrides it for that `--run` invocation. Selecting a listed backend
restarts the owned worker and closes capture, discarding pending audio, review
and any prior focus/delivery authorization. A failed preference write leaves the
choice active only for this session. The currently listed Redux model is the
only inference implementation supplied; adding a manifest alone does not add
an inference runtime. The Models tab labels local checks as checking, missing
(`not_installed`), invalid, or installed/verified; for the selected model it
also reports loading, ready or failed. An install-in-progress note reports
model provisioning, followed by a new offline check; a verified status alone
is not proof the CPU runtime loaded. Recording is disabled until the selected
files verify offline and the worker warms; missing files never trigger a silent
download. Model loading and the default open-while-Ready idle microphone policy
are unchanged. The panel's state and successful helper calls do **not** prove
microphone transcription, input delivery or headset acceptance.
For a missing/invalid selected model, **Install** first presents a separate
confirmation showing pinned source, approximate download size, license text,
attribution and the exact raw manifest SHA-256 fingerprint. Only **Confirm
Install** launches the local installer with that fingerprint; leaving the view
or changing metadata invalidates consent. The installer rechecks the *installed*
manifest bytes under its model lock before creating a download target or using
the network, downloads only on the explicit click, hashes pinned files and then
the app checks them again offline before enabling recording. Existing invalid
or unsafe model files are refused rather than silently overwritten. This
installs model files, **not** Python, Torch, moondream, Kestrel or other runtime
dependencies.
The source-tree UI needs an installed release for installer-backed provisioning;
a hand-edited manifest is not an approved artifact. The installer handoff and
native UI have not yet been accepted on a clean Frame. See
[packaging](packaging.md#consumer) for an inspection-first CLI path.
### Experimental controller input priority
There are two independent gates:
@@ -348,7 +404,7 @@ recentring and readability still require a separately authorized headset check.
The opt-in native `--check-controls` probe logs pointer counters and action
callbacks to the terminal rather than repainting them on the panel. Its canvas
stays static for Record/Cancel/Insert/Enter clicks so those clicks can be checked
stays static for Record/Cancel/Type/Type + Enter clicks so those clicks can be checked
without diagnostic texture uploads. Switching tabs or mount still updates
the visible panel. Diagnostics identify `renderer=Vulkan` and count
`textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions.
@@ -367,8 +423,10 @@ cmake --build build-native --target frameyap_texture_check
It does not initialize OpenVR or establish compositor/headset acceptance.
`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert,
Enter. `bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
`assets/actions.json` names seven actions: left/right grip, PTT, cancel,
Type, Type + Enter and Quick phrases. `insert`, `enter`, and `quick_chat` remain
internal binding keys; visible controls read Type, Type + Enter, Quick phrases.
`bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
the grip bindings remain for optional remapping/diagnosis. In one dashboard
probe grips were inactive; a later controls-only probe delivered repeated right
X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions
@@ -384,7 +442,7 @@ second squeeze **down** within 350 ms starts capture; hold as long as needed
action explicitly begins on down and ends on up. On tracking-pose invalidity,
action inactivity or overlay focus loss, a held capture emits Cancel, and
reconnection requires a neutral observation before any new press. PTT and left
Enter require an enabled panel; clickable Record remains available for retry
Type + Enter require an enabled panel; clickable Record remains available for retry
after an error and Cancel is always available. The action set defaults to normal
priority; the experimental config request is described above. Neither priority
guarantees delivery while a game or dashboard owns input.
+89 -26
View File
@@ -11,53 +11,71 @@ build with `FRAMEYAP_NATIVE=ON`, and stage this layout (regular files, no links)
```
bin/frameyap
lib/* # compatible bundled native libraries
assets/actions.json # and adjacent controller binding JSON
bin/install.sh # managed installer helper when required by layout
lib/* # explicitly supplied compatible native libraries
assets/actions.json # plus binding JSON and backends/redux.json
fonts/font.ttf
python/frameyap/*.py
scripts/model-status.py # offline CLI verifier; also backend-service.py, fetch-model.py
licenses/THIRD_PARTY_NOTICES.txt
model/* # optional pinned public weights + attribution
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
```
For the current **external-runtime** package, `scripts/stage-native-poc.py --help`
documents explicit inputs. It invokes `cmake --install` on an existing native build,
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
The stage copies the self-contained installer into `bin/` and CMake installs
`scripts/model-status.py` alongside the backend manifests. The package allowlist
permits `scripts/`; installed `frameyap --list-models` expects the verifier at
`../scripts/model-status.py`. **No actual staged archive has been audited/tested
from a clean account for publication**; exercise the entire producer pipeline
and the installed CLI before treating the payload layout as release-ready.
For the current **external-runtime** package, `scripts/stage-native.py --help`
documents explicit inputs. The old `scripts/stage-native-poc.py` is retained as
a deprecated migration wrapper for that command, not the documented or shipped
staging interface. The stage invokes `cmake --install` on an existing native
build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
not build, download, run the app, or copy an ASR runtime. The native
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
libstdc++ and glibc; audit `ldd` on the installed binary.
libstdc++ and glibc. Audit the actual staged ARM64 binaries' `NEEDED`,
`GLIBC_*`/`GLIBCXX_*` symbol versions, ELF interpreter and notices; then test
on a clean target. No compatible libc floor is yet established.
SDL/OpenVR resolve inside its own `lib/`, not a producer
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
```sh
python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \
--version 2026-09-24T162712Z-g417f81c --arch linux-aarch64 \
--version 0.1.202609241627 --arch linux-aarch64 \
--model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime
```
Use the actual binary's UTC build stamp (`frameyap --version`) for the archive
tag, not this illustrative timestamp. CMake generates `YYYY-MM-DDTHHMMSSZ`
at configuration time (plus `-gSHORTSHA` for a Git checkout and `-dirty` for
uncommitted tracked changes); producers may pin `-DFRAMEYAP_VERSION=...` to
embed a vetted release stamp. A timestamp distinguishes same-day archives;
the installer still refuses a reused tag whose contents have changed. Historic
`v0.1.0-poc*` local artifacts remain valid for reinstall/rollback.
Use the actual binary's numeric `MAJOR.MINOR.YYYYMMDDHHMM` UTC version
(currently `0.1`), not this illustrative value, for `--version` and filename;
tag a vetted clean tree as `vVERSION`. CMake generates this stamp at
configuration time (`SOURCE_DATE_EPOCH` may supply it);
`-DFRAMEYAP_VERSION=0.1.YYYYMMDDHHMM` can pin it. Development `--version`
may print a *separate* `git HASH` line, with `(uncommitted changes)` only if
dirty: that line is not part of the version, tag or archive name. The installer
rejects a reused tag with different contents; historic local versions may
remain selectable for rollback, not as new releases.
`--external-runtime` refuses a runtime directory and records
`runtime: external-authorized-python` in `release.json`. The installer explicitly
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
in the archive. Staging validation is not an inference test.
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz`
plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal,
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing
`HASH FILENAME`. Archive extraction rejects traversal,
links/special files, duplicate members, oversized metadata/payloads and invalid
layout. Checksums detect corruption, not a malicious/compromised publisher;
authenticate release metadata independently. No packaging/installation model fetch.
## Consumer
Bootstrap: Linux ARM64/glibc, Python 3.12+, curl, sha256sum and tar. **No compiler,
sudo, Steam store AppID or engine checkout.** Download/inspect a pinned installer
Bootstrap for a binary archive: Linux ARM64/glibc, Python 3.12+ (installer
bootstrap, **not** the inference runtime), curl for network release downloads,
sha256sum and tar for archive handling. **No compiler, sudo, Steam store AppID
or engine checkout in binary mode.** No minimum glibc floor has been certified;
preflight alone cannot guarantee compatibility. Download/inspect a pinned installer
before running it. Current local artifact route:
```sh
@@ -65,17 +83,62 @@ sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
--sha256 64_HEX_DIGIT_HASH --version VERSION
```
After an actual vetted release exists, `sh install.sh --version TAG` retrieves
that GitHub release and its versioned checksum; no `latest` or moving-branch
After an actual vetted release exists, use a real numeric version (for example,
`sh install.sh --mode binary --version 0.1.202609241627 --yes`); the installer
will retrieve tag `v0.1.202609241627` and its versioned checksum; no `latest` or moving-branch
lookup. A pipe invocation is supported, never prompts on stdin, and must also
pin a real published tag. **There is no functional public download command yet.**
`--without-model` omits bundled model files from staging, never deletes a current
model on rerun, and records the choice. Same digest/version/choice is idempotent
and repairs missing managed wrappers. Different digest or model choice for the
same version is refused. External model provisioning is always deliberate.
`--without-model` omits any model files in the selected archive, never deletes
an existing current model on rerun, and records the choice. Same
digest/version/choice is idempotent and repairs missing managed wrappers. Different digest or model choice for the
same version is refused. External model provisioning is always deliberate: the
local installer accepts `sh install.sh --install-model --backend redux --yes`
(optional `--model-dir /absolute/path`) to explicitly download and verify files from the
installed pinned manifest, not an ASR runtime. Inspect the model/size first
with `sh install.sh --install-model --backend redux --print-plan --json`.
`--expected-manifest-sha256 HASH` additionally binds consent to the exact raw
installed `redux.json` bytes: under the model lock a mismatch fails **before**
a model directory is created or any network request. In the locally implemented
Models UI, Install displays source, rounded size, license text, attribution and
the fingerprint; only the second Confirm Install click passes that fingerprint
through the backend helper to the installer. Installation alone does not
provision a CPU Python runtime; no in-panel flow has been accepted on Frame.
In the source tree, `python3 scripts/model-status.py --list-models` or
`--check-model redux --model-dir /absolute/model` hash-checks local files;
the native `frameyap --list-models` / `--check-model` entry points use the
adjacent installed verifier script; verify this in a staged archive before release. Manifest schema,
source, size/hash and attribution live in `assets/backends/redux.json` and are
validated by `python/frameyap/model_files.py`.
The installed launcher defaults to `--run`. For a native-only package, supply
For an explicit local **source** install, e.g.:
```sh
sh install.sh --mode source --source /absolute/source --openvr-root /absolute/sdk \
--openvr-library /absolute/libopenvr_api.so --openvr-license /absolute/openvr/LICENSE \
--sdl-library /absolute/libSDL3.so.0 --sdl-license /absolute/sdl/LICENSE \
--version 0.1.202609241627
```
It checks tools and native dependencies, builds/stages/packages in a private
workspace, then installs the result. Unlike binary mode, this requires a C++
compiler, CMake, SDK, SDL3, Wayland/scanner, libxcb, FreeType, Vulkan development
files and producer-supplied
licenses; it still does not install Python ASR packages. `--print-plan` performs
a read-only plan, `--json` gives machine-readable results/errors (model installs
also stream file events), and `--yes` authorizes network downloads. Bare TTY
invocation can guide choices and prints equivalent flags; non-TTY runs require
explicit arguments and never prompt. `--autolaunch`/`--no-autolaunch` are explicit
OpenVR registration choices, off by default; do not pass either during an inert
install if a running SteamVR session must remain untouched.
The installed launcher defaults to `--run` and forwards explicit run flags
(including `--backend ID`, `--model-store /absolute/store` and
`--manifest-dir /absolute/manifests`) to the binary. These overrides select
local metadata/model paths, not a runtime download; the latter two require
absolute paths without dot segments. `--backend` selects for that run unless
changed in the panel; a saved `config.json` backend is otherwise used. For a
native-only package, supply
`FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`,
or override `--python`/`--model` on an explicit `--run`. For menu launches, create
`$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`):
@@ -157,7 +220,7 @@ without configured model/runtime cannot transcribe; it should show **Unavailable
(possibly after a brief Warming transition), rather than record or infer.
Only perform this check on an unconfigured native-only installation: verify
that `current/runtime/bin/python3` and `current/model` are absent and no
user-local paths are configured; do not press Record, Insert or Enter.
user-local paths are configured; do not press Record, Type or Type + Enter.
The installer also provides a desktop entry; where Steam's UI supports adding a
non-Steam app, the user may select that entry or browse to the installed launcher.
Shortcut discovery/persistence after a normal restart is not yet verified. Do
+82 -74
View File
@@ -1,91 +1,99 @@
# Dependency provenance and release boundary
FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project
owner. This does not relicense external models, fonts, protocols or runtimes.
There is no dependency on another application's checkout, assets or environment.
FrameYap's original code is [MIT licensed](../LICENSE). This does not relicense
models, fonts, protocols, native libraries or Python wheels. There is no dependency
on another application's checkout, assets or environment. **This is an upstream
license inventory, not an audit of a particular release binary. No public release
has been published.** Before shipping any archive, inspect the actual staged
files, their transitive dependencies and notices, and test on a clean supported
host. An external dependency does not make the current archive self-contained.
## Included source
## Included source/assets
`protocol/gamescope-input-method.xml` is the unmodified public Gamescope
**3.16.28** protocol, downloaded from
<https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`.
Its embedded permissive copyright/license notice is preserved. Generated bindings
are build outputs, not hand-written wire encoding. The private protocol may change
with SteamOS; compatibility must be rechecked.
- `protocol/gamescope-input-method.xml`: unmodified public Gamescope **3.16.28**
protocol, <https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
SHA-256 `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`;
embedded permissive copyright/license notice preserved. Generated bindings are
build outputs; this private Gamescope extension needs rechecking after updates.
- `assets/fonts/Inconsolata-Regular.ttf`: unchanged Inconsolata Regular from
<https://github.com/googlefonts/Inconsolata>, copyright 2006 The Inconsolata
Project Authors, **SIL OFL 1.1**. SHA-256
`e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
`assets/fonts/OFL-Inconsolata.txt` retains the license. The font remains OFL,
not MIT, and is not sold by itself. An override font requires its own license.
- Frame controller bindings and the mint-to-blue CPU-rasterized panel were
authored here, using public profile names; no SteamVR driver artwork, MSDF
atlas, other app renderer or protected kernels were copied.
Frame controller bindings were authored here using the observed public input
profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code,
images, protected kernels or another application's assets were extracted for these bindings.
Native staging uses `scripts/stage-native.py`: it includes the chosen font/license
and copied SDL3 and OpenVR notices in `licenses/THIRD_PARTY_NOTICES.txt`.
It does **not** bundle ASR. A4 is not closed for release: inspect the final
notice file, especially FreeType attribution, plus the selected native binary
and any bundled wheel/licenses before publishing.
## Bundled font and UI reference
## Native build/runtime inventory
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular.
SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**,
retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace
normalized; license text unchanged). The reviewed OFL permits
bundling and redistribution with its copyright/license notice; the font remains
OFL, not MIT, and is not sold by itself. Upstream: <https://github.com/googlefonts/Inconsolata>.
The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies
were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph.
| Component | Upstream license / evidence | Current packaging boundary / action |
| --- | --- | --- |
| Valve OpenVR SDK 2.15.6 | BSD-3-Clause-style license, SDK LICENSE. | Staging copies its loader and explicitly supplied license; verify chosen binary and transitive closure. |
| SDL3 (device trial 3.2.16) | zlib, local `/usr/share/licenses/sdl3/LICENSE`. | Staging copies explicit library and license. Confirm exact build options/version and its transitive libraries. |
| Wayland client + scanner | MIT/Expat-style, local `/usr/share/licenses/wayland/COPYING`. | Client is linked from system; scanner is build-time. If shipped, include copyright/license and audit closure. |
| libxcb | MIT-style with name-use restriction, local `/usr/share/licenses/libxcb/COPYING`. | Xwayland focus guard uses client library at runtime; not bundled by current native stage. Audit exact binary. |
| FreeType 2 | **FreeType Project License (FTL) selected** for this project, local `/usr/share/licenses/freetype2/FTL.TXT`; upstream also offers a GPL option. | Current stage uses system library, not bundled. Credit FreeType Team for use; if distributing its binary, meet FTL binary disclaimer/notice obligations and review the precise build. Do not silently substitute GPL terms. |
| Vulkan loader, driver, system graphics dependencies | Loader/driver licenses vary by build and vendor. | Current stage depends on system Vulkan loader/driver; no GPU runtime is bundled. Audit the chosen loader if ever bundled. |
| Compiler runtime (`libstdc++`, `libgcc_s` when used) | GCC libraries: GPL with **GCC Runtime Library Exception** in upstream distribution; local `/usr/share/licenses/libstdc++/RUNTIME.LIBRARY.EXCEPTION` and `libgcc/...` are exception texts, not a full installed release audit. | Current stage relies on system runtime. Audit dynamic linkage, C++ ABI/`GLIBCXX_*` and exception coverage for *any* bundled compiler libraries; include corresponding complete notices/source obligations as applicable. |
| glibc/loader | GNU LGPL-2.1-or-later for core GNU C Library, with component-specific exceptions and other licenses to inspect. | Current stage relies on system libc/loader. Audit exact target binary symbol versions (`GLIBC_*`), ELF interpreter and its transitive closure; no minimum glibc/`GLIBCXX`/kernel floor is certified here. |
The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted
selection) is implemented independently on a single CPU RGBA surface.
CMake installs the font and OFL with assets; native staging defaults to that
font, places the launcher copy at `fonts/font.ttf`, and includes its license in
`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license.
## Explicit native build inputs (not vendored)
- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015;
retain its LICENSE with redistributed loader binaries.
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
- Wayland client and scanner: retain upstream MIT-style notices.
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
Xwayland focus observer. Include it in native runtime dependency checks;
no X server is started by normal operation.
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
- Optional font override: the earlier device check used system Hack Regular.
A custom release font must include its own license and assessed glyph coverage;
new builds default to the bundled Inconsolata described above.
- Compiler runtime, libc minimum and transitive shared libraries require a release
dependency audit. Passing a developer build is not a portable-runtime guarantee.
The historical Frame CPU trial used Torch **2.8.0+cpu** on a host with **glibc
2.39**. Those are *observed trial versions*, **not** minimum compatible versions
for FrameYap, Python wheels or a future released artifact. `ldd` on a developer
machine alone is not sufficient: inspect the staged ARM64 binaries with `readelf`
(`NEEDED`, ELF interpreter, symbol-version requirements), `ldd` on a trusted
clean target, actual bundled libraries and notices, and test the final archive on
a clean supported Frame. Check the chosen compiler, CPU instruction/kernel,
FreeType/Wayland/XCB/SDL/OpenVR/Vulkan ABI, Python/native wheels and licenses.
Do not invent a libc floor from a build host's version.
## Redux weights and inference runtime
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution:
Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3.
No modifications to the supplied weights are made. Exact model/config/tokenizer
sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`.
`fetch-model.py` fetches and retains the original model card alongside the files.
No weights are committed to this repository.
Model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**.
Attribution: Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT
0.6B v3. No modifications to the supplied weights are made. Pinned model/config/
tokenizer/card sizes and SHA-256 hashes, source and attribution are recorded in
`assets/backends/redux.json` (schema validated by `python/frameyap/model_files.py`),
not hardcoded in `model_files.py`. `fetch-model.py` fetches and retains the model
card alongside the weights on **explicit** request; neither build/tests nor a
normal app launch downloads them. `scripts/model-status.py` and the native
`--list-models` / `--check-model` interface inspect/hashes local files offline.
No weights are committed to Git.
Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels`
dependencies, installed by the user from PyPI into their own environment. Upstream's
Kestrel README states: "Local inference is free and requires no API key"
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs a Moondream
API key and is not used here. FrameYap's release archives do not vendor or bundle
these packages; they are installed from PyPI onto the user's machine, either by the
user or by the installer at the user's request.
The current Redux worker uses separately provisioned `moondream` Python and its
`kestrel` / `kestrel-kernels` dependencies, not a bundled runtime. Kestrel's
upstream README says “Local inference is free and requires no API key”
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs an API
key and is **not** this path. The native-only installer **does not run pip**,
provision an interpreter, or make a native-only artifact able to transcribe on
its own. A person supplying a Python environment must review/authorize its
exact dependency closure. No bundled-ASR artifact is licensed/approved by this
inventory.
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need
their own notice/license inventory before a bundled release.
| Python/native package | Upstream license inventory (not a wheel audit) | Release action |
| --- | --- | --- |
| PyTorch / Torch CPU | PyTorch project: BSD-3-Clause; third-party components/wheels carry additional notices and dependencies. | No Torch wheels bundled. Pin CPU-only ARM64 wheel if building a distribution; audit its `LICENSE`, `NOTICE`, `third_party`/wheel contents and `NEEDED`/symbol versions. |
| NumPy | NumPy core: BSD-3-Clause; dependencies/embedded algorithms carry additional BSD, MIT, 0BSD, zlib, CC0 and other notices depending on wheel. Local `python-numpy` 2.5.3 package metadata (`/usr/lib/python3.14/site-packages/numpy-2.5.3.dist-info/METADATA`) declares `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` with many `License-File` entries (different from the historical trial environment). | Not bundled. Keep *all* license files and inspect the exact target wheel, BLAS/OpenBLAS and runtime closure before redistribution. |
| Hugging Face `tokenizers` | Upstream `huggingface/tokenizers` is Apache-2.0; native/Rust crate dependencies need separate inventory. | Not bundled. Confirm actual installed wheel version, package LICENSE/NOTICE and transitive Rust/native code if ever distributed. |
| `moondream`, Kestrel/kernels/native, Python interpreter | Distinct packages with distinct license files and native transitive code; Kestrel local-use statement is not a blanket redistribution license. | None bundled. Exact versions, permissions, wheel notices, CPython build and native linkage require review before any runtime bundle. |
## CPU trial dependency choice
Development CPU trial *interfaces*, not package-floor promises: moondream
**2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, native **0.1.8**, Python
**3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified moondream install
initially resolved CUDA-enabled Torch and NVIDIA wheels; the owned trial venv
was corrected before measurement (`torch.version.cuda is None`). Do **not**
repeat unconstrained `pip install moondream` as a CPU setup recipe.
Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**,
native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified
moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels;
these were replaced/removed from the owned venv before measurement. The measured
Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
`pip install moondream` as a CPU setup recipe.
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
was downloaded but not bundled with any ASR runtime:
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
A standalone CPython 3.12.14 ARM64 distribution was downloaded for packaging
research but **not bundled**: <https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
Its included licenses and compatible native dependencies still require review
before any release. No public prebuilt FrameYap release has been published.
Its own licenses and native compatibility need review before any release.
+38 -15
View File
@@ -1,8 +1,10 @@
# Offline Redux worker adapter (component, not an installed product)
# Offline worker adapter and backend dispatcher
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
threads=2, advanced_debug=false)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for
one `WorkerReply` (text or privacy-safe per-request error). One request at a time;
threads=2, advanced_debug=false)` explicitly for the legacy Redux script, or
supply the optional `backend`, `manifest_dir`, `root` arguments for the manifest
dispatcher. Poll until `ready()`, then `submit(id, pcm)` and poll for one
`WorkerReply` (text or privacy-safe per-request error). One request at a time;
no queue, no capture and no input injection. `stop()` discards pending audio,
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
and is safe to repeat. Destruction stops it. `start()` returns without waiting
@@ -36,13 +38,19 @@ requests reading the fixed clip; `Y` means ready; `F` means load failure
missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8
text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
or oversized frame stops the worker. Warmup deadline is 120 s, transcription
deadline 60 s; `poll()` must be called regularly to enforce deadlines. It
never initializes a headset or starts a recording. There is no auto restart.
or oversized frame stops the worker. A correlated `E` reply or an invalid
transcript is a **request-level** failure: the Controller drops that clip,
keeps the ready model loaded and allows Record to retry. A broken protocol,
worker crash or explicit in-flight cancellation is different and can require a
reload. Warmup deadline is 120 s, transcription deadline 60 s; `poll()` must
be called regularly to enforce deadlines. It never initializes a headset or
starts a recording. There is no auto restart after a process failure.
`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI
startup, with HF/Transformers/Datasets offline variables and bounded native
thread-pool variables set before import. It uses
Parakeet Redux is the **model** (`moondream/parakeet-redux`); `moondream` is
its Python inference package, not a second model or cloud endpoint. Kestrel is
a native dependency of that local runtime. `python/frameyap/worker.py` lazily
imports `moondream` only after explicit CLI startup, with HF/Transformers/Datasets
offline variables and bounded native thread-pool variables set before import. It uses
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
device="cpu", cpu_threads=threads)` and persistent
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
@@ -51,16 +59,31 @@ moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
preinstalled local weights from revision
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
against `model_files.py` before importing model libraries. Protocol stdout is
isolated at the file-descriptor level from third-party diagnostics. Thread limits
cover Torch interop/native pools and CUDA is not selected. Offline environment
against `assets/backends/redux.json` via the shared, offline
`python/frameyap/model_files.py` schema/verifier before importing model libraries.
`frameyap --list-models` and `frameyap --check-model redux --model-dir /absolute/model`
(or `scripts/model-status.py`) read local manifests and optionally hash local
files; they never load the worker or download weights. For native `--run`,
`--backend ID`, `--model-store /absolute/store`, and
`--manifest-dir /absolute/manifests` are wired through the installed launcher
as explicit overrides; default selection is saved in `config.json` (`redux`). The dispatcher
`python/frameyap/backend_worker.py` validates a selected manifest and hashes
its local model, resolves an in-release relative Python/executable launcher
without a shell, then supervises one child over `frameyap-worker-v1` (ready,
correlated request/reply and failure frames). The launcher must accept
`{model_dir}` and `{clip_dir}` (optional `{threads}`); matching manifest and
pinned weights are **not** a runtime or automatically trusted new backend.
Only Redux is supplied today. Protocol stdout is isolated at the file-descriptor
level from third-party diagnostics. Thread limits cover Torch interop/native
pools and CUDA is not selected. Offline environment
flags do not prove every third-party internal is unable to access a network.
Runtime/build/tests perform no downloads; the separate explicit setup utility
`scripts/fetch-model.py` can provision the public pinned weights.
Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public
runtime bundle has been released. Limited ARM64 measurements were taken during development;
they are not a claim of complete headset acceptance.
Current native-only archives do not bundle or pip-install the runtime; see
[third-party notes](third-party.md). No public runtime bundle has been released.
Limited ARM64 measurements were taken during development; they are not a claim
of complete headset acceptance.
## Advanced debugging