mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-06 01:00:04 +02:00
docs: align release scope and track remaining device acceptance gates
This commit is contained in:
baketnk committed
1 parent
4498ef5d04
commit
1321ca8316
9 files changed
+622
-527
No files matched your search
@@ -1,138 +1,47 @@
|
||||
# FrameYap
|
||||
|
||||
Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress).
|
||||
Voice typing on Steam Frame, with recognition on the headset rather than a desktop or cloud server.
|
||||
Hold a controller button to record, review the transcript, then deliberately type it into the focused app.
|
||||
Standalone MIT-licensed OpenVR overlay; no Steam store AppID or sudo. First v0.1 release is in progress.
|
||||
|
||||
Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture,
|
||||
persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope,
|
||||
and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR
|
||||
server, cloud fallback or unrelated application dependency.
|
||||
## Requirements
|
||||
|
||||
**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility,
|
||||
Gamescope discovery and native-only installation have been exercised on Frame.
|
||||
Live microphone → reviewed text → real target delivery is **not yet accepted**.
|
||||
- Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format. Binary compatibility must be checked against each actual release artifact, not inferred from a developer build.
|
||||
- For voice recognition, separately provision a compatible **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0 and dependencies) and the pinned local Parakeet Redux model. Neither is bundled or installed with pip by the current native-only installer. There is no fallback ASR service.
|
||||
- A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](docs/build.md).
|
||||
|
||||
**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the
|
||||
`moondream` Python package (its Kestrel runtime states that local inference is free
|
||||
and needs no API key). The build and tests never download it; the installer or you
|
||||
install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md).
|
||||
No GitHub release is published yet.
|
||||
## Install (local artifacts only)
|
||||
|
||||
## Controls
|
||||
|
||||
- **Right X (default Frame binding):** hold to record; release to
|
||||
transcribe. Repeated presses reached the controls-only diagnostic on Frame;
|
||||
live mic capture through this shortcut still needs guided acceptance. This
|
||||
PTT action is remappable through SteamVR bindings.
|
||||
- **Right B:** cancel/discard (or close quick chat). **Right A:** insert reviewed text with a trailing space.
|
||||
**Right Y:** open quick chat; press again to cycle its highlighted choice.
|
||||
**Submit** on the overlay (or double-tap left grip when active) sends that choice
|
||||
verbatim then Enter; outside quick chat it inserts pending review + Enter, or
|
||||
sends Enter alone when there is no text. Edit the `quick_inputs` list in
|
||||
`$XDG_CONFIG_HOME/frameyap/config.json` (restart to apply). Nothing submits
|
||||
automatically. Auto Insert is opt-in and off by default.
|
||||
- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Submit and Hold Quit
|
||||
(hold the button for 0.9 seconds before releasing).
|
||||
Review and settings share one Inconsolata/neon-framed surface. The header
|
||||
shows local time/date; Settings selects 12/24-hour time and date format/off.
|
||||
Hold the thin bar below the panel to freely move and rotate it with your
|
||||
controller; release to leave it at that pose. Drag the external lower-right
|
||||
bracket to scale it (world, head or wrist), keeping the upper-left anchored.
|
||||
Both handles use the app's gradient in transparent margins, like Steam's
|
||||
window handles. Settings → **Lock grab/scale** hides and disables both handles;
|
||||
the lock is saved. Position/size changes last for this run only.
|
||||
Wrist mounting fades the panel as its full orientation turns away from an
|
||||
upright viewer-facing pose (60°–75°), hiding interaction past that angle;
|
||||
world and head mounting do not fade.
|
||||
**Bindings** requests SteamVR's binding editor directly. Existing SteamVR overrides may supersede defaults. World-space by
|
||||
default; settings offer left wrist, right wrist and head mounting, plus recenter.
|
||||
Dashboard lasers provide clickable controls. Settings → Lasers anytime is an
|
||||
opt-in, default-off system-wide laser mode while the panel is visible; it may
|
||||
affect games and is separate from experimental input overrides.
|
||||
- **Theme and controls:** optional `$XDG_CONFIG_HOME/frameyap/config.json` selects
|
||||
panel colors, a font path and Frame controller button mappings; missing fonts
|
||||
fall back to bundled Inconsolata. The installer creates/checks this file and
|
||||
backs it up before repairs. See [overlay configuration](docs/overlay.md#user-theme-and-controller-configuration).
|
||||
- **Advanced debugging:** Settings toggle / `"advanced_debug": true` in config.
|
||||
Off by default. Restarts the worker and discards current work; full exceptions,
|
||||
worker output and transcripts go to private, bounded local logs. No raw audio
|
||||
archive. See [diagnostics](docs/worker.md#advanced-debugging).
|
||||
- **Experimental input priority:** set `"input_priority": "experimental"` in
|
||||
that config and enable SteamVR's Developer option **Enable global input from
|
||||
overlays**. FrameYap then requests priority for its bound controller sources.
|
||||
This may consume controls used by games or the dashboard; coexistence on Frame
|
||||
is under test. The default is `"normal"`; restart FrameYap after changing it.
|
||||
- **Review by default:** focus your destination, then press Insert (text + space) or
|
||||
explicitly Enter (text + space, then Enter). Settings → Auto insert is off by default:
|
||||
when enabled, it queues text + space only if Xwayland keyboard focus, active
|
||||
window and Gamescope focus match continuously from recording through delivery.
|
||||
Any uncertainty leaves a preview for manual Insert; it never sends Enter.
|
||||
This is not yet validated for live transcription on Frame. Maximum clip 20 seconds;
|
||||
accidental taps under 200 ms are discarded.
|
||||
- While the native app is Ready, it keeps the mic device open and discards idle
|
||||
audio instead of opening/closing on every PTT. Quit releases the device. Other
|
||||
apps may still hear/transmit your voice; FrameYap does not mute them.
|
||||
|
||||
Physical gesture timing, global bindings during games, mic capture, target-app
|
||||
compatibility and headset comfort still need coordinated validation. Successful
|
||||
API initialization is not delivered input or human acceptance.
|
||||
|
||||
## Build and test offline
|
||||
|
||||
CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests.
|
||||
**No public release is published.** To install a locally vetted, checksummed ARM64 native-only archive without a compiler:
|
||||
|
||||
```sh
|
||||
cmake -S . -B build
|
||||
cmake --build build
|
||||
ctest --test-dir build --output-on-failure
|
||||
./build/frameyap --help
|
||||
```
|
||||
|
||||
Default build has no hardware backends. It never downloads packages/models or
|
||||
initializes SteamVR, a microphone or input injection. Native dependencies and
|
||||
explicit launch/check commands are documented in [the build guide](docs/build.md).
|
||||
`--version` uses an ISO-like UTC build timestamp (with Git hash when available),
|
||||
not a numbered release; use that same tag when packaging the binary.
|
||||
|
||||
## Installation
|
||||
|
||||
The real installer accepts a versioned, checksummed prebuilt ARM64 archive:
|
||||
|
||||
```sh
|
||||
sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
||||
sh install.sh --mode binary --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
||||
--sha256 ARCHIVE_SHA256 --version VERSION
|
||||
```
|
||||
|
||||
This is the **local-artifact command shape**, not an available public download.
|
||||
Installation is user-local, retains rollback, refuses active-app upgrades and
|
||||
foreign files, and does not launch or register automatically. Registration uses
|
||||
OpenVR application key `local.frameyap.overlay`, not a Steam store AppID.
|
||||
Autolaunch is opt-in. An installed desktop entry can be selected manually as a
|
||||
non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the
|
||||
panel on one Frame; registration alone did not show an entry in the first checked
|
||||
dashboard menu. See [packaging and lifecycle](docs/packaging.md).
|
||||
For a native-only install, menu-driven inference needs a Python runtime you
|
||||
provide and the pinned model. Configure their absolute paths in
|
||||
`~/.config/frameyap/paths.conf` as described in the packaging guide; they are
|
||||
never fetched or bundled implicitly.
|
||||
`VERSION` is numeric `0.1.YYYYMMDDHHMM` for this release line; a future release tag will be `vVERSION`. The installer has an explicit local source-build path with toolchain/dependency inputs, and a read-only `--print-plan`/machine-readable `--json` mode; the producer stage→archive path still needs a clean-account artifact test and release audit. See [packaging](docs/packaging.md). Installation is user-local with rollback, no automatic launch or registration, and no Steam store AppID. Before voice typing, independently supply a licensed Python CPU environment and pinned weights, and configure their absolute paths in `~/.config/frameyap/paths.conf` (or the XDG config equivalent). Model download is opt-in and separate: `sh install.sh --install-model --backend redux --print-plan --json` inspects pinned metadata, while `--yes` explicitly authorizes installation from the *installed* manifest; `--expected-manifest-sha256 HASH` additionally binds consent to its exact bytes. The locally implemented Settings → Models chooser offers selection/restart and a two-click Install/Confirm Install with source, size, license, attribution and manifest SHA-256; it has not been validated as an installed/headset flow. No chooser action installs the Python runtime. Do not mistake a native-only install for working ASR. The pinned GitHub download route must not be advertised as functional until a vetted release exists.
|
||||
|
||||
A pinned GitHub one-command route is implemented in `install.sh --version TAG`,
|
||||
but **do not advertise or run it as a working public installation until a vetted
|
||||
release exists**. Native-only artifacts support the overlay/checks without an
|
||||
end-user compiler; a bundled-ASR distribution is not yet offered.
|
||||
## Controls (default Frame binding)
|
||||
|
||||
## Project map
|
||||
| Button / control | Action |
|
||||
| --- | --- |
|
||||
| Right X, hold / release | Record while held; release to transcribe. |
|
||||
| Right B | Cancel/discard, or close the Quick phrases picker. |
|
||||
| Right A | **Type:** queue reviewed text, normally with a trailing space. |
|
||||
| Right Y | Open **Quick phrases**; press again to cycle the selection. |
|
||||
| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. |
|
||||
| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. |
|
||||
| Overlay Type / Type + Enter | The same deliberate text / Enter actions. |
|
||||
| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). |
|
||||
|
||||
- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests.
|
||||
- [Design](docs/design.md): full target design; some features remain proposed.
|
||||
- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md).
|
||||
- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md),
|
||||
[dependency/license inventory](docs/third-party.md).
|
||||
- [TODO](TODO.md): release plan and open work items.
|
||||
Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Check the focused destination before Type or Type + Enter. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](docs/overlay.md).
|
||||
|
||||
Dated device-test records are kept locally (untracked) and are not authority for
|
||||
current device availability. Live microphone → reviewed text → target delivery has
|
||||
not been formally accepted; see Status above.
|
||||
## Status / not yet validated
|
||||
|
||||
No recordings, transcripts, private logs, model weights or runtime binaries are
|
||||
committed. The worker boundary is intentionally small for forks experimenting
|
||||
with other models/APIs; the default remains local-only Redux.
|
||||
Native ARM64 build, CPU inference on a public clip, overlay visibility, Gamescope API discovery, a controls-only Right X probe and native-only installation have been exercised on Frame. **Live microphone → reviewed transcript → real target delivery has not been formally accepted.** Focus guard and insertion have offline/owned-target checks, not general app compatibility or human headset acceptance. Global bindings, physical gesture feel, Auto insert with speech, latency, battery/thermal cost and controller coexistence still require opt-in headset testing. Front-prefix loss on repeated submissions (P1) remains under separate investigation; do not treat it as fixed. An old-code fixture crashed the Gamescope session, not the OS, and does not prove the new delivery path. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it.
|
||||
|
||||
The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](docs/overlay.md), [worker](docs/worker.md), [design](docs/design.md).
|
||||
|
||||
Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.<timestamp>` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](docs/third-party.md); [TODO](TODO.md).
|
||||
|
||||
No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection.
|
||||
@@ -6,95 +6,110 @@ M ≈ a day, L ≈ multi-day.
|
||||
|
||||
Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
|
||||
downloads, no automatic Enter, small verified commits, docs must state implemented
|
||||
vs proposed behavior honestly.
|
||||
vs proposed behavior honestly. Checked source-work items below mean the local
|
||||
implementation is present, **not** a shipped or headset-accepted release.
|
||||
|
||||
## Implemented locally; offline tested; installed/headset validation pending
|
||||
|
||||
The current working tree includes the source changes for A3, A5–A7, B1–B4,
|
||||
C1–C3, D1–D2, E2–E3 and F1–F2. Offline checks passed 23/23 default,
|
||||
24/24 strict UI, and 28/28 on a private native ARM64 snapshot. Later review
|
||||
fixes require a fresh final native build before deployment.
|
||||
These checkboxes close the *source tasks*, not their empirical acceptance gates.
|
||||
C1's second backend is a fake executable fixture, **not** a second shipped ASR
|
||||
engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's
|
||||
source/attended installer paths still need an audited native archive and an
|
||||
installed clean-account/Frame exercise. A4 (exact artifact ABI/license closure),
|
||||
D3 (publication and clean-account acceptance), P1 (real-target delivery), G1
|
||||
(browser), and H (live headset acceptance) remain open. No new build is claimed
|
||||
deployed to Frame; local code/tests cannot establish a fixed delivery regression.
|
||||
|
||||
## Priority regression (reported during implementation)
|
||||
|
||||
- [ ] **P1. Repeated delivery loses the beginning of later submissions.** After
|
||||
the first Type + Enter, later text reportedly loses a dozen to a few dozen leading
|
||||
bytes. Investigate preview versus destination loss, retain the full bounded
|
||||
literal transcript, and add repeated/long/Unicode delivery regression tests.
|
||||
Do not assume a larger buffer fixes it or retry uncertain delivery automatically.
|
||||
An old-code delivery fixture crashed the Gamescope session, **not** the OS;
|
||||
this is not evidence of a fix. Real-target confirmation remains required after
|
||||
a tested fix is deployed.
|
||||
|
||||
## A. First release (v0.1) blockers
|
||||
|
||||
- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
|
||||
states local inference is free; blocker text removed from README/docs/scripts and
|
||||
replaced with a neutral dependency note in `docs/third-party.md`.
|
||||
- [x] **A2. Remove references to the unrelated app (kouseki).** (S)
|
||||
Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a
|
||||
final grep. Keep the Inconsolata/OFL attribution and cite the
|
||||
upstream font source (googlefonts/Inconsolata) instead.
|
||||
*Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains.
|
||||
- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
|
||||
"FrameYap"). (S)
|
||||
- [x] **A2. Remove references to the unrelated app.** (S)
|
||||
Tracked references were removed while preserving Inconsolata/OFL attribution,
|
||||
font SHA-256 and upstream googlefonts/Inconsolata source. A tracked,
|
||||
case-insensitive grep for the former app name must remain empty.
|
||||
- [x] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
|
||||
"FrameYap"). (S) Done in baseline checkpoint `07c03ea`.
|
||||
- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M)
|
||||
Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL
|
||||
option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt
|
||||
archive that includes any of them.
|
||||
- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README,
|
||||
`scripts/stage-native-poc.py`, `CMakeLists.txt` messages,
|
||||
installer strings. v0.1 is a first small release.
|
||||
- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install,
|
||||
controls **table** (button → action), then a "Status / not yet validated" section.
|
||||
Today it reads as a lab notebook and Controls is a wall of text.
|
||||
Upstream inventory and the FreeType FTL choice are documented, but the exact
|
||||
staged ARM64 native/runtime binaries, transitive wheel/library notices, symbol
|
||||
versions, loader and libc floor still require artifact-specific review before
|
||||
publishing any prebuilt archive.
|
||||
- [x] **A5. Drop "POC" from the shipped surface.** (S) Public help, README,
|
||||
CMake and installer use the release name. `scripts/stage-native-poc.py` remains
|
||||
a deprecated compatibility wrapper for `scripts/stage-native.py`, not the
|
||||
documented or shipped staging entry point. v0.1 is a first small release.
|
||||
- [x] **A6. Rewrite the README front.** (M) Pitch, requirements, local-only install,
|
||||
controls table and explicit "Status / not yet validated" section are present.
|
||||
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
|
||||
to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
|
||||
Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and
|
||||
hedging language before v0.1.
|
||||
Current design/overlay/packaging docs distinguish implemented local behavior
|
||||
from proposed and unaccepted headset/release behavior.
|
||||
|
||||
## B. Correctness / robustness
|
||||
|
||||
- [ ] **B1. A malformed transcript must not kill the worker.** (S)
|
||||
`src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control
|
||||
characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and
|
||||
`session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a
|
||||
request-level error (like the `E` path: keep the worker, show "transcription
|
||||
failed", allow retry). Add a test with a control-character reply asserting the
|
||||
worker stays ready.
|
||||
- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes
|
||||
"moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a
|
||||
worker-supplied message code. Prerequisite for C1.
|
||||
- [ ] **B3. "Close mic when idle" setting, default OFF.** (M)
|
||||
Default keeps the mic open while Ready (opening/closing per PTT causes an audio
|
||||
spike on the physical hardware, and it avoids first-syllable clipping). The
|
||||
setting closes it between clips for people who don't want a live device. Document
|
||||
the tradeoff (spike/latency) next to the toggle, in Settings and in the docs.
|
||||
Persist in `config.json`; add to the panel Settings tab.
|
||||
- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L)
|
||||
`run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no
|
||||
tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`)
|
||||
so PTT, cancel, quick phrases, auto-insert and error transitions are testable
|
||||
without hardware. B1 is the first regression test.
|
||||
- [x] **B1. Keep malformed transcripts request-local.** (S)
|
||||
`Controller::tick()` catches a correlated bad UTF-8/control reply and fails the
|
||||
request without stopping the ready worker; hardware-free fakes test retry.
|
||||
- [x] **B2. Use engine-neutral C++ worker errors.** (S)
|
||||
`F`/`I` messages no longer name Redux's Python engine.
|
||||
- [x] **B3. "Close mic when idle" setting, default OFF.** (M)
|
||||
Config/Settings and fake-backed mic-lifetime tests cover default idle draining
|
||||
versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
|
||||
- [x] **B4. Extract/test interaction logic from `run()`.** (L)
|
||||
`Controller` receives injectable audio/worker/focus/delivery interfaces; offline
|
||||
tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.
|
||||
|
||||
## C. Backends and model management
|
||||
|
||||
- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L)
|
||||
Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper,
|
||||
sherpa-onnx Parakeet, …) as separate worker executables speaking the existing
|
||||
`Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name,
|
||||
launcher, pinned model files + hashes + attribution, license text, CPU/GPU
|
||||
requirements) so nothing is hardcoded in C++ or `model_files.py`.
|
||||
*Done when:* Redux is expressed as a manifest, and a second backend can be added
|
||||
without touching `worker.cpp`/`runtime.cpp`.
|
||||
- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4.
|
||||
Settings page listing backends/models with state (not installed / installed and
|
||||
verified / loading / ready / failed), the active one marked, and selection that
|
||||
restarts the worker. Target user is non-technical: an **Install** button on the
|
||||
panel runs the installer's machine-readable mode (D2) as a child process and shows
|
||||
progress/errors in the panel, so nobody needs a terminal. The click is the consent;
|
||||
there are still no implicit or background downloads, and the panel states what
|
||||
will be downloaded and how large it is before it starts. Persist the choice in
|
||||
`config.json`.
|
||||
- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID`
|
||||
(offline, hash-verifies installed files) so the UI and installer share one
|
||||
implementation.
|
||||
- [x] **C1. Manifest-driven worker backends (source capability).** (L)
|
||||
Redux's pinned manifest and a generic local dispatcher implement the existing
|
||||
`Y`/`T`/`R`/`E` framing. An offline fake second executable backend works without
|
||||
edits to C++ worker/runtime; only Redux has a shipped inference engine. New
|
||||
backends still require license/runtime and actual inference validation.
|
||||
- [x] **C2. Model/backend state and chooser (local source).** (L)
|
||||
Settings lists manifest-backed model status and active/loading/ready/failure
|
||||
states; selection persists and invalidates/restarts worker, clip and review.
|
||||
Install → Confirm Install displays pinned source, size, license, attribution and
|
||||
manifest SHA-256; confirmation launches an owned installer helper with that
|
||||
digest and offline rechecks afterward. Full consent metadata is paginated;
|
||||
the panel shows bounded per-file download/verification events and sanitized
|
||||
errors. Installer output alone never proves success. No implicit downloads or
|
||||
runtime install. **Installed chooser/consent behavior on Frame remains unvalidated.**
|
||||
- [x] **C3. Offline model status CLI.** (S) `frameyap --list-models` and
|
||||
`--check-model ID` dispatch pinned local manifest/hash checks, without ASR or
|
||||
downloads. Installed archive CLI still needs clean-account verification.
|
||||
|
||||
## D. Installer
|
||||
|
||||
- [ ] **D1. Installer with a binary-or-source choice.** (L)
|
||||
`install.sh` offers *prebuilt archive* (checksummed) or *build from source*
|
||||
(checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then
|
||||
continues automatically through install after the user's choices. Retain rollback,
|
||||
idempotency, no sudo, no Steam AppID, opt-in autolaunch.
|
||||
- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M)
|
||||
Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`,
|
||||
`--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`,
|
||||
`--json` output) so a model/agent can run it non-interactively; interactive
|
||||
prompts only run on a TTY and print the equivalent flags they chose. Exit codes
|
||||
and messages must be machine-readable.
|
||||
- [x] **D1. Local binary-or-source installer paths.** (L)
|
||||
`install.sh` has a checksummed binary route and explicit local source/toolchain
|
||||
preflight/build/stage/install route, with rollback and no default registration.
|
||||
A native archive was exercised in private HOME/XDG roots on Frame: install,
|
||||
installed model-status CLI, idempotency, wrapper repair and uninstall. This
|
||||
same-user/system-library fixture is not clean-account acceptance.
|
||||
- [x] **D2. Attended-or-unattended model-agnostic installer interface.** (M)
|
||||
TTY choices have equivalent flags; `--mode binary|source`, `--backend ID`,
|
||||
`--model-dir`, `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`,
|
||||
`--print-plan` and `--json` support offline plans and structured outcomes;
|
||||
explicit model installs use installed pinned manifests. Tested with local
|
||||
fixtures only, not a released archive or an installed Frame UI handoff.
|
||||
- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the
|
||||
release checklist in `docs/packaging.md`; do not advertise the one-command route
|
||||
until the archive and its checksum are actually published and tested from a clean
|
||||
@@ -106,32 +121,24 @@ vs proposed behavior honestly.
|
||||
says what it is; no rename churn before the first tag. If a hardware-neutral
|
||||
project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
|
||||
front end), decide it before the cross-window work in G, not now.
|
||||
- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text +
|
||||
space) and **Type + Enter**; align overlay buttons, `--check-controls` output,
|
||||
README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only
|
||||
if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings
|
||||
explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux"
|
||||
vs `moondream` naming once in `docs/worker.md`.
|
||||
- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S)
|
||||
e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts
|
||||
correctly, keeps the build date visible, URL/filename-safe. The version field is
|
||||
just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted
|
||||
changes) suffixes were only for telling developer builds apart, so they move out
|
||||
of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev
|
||||
build adds a second line like `git abc12345 (uncommitted changes)`. Release
|
||||
archives are built from a clean tag, so users never see it. Update the CMake
|
||||
version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`,
|
||||
installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the
|
||||
timestamp. Tag releases `v0.1.<timestamp>`.
|
||||
- [x] **E2. User-facing Type / Type + Enter / Quick phrases labels.** (S)
|
||||
Overlay, diagnostics, README/help and overlay docs align on these controls;
|
||||
`insert`, `enter`, `quick_chat` remain internal binding/API names. Settings
|
||||
explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model
|
||||
from its `moondream` Python inference package.
|
||||
- [x] **E3. Numeric `MAJOR.MINOR.YYYYMMDDHHMM` version source work.** (S)
|
||||
CMake, CLI/tests, package producer and installer use numeric release versions;
|
||||
dev git info is a separate `--version` line. `SOURCE_DATE_EPOCH` can supply
|
||||
the configuration timestamp. A clean release tag/archive still needs D3.
|
||||
|
||||
## F. Code structure (non-urgent)
|
||||
|
||||
- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of
|
||||
inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a
|
||||
table-driven option parser.
|
||||
- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag
|
||||
state, save-failure flags, counters, pose caches): separate drag, persistence and
|
||||
diagnostics.
|
||||
- [x] **F1. Move `--check-*` diagnostics out of `main.cpp`.** (S)
|
||||
`src/check.cpp` owns native checks; `src/cli.cpp` provides the table-driven
|
||||
parser. Device behavior remains separately gated.
|
||||
- [x] **F2. Split `overlay.cpp`'s `Impl`.** (M)
|
||||
Drag state, persistence/save failures and diagnostics now have separate grouped
|
||||
owners; native snapshot build and offline panel/drag tests cover the refactor.
|
||||
|
||||
## G. Other windows (post-release)
|
||||
|
||||
@@ -158,7 +165,7 @@ this app is future design and out of scope for v0.1.
|
||||
across a normal restart (without restarting sessions just for the test). Confirm no
|
||||
runtime/model environment override before future live checks.
|
||||
- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery,
|
||||
Auto Insert with speech, physical resize.
|
||||
Auto insert with speech, physical resize and a verified deployed version.
|
||||
|
||||
---
|
||||
|
||||
@@ -179,4 +186,5 @@ this app is future design and out of scope for v0.1.
|
||||
|
||||
## Open questions
|
||||
|
||||
None currently blocking.
|
||||
Release artifact compatibility/license audit, publication, P1 real-target behavior
|
||||
and live headset validation are unresolved gates, not implied by checked source tasks.
|
||||
+55
-25
@@ -11,8 +11,9 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
||||
mode while FrameYap is visible; it may affect games, and is not an input override.
|
||||
- Remappable SteamVR actions. The default Steam Frame binding maps right X
|
||||
(hold to record, release to transcribe) to the existing PTT action using the
|
||||
observed `frame_controller` profile. Right B cancels, A inserts + space, and Y
|
||||
inserts pending text + Enter (or Enter only with no preview). The Bindings button
|
||||
observed `frame_controller` profile. Right B cancels, A requests Type (text +
|
||||
space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay
|
||||
control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button
|
||||
requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
|
||||
actions were inactive in the observed dashboard check; do not rely on them.
|
||||
If left grip becomes active, two short taps request explicit Enter.
|
||||
@@ -22,27 +23,41 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
||||
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
|
||||
20 second maximum. In the native app the microphone stream opens after model
|
||||
warm-up, stays running between utterances, and discards idle samples; PTT does
|
||||
not open/pause/close the device. Quit, worker restart, device failure or capture
|
||||
failure closes it. This avoids repeated capture-device transitions but does not
|
||||
promise glitch-free playback on every audio stack. Other apps may still
|
||||
not open/pause/close the device **by default**. Settings → Close mic when idle
|
||||
(`"close_mic_when_idle": true`, default **false**) closes it between clips and
|
||||
reopens on PTT; repeated transitions caused an audio spike on Frame and may
|
||||
add latency or clip the first syllable. Quit, worker restart, device failure
|
||||
or capture failure also closes it. Keeping it open does not promise glitch-free
|
||||
playback on every audio stack. Other apps may still
|
||||
transmit your voice: this app does **not** mute VRChat or any other app.
|
||||
- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed
|
||||
backend restarts the worker and invalidates pending clip/review/focus authority;
|
||||
a missing or invalid model disables recording. A separate Install then Confirm
|
||||
Install exposes pinned source/size/license/attribution and exact manifest-byte
|
||||
SHA-256 before installer handoff. These are local implementations, not a tested
|
||||
installed UI or an approved additional inference engine. Only Redux is supplied.
|
||||
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
|
||||
cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
|
||||
Model imports are lazy and loading is offline. Normal repeats, request-local
|
||||
transcription failures and microphone failures retain the loaded model;
|
||||
Model imports are lazy and loading is offline. A correlated bad transcript/
|
||||
`E` reply is a request-level failure: Record can retry without dropping the
|
||||
ready model. Normal repeats and microphone failures retain the loaded model;
|
||||
microphone device failure releases the stream for explicit retry. A cancelled
|
||||
in-flight request or broken worker protocol may require reloading. No
|
||||
cloud/desktop fallback.
|
||||
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
|
||||
handling, UTF-8/control validation and explicit Submit action for Enter.
|
||||
Insert ensures a trailing space without doubling an existing one. Enter
|
||||
first inserts pending review, releases the text lease, then acquires a fresh
|
||||
lease for Submit. Failed/uncertain text never proceeds to Submit; failed
|
||||
Submit acquisition never replays text. A full 4096-byte transcript without
|
||||
room for a space is preserved with an error, never silently truncated.
|
||||
- Idempotent user-local release-archive installer: SHA-256, safe extraction,
|
||||
atomic current-version selection, retained rollback, runtime/install lock,
|
||||
foreign-file refusal and explicit unregister-before-uninstall acknowledgement.
|
||||
handling, UTF-8/control validation and explicit Type + Enter action.
|
||||
Type ensures a trailing space without doubling an existing one. Type + Enter
|
||||
first types pending review, releases the text lease, then acquires a fresh
|
||||
lease for Enter. Failed/uncertain text never proceeds to Enter; failed
|
||||
Enter acquisition never replays text. If a validated transcript fills the
|
||||
4096-byte bound and lacks a trailing space, Type preserves all its bytes and
|
||||
queues it **without** the usual space; it does not signal a separate error.
|
||||
Destination consumption and repeated-delivery behavior remain unaccepted.
|
||||
- User-local installer with checked binary-archive or explicitly provisioned
|
||||
source-build mode, safe extraction, atomic current-version selection, retained
|
||||
rollback, runtime/install lock, foreign-file refusal and explicit
|
||||
unregister-before-uninstall acknowledgement. Source mode needs a local
|
||||
compiler, SDK, libraries and license inputs; no runtime is pip-installed.
|
||||
|
||||
## Deliberately not claimed
|
||||
|
||||
@@ -63,15 +78,26 @@ fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip binding
|
||||
are not guaranteed globally active in every scene/dashboard state, and the app
|
||||
never enables SteamVR's experimental overlay overrides on your behalf.
|
||||
|
||||
This is a compact prototype panel, not yet the proposed polished miniature status
|
||||
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
|
||||
impact and target application compatibility require further headset work.
|
||||
This is one compact panel, not a separate miniature status chip. Front-prefix
|
||||
loss on repeated submissions (P1) is under separate investigation, **not fixed**
|
||||
by the chooser/worker documentation or any speculative delivery change. Font
|
||||
coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact
|
||||
and target application compatibility require further headset work.
|
||||
|
||||
## Inference runtime
|
||||
|
||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference
|
||||
runs through the `moondream` Python package and its Kestrel runtime, which you
|
||||
install in your own environment; builds/tests never fetch it. See
|
||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Pinned
|
||||
file sizes/hashes and attribution live in `assets/backends/redux.json`; offline
|
||||
`--list-models`/`--check-model redux --model-dir /absolute/model` (or
|
||||
`scripts/model-status.py`) verify without inference or downloads. Manifest
|
||||
schema/verification are in `python/frameyap/model_files.py`. The local generic
|
||||
dispatcher resolves a manifest's in-release Python/executable launcher and
|
||||
checks request/reply correlation; a new manifest still needs its own licensed,
|
||||
compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and
|
||||
`--manifest-dir /absolute/manifests` are wired through
|
||||
the installed launcher as an explicit override, not a provisioning command.
|
||||
Inference uses the `moondream` Python package and Kestrel runtime, separately
|
||||
provisioned in your own environment; builds/tests/installer do not pip-install them. See
|
||||
[third-party notes](third-party.md).
|
||||
|
||||
## Developer native build
|
||||
@@ -142,12 +168,16 @@ kestrel 0.8.0) and local weights:
|
||||
```
|
||||
|
||||
Use a disposable text destination first. `--run` loads the model but does not
|
||||
record until an explicit recording control. Click Insert only after focusing your
|
||||
record until an explicit recording control. Click Type only after focusing your
|
||||
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
|
||||
and terminates only the owned worker. Installer upgrades refuse an active app.
|
||||
|
||||
The worker API is intentionally small: a fork can replace the worker implementation
|
||||
or add its own model/API integration without changing overlay and delivery code.
|
||||
The `Controller` receives injected `ControllerAudio`, `ControllerWorker`,
|
||||
`ControllerFocus` and delivery factory interfaces; hardware-free fakes can test
|
||||
state, retry, authorization and mic lifetime without initializing OpenVR or
|
||||
recording speech. The worker API is intentionally small: a fork can replace the
|
||||
worker implementation or add its own model/API integration without changing
|
||||
overlay and delivery code.
|
||||
The default product remains local-only Redux; extending a fork does not authorize
|
||||
sending existing users' audio to a service.
|
||||
|
||||
|
||||
+61
-39
@@ -1,11 +1,16 @@
|
||||
# FrameYap native dictation overlay — proposal
|
||||
# FrameYap native dictation overlay — design and remaining goals
|
||||
|
||||
Standalone project design. This document is the full target design, not a blanket
|
||||
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
|
||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer
|
||||
with explicit source-build option. Redux's pinned manifest, offline status CLI,
|
||||
manifest-driven dispatcher and in-panel Models selection/confirmation are
|
||||
implemented locally. The installed chooser/installer handoff is **not yet
|
||||
end-to-end verified on Frame**; the native-only archive needs an external CPU
|
||||
runtime for speech.
|
||||
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
|
||||
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
|
||||
acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md).
|
||||
accepted for live automatic typing on Frame. A separate status chip and full hardware
|
||||
acceptance remain possible future work. See [build and scope](build.md) and [third-party notes](third-party.md).
|
||||
Future work is tracked in [TODO.md](../TODO.md).
|
||||
|
||||
## Recommendation
|
||||
@@ -17,7 +22,9 @@ as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene
|
||||
renderer, avatar, desktop capture or root service is needed in the primary path.
|
||||
|
||||
A first-class product goal is a **one-command GitHub install without a Steam store
|
||||
AppID**. Package a prebuilt native executable and isolated CPU runtime; use a normal
|
||||
AppID**. Package a prebuilt native executable; the current native-only archive requires a
|
||||
separately supplied, compatible CPU runtime (no automatic pip install). A future
|
||||
isolated runtime bundle requires its own license and compatibility audit. Use a normal
|
||||
OpenVR application key for registration, not Steamworks. Installation must remain
|
||||
user-local with opt-in autolaunch. See [installation design](install-design.md).
|
||||
|
||||
@@ -31,12 +38,14 @@ one persistent local Parakeet Redux CPU worker
|
||||
correlated literal transcript → focus/delivery policy
|
||||
↓
|
||||
Gamescope IME set_string + commit → focused Frame application
|
||||
↘ overlay preview / error / explicit Insert when delivery is unsafe
|
||||
↘ overlay preview / error / explicit Type when delivery is unsafe
|
||||
```
|
||||
|
||||
The CPU worker is local process isolation, not remote inference or a service
|
||||
framework. Implement a small transport owned by this repository. Keep the model
|
||||
loaded between utterances; do not spawn Python/load 178 MB for every release.
|
||||
framework. The locally implemented manifest dispatcher uses a small bounded
|
||||
transport owned by this repository; only Redux inference is shipped. Keep the
|
||||
model loaded between utterances; do not spawn Python/load 178 MB for every release.
|
||||
A correlated bad transcript is a request-level failure, not a model unload.
|
||||
|
||||
## Minimal interaction
|
||||
|
||||
@@ -53,27 +62,29 @@ armed remains a possible refinement, not a second implemented overlay. See
|
||||
Recording… 00:04 [ Cancel ]
|
||||
|
||||
"The recognized text appears here."
|
||||
[ Insert + space ] [ Discard ] [ Insert + Enter — explicit ]
|
||||
[ Type + space ] [ Discard ] [ Type + Enter — explicit ]
|
||||
```
|
||||
|
||||
- States: disabled, warming, ready, recording, transcribing, review, inserted,
|
||||
unavailable/error. Recording uses visible icon + text, not color alone.
|
||||
- States: warming, ready, recording, transcribing, review, input queued,
|
||||
unavailable/error. Input queued is not an application receipt. Recording uses
|
||||
visible icon + text, not color alone.
|
||||
- Default Frame bindings: hold right X to speak, release to finish; B cancels,
|
||||
A inserts with a trailing space, Y opens/cycles the quick-chat selection.
|
||||
Overlay Submit or left-grip double-tap submits the selected literal + Enter,
|
||||
or pending review + Enter, or Enter alone if neither is present.
|
||||
The alternate right-grip tap-then-hold gesture and left-grip double-tap Submit
|
||||
A types with a trailing space, Y opens/cycles the Quick phrases selection.
|
||||
Overlay Type + Enter or left-grip double-tap submits the selected literal + Enter,
|
||||
or pending review (normally + space) then Enter, or Enter alone if neither is present.
|
||||
The alternate right-grip tap-then-hold gesture and left-grip double-tap Type + Enter
|
||||
remain available. All are remappable through the Bindings button's SteamVR editor.
|
||||
A click-to-start/stop overlay button provides
|
||||
a binding-independent alternative. Bound recording to 20 seconds; discard
|
||||
accidental taps (initial threshold: 200 ms).
|
||||
- **Quick typing (opt-in, locally implemented):** insert on completion only when
|
||||
- **Auto insert (opt-in, locally implemented):** type on completion only when
|
||||
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
|
||||
wait for Insert. Bring up review on uncertainty rather than silently losing a
|
||||
wait for Type. Bring up review on uncertainty rather than silently losing a
|
||||
transcript or typing into a new target. Live Frame acceptance remains open.
|
||||
- Submit requires its own explicit control activation: insert pending review with
|
||||
a trailing space, then queue Enter only if the text step succeeds. With no
|
||||
review, it queues only Enter. Never interpret "submit", "delete" or other speech
|
||||
- Type + Enter requires its own explicit control activation: type pending review
|
||||
(normally with a trailing space), then queue Enter only if the text step succeeds.
|
||||
At the 4096-byte bound, preserve the entire transcript without a suffix if no
|
||||
space fits. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech
|
||||
as commands. Transcription completion never auto-submits.
|
||||
- No generic "undo last dictation" initially: another application's edits/cursor
|
||||
cannot be reliably rolled back by a guessed number of backspaces.
|
||||
@@ -109,7 +120,7 @@ license-reviewed extraction, never a runtime path into another project's checkou
|
||||
|
||||
### Controller bindings
|
||||
|
||||
Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the
|
||||
Expose PTT, cancel and explicit Type/Type + Enter as named SteamVR actions; let the
|
||||
user bind them. Do not assume a scene app's left-bumper mapping works globally or
|
||||
silently takes a game's button away. Check action activity and neutral rearm.
|
||||
|
||||
@@ -143,7 +154,7 @@ The server implements text through synthetic key events and a temporary keymap,
|
||||
not a guaranteed rich-text/IME edit operation in every app. Verify actual target
|
||||
toolkits and games. Respect singleton/unavailable handling and Steam-keyboard
|
||||
coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe:
|
||||
its inspected sample loop is byte-oriented and interprets newline as Submit.
|
||||
its inspected sample loop is byte-oriented and interprets newline as Enter.
|
||||
|
||||
### 2. Explicit fallbacks, not a framework built up front
|
||||
|
||||
@@ -183,11 +194,11 @@ Gamescope exposes focus-display/window root properties, but their encoding and
|
||||
relationship to seat focus need implementation-specific validation. Do not infer
|
||||
that X display `:0` is always the destination, or that an X focus observation
|
||||
identifies a native Wayland text field. For unobservable native Wayland focus,
|
||||
require explicit review/Insert; do not advertise safe auto-targeting.
|
||||
require explicit review/Type; do not advertise safe auto-targeting.
|
||||
|
||||
If focus changes, keep the result in review. A fresh Insert explicitly approves
|
||||
If focus changes, keep the result in review. A fresh Type explicitly approves
|
||||
the current destination and creates a new delivery authorization. Recheck again
|
||||
at insertion. This minimizes stale delivery but does **not** eliminate a race
|
||||
at typing. This minimizes stale delivery but does **not** eliminate a race
|
||||
between the final check and global input processing; do not claim otherwise.
|
||||
A Wayland roundtrip means compositor processing, not application consumption.
|
||||
Report `input queued`, never `message sent`.
|
||||
@@ -221,9 +232,11 @@ that tiny dataset; keep transcript visibility and a cheap retry.
|
||||
|
||||
Implement a small independent audio/worker adapter with explicit capture,
|
||||
single-request bounds, owner-only runtime files, correlated replies and cancellation.
|
||||
The worker should be implemented independently; do not link, vendor or import
|
||||
another application's speech code. Avoid a generic provider framework: one
|
||||
explicit Redux worker is enough for the first version.
|
||||
The worker is independently implemented; do not link, vendor or import another
|
||||
application's speech code. A narrow manifest dispatcher now selects a checked
|
||||
local backend launcher; this is not a general cloud/provider framework and only
|
||||
Redux is supplied with a runtime implementation. Other manifests require their
|
||||
own audited offline engine, launcher and tests.
|
||||
|
||||
Suggested ownership, introduced only as implementation needs it:
|
||||
|
||||
@@ -243,12 +256,20 @@ processing timeout. No shell commands in IPC and no input authority in the worke
|
||||
native kernel pools. Choose measured latency versus compositor contention,
|
||||
not the desktop's thread count by habit. No real-time scheduling or permanent
|
||||
CPU pinning initially; inspect runtime affinity behaviour during measurement.
|
||||
- Explicit local model path and offline loading. Missing runtime/weights produces
|
||||
an actionable error, not an unsolicited download/network fallback.
|
||||
- Optional explicit enable/warm-up before first PTT; warming must not record audio.
|
||||
Expose that lifecycle explicitly rather than warming on import or construction.
|
||||
Otherwise display first-use loading honestly. Keep
|
||||
model reuse after normal completion; cancellation may restart the owned worker.
|
||||
- Explicit local model path and offline loading. Pinned files and attribution live
|
||||
in `assets/backends/redux.json`; `python/frameyap/model_files.py` validates its
|
||||
schema and hashes. The CLI's offline `--list-models`/`--check-model` checks
|
||||
never start inference. Settings → Models selects/saves the manifest ID and
|
||||
restarts the worker, invalidating any audio/review/delivery authorization.
|
||||
Install requires a separate confirmation showing source, size, license,
|
||||
attribution and exact-manifest SHA-256; only that click hands off to the
|
||||
installer for pinned model files. Missing runtime/weights produces an
|
||||
actionable error, not an unsolicited download/network fallback.
|
||||
- Native `--run` explicitly starts offline verification then warms a verified
|
||||
selected local model before PTT; warming must not record audio. Missing or
|
||||
unverified files disable Record rather than prompting a background download.
|
||||
Keep model reuse after normal completion and request-local transcript errors;
|
||||
cancellation may restart the owned worker.
|
||||
- Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded
|
||||
tmpfs-backed clips; remove them on completion, error, cancellation and shutdown.
|
||||
Avoid persistent audio/transcripts by default. Local IPC is not a network hop;
|
||||
@@ -278,14 +299,14 @@ These are proposed implementation gates, **not completed acceptance**:
|
||||
confirm dashboard/hand placement, input events, close/reopen and no scene-focus
|
||||
takeover. Validate global PTT separately rather than blocking the clickable
|
||||
prototype on experimental override support.
|
||||
3. **Real dictation path:** microphone → local Redux → preview → explicit insert
|
||||
into a disposable target; then enable quick typing after target tracking tests.
|
||||
3. **Real dictation path:** microphone → local Redux → preview → explicit Type
|
||||
into a disposable target; then validate opt-in Auto insert after target tracking tests.
|
||||
Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate
|
||||
replies, lost mic, worker crash and missing model without persisting speech.
|
||||
4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game
|
||||
text fields; Steam keyboard coexistence; native Wayland targets separately.
|
||||
Test focus changes during capture/inference, rapid loss/regain, held modifiers,
|
||||
explicit Insert retargeting, no hidden Enter, and no second delivery.
|
||||
explicit Type retargeting, no hidden Enter, and no second delivery.
|
||||
5. **In-headset acceptance:** readable feedback and comfortable PTT; measured
|
||||
release-to-insert latency and compositor timing while an actual scene runs;
|
||||
no noticeable sustained thermal/battery regression. Set numeric budgets after
|
||||
@@ -301,8 +322,9 @@ without moving recognition off Frame. It is not part of this initial design.
|
||||
The default hardware-free build needs only CMake and a C++20 compiler (Python
|
||||
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
|
||||
SDL3, FreeType and Wayland client/generated protocol bindings. A separately
|
||||
authorized Python Redux environment is explicitly supplied at launch. Pin revisions
|
||||
and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery.
|
||||
authorized Python Redux environment is explicitly supplied at launch; the
|
||||
native-only installer neither bundles it nor pip-installs one. Pin revisions
|
||||
and review licenses for each introduced dependency. No automatic fetch/install in configure or normal tests; no external checkout discovery.
|
||||
|
||||
Hardware-free tests should cover state transitions, bounded PCM/transcripts,
|
||||
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
|
||||
|
||||
+73
-99
@@ -1,108 +1,82 @@
|
||||
# Installation and distribution goal
|
||||
# Installation goal and current boundary
|
||||
|
||||
**User goal:** install from GitHub with a `curl … | bash`-style command, without a
|
||||
Steam store AppID. An idempotent archive installer and native-only local artifacts
|
||||
are now implemented/tested; no public release is published. A bundled-ASR
|
||||
experience is not yet offered. This document retains the target
|
||||
design; see [current packaging](packaging.md) and [third-party notes](third-party.md).
|
||||
Planned installer work (binary-or-source choice, flag-driven operation) is in
|
||||
[TODO.md](../TODO.md).
|
||||
Goal: a one-command, pinned GitHub release install for Steam Frame without a Steam
|
||||
store AppID, sudo or end-user compiler. **No public archive or verified clean
|
||||
install is published. Do not advertise a `curl | sh` command as functional.**
|
||||
The local installer has binary-archive and explicitly provisioned source-build
|
||||
modes, machine-readable plans/results and an attended TTY path. The current
|
||||
native-only artifact does not include or pip-install an ASR runtime; it is not
|
||||
a one-command voice-typing experience. See [packaging](packaging.md) for exact
|
||||
flags and [third-party inventory](third-party.md) for open license/ABI audits.
|
||||
|
||||
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable
|
||||
for the **proposed** Linux ARM64 glibc package format and has the expected basic
|
||||
bootstrap utilities. It reports system Python, Git and uv, but none is required
|
||||
for the intended bundled release. A read-only check on one Frame observed
|
||||
Python 3.12.3 and Git, but not uv; availability may change. This script
|
||||
is not an installer or a model/runtime compatibility test; it has no downloads,
|
||||
registration, SteamVR initialization or persistent changes. No glibc minimum can
|
||||
be certified until release artifacts are chosen and tested.
|
||||
`scripts/install-preflight.sh` is a read-only Linux ARM64/glibc/bootstrap check;
|
||||
`--source` adds toolchain/library checks. It does not download, install, register,
|
||||
or certify model/runtime compatibility or a minimum libc version. Its report of
|
||||
Python/Git/uv availability is not a runtime guarantee: `install.sh` currently
|
||||
needs Python 3.12+ **for its bootstrap**, while native-only voice inference needs
|
||||
a separately provisioned compatible CPU Python environment and pinned weights.
|
||||
|
||||
## Non-Steam overlay identity
|
||||
## OpenVR identity
|
||||
|
||||
OpenVR overlay applications do not require a Steam store AppID or Steamworks.
|
||||
The native executable initializes as `VRApplication_Overlay`. For discoverability
|
||||
and optional autolaunch, register an OpenVR application manifest with a stable,
|
||||
project-owned **string application key** (proposed: `local.frameyap.overlay`).
|
||||
That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam
|
||||
library shortcut should be necessary for the normal route.
|
||||
`local.frameyap.overlay` is a string OpenVR application key, **not** a Steam
|
||||
store AppID. The installer creates a manifest/desktop launcher user-locally but
|
||||
does not register/launch the app by default. Explicit `frameyap --register
|
||||
/absolute/manifest/path` uses the OpenVR registration API; registration alone
|
||||
did not reveal a launcher in the first checked dashboard menu. On one Frame the
|
||||
user opened the panel from the **Non-Steam** section and quit; shortcut discovery
|
||||
and persistence after a normal restart are still unverified. Optional
|
||||
`--autolaunch`/`--no-autolaunch` on the installer explicitly request OpenVR
|
||||
registration/autolaunch choices; no SteamVR settings or sessions are changed
|
||||
without that request. Unregister explicitly before uninstall. SteamVR must
|
||||
already be available for registration; never start/restart it for installation.
|
||||
|
||||
The manifest identifies the installed executable; the installer/registration
|
||||
helper should use `IVRApplications::AddApplicationManifest` and the corresponding
|
||||
remove operation, not hand-edit Steam's internal JSON. Autolaunch uses the OpenVR
|
||||
application setting only when explicitly requested. Validate the exact manifest,
|
||||
launch behaviour, registration persistence and uninstall on native Frame before
|
||||
claiming this route works end to end. Existing probes established overlay client
|
||||
initialization, not manifest installation.
|
||||
## Installation workflows
|
||||
|
||||
SteamVR/OpenVR must already be installed and usable. If registration needs a
|
||||
running runtime, defer it to the first explicit launch rather than starting or
|
||||
restarting SteamVR behind the user's back.
|
||||
- **Binary mode** (no compiler): verify a locally supplied ARM64 archive digest,
|
||||
or, after a vetted release actually exists, explicitly approve retrieval of
|
||||
a pinned `v0.1.YYYYMMDDHHMM` GitHub tag and checksum. No moving `latest` tag.
|
||||
`--mode binary --archive FILE --sha256 HASH --version 0.1.YYYYMMDDHHMM`
|
||||
selects the local-artifact path.
|
||||
- **Source mode**: requires explicit local source, SDK, SDL/OpenVR libraries and
|
||||
their notices, CMake/C++20, native build dependencies and a version. It builds,
|
||||
stages, packages and continues through local installation; it does **not**
|
||||
provision ASR packages or bypass producer license obligations. See
|
||||
[packaging](packaging.md) for all flags.
|
||||
- **Model**: only an explicit `--install-model --backend redux --yes` fetches
|
||||
pinned public files for the *already installed* backend. Inspect the read-only
|
||||
`--print-plan --json` first; it includes model size, attribution and license
|
||||
metadata from `assets/backends/redux.json`. `--expected-manifest-sha256 HASH`
|
||||
binds consent to the exact installed manifest bytes and fails before model
|
||||
directory creation/network if they changed. The local in-panel chooser shows
|
||||
source, size, license text, attribution and manifest digest, then requires a
|
||||
second **Confirm Install** click; the installed/native UI route still needs
|
||||
clean-target and headset acceptance. `--without-model` permits an
|
||||
archive install without bundled model files. Neither operation installs Torch,
|
||||
moondream, Kestrel or an interpreter. Launch paths to an independently
|
||||
authorized runtime/model can be set in `paths.conf`.
|
||||
- **Noninteractive**: supply flags and `--yes` for network/model consent.
|
||||
`--print-plan` is read-only; `--json` provides structured results/errors and
|
||||
progress events for a model download. No prompt reads stdin in a pipe. An
|
||||
empty TTY invocation offers a local menu and prints equivalent flags.
|
||||
|
||||
## Intended user experience
|
||||
Installation is user-local under XDG data/config paths with a managed launcher,
|
||||
retained rollback, SHA-256/path validation, foreign-file refusal and a lock
|
||||
shared with the app. No OS package changes, udev rule, root service, Steam store
|
||||
listing, unrelated application dependency, microphone recording, input injection
|
||||
or automatic update daemon. Hashes detect accidental/unauthorized alteration
|
||||
of a downloaded artifact but do not authenticate a compromised publisher;
|
||||
release metadata needs independent trust. No installer operation silently runs
|
||||
pip or launches inference. Native-only archives support overlay checks but need
|
||||
an externally provisioned runtime/weights before voice typing. Selecting an
|
||||
uninstalled backend does not authorize a download or supply its inference engine.
|
||||
|
||||
1. Run one documented command from the eventual GitHub repository/release.
|
||||
2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and
|
||||
explains/downloads the application, compatible CPU runtime and pinned model.
|
||||
3. User-local installation provides a simple `frameyap` launcher, desktop
|
||||
entry where supported, and an OpenVR manifest. No compiler, engine checkout,
|
||||
Python dependency troubleshooting or separate ASR server for ordinary users.
|
||||
4. The user explicitly launches/enables dictation. No installation-time microphone
|
||||
recording, input injection, inference benchmark or overlay takeover.
|
||||
## Gate before publishing the goal as fulfilled
|
||||
|
||||
Illustrative command shape only; `OWNER`, `REPO` and `VERSION` are placeholders:
|
||||
|
||||
```sh
|
||||
curl --fail --silent --show-error --location \
|
||||
https://raw.githubusercontent.com/OWNER/REPO/VERSION/install.sh | bash
|
||||
```
|
||||
|
||||
Also document a download-inspect-run path for users who do not want to pipe remote
|
||||
code into a shell. Pin a release/tag instead of executing a moving branch by default.
|
||||
Offer explicit version selection and noninteractive flags; do not read interactive
|
||||
confirmation from stdin while the installer itself is arriving through that pipe.
|
||||
|
||||
## Packaging boundary
|
||||
|
||||
- Prebuilt ARM64 executable plus a known-compatible, isolated CPU inference runtime;
|
||||
no external application libraries/assets and no system Python modification.
|
||||
- Model fetched during explicit installation/setup, with pinned revision/hash and
|
||||
attribution. A documented `--without-model` option can defer the large download.
|
||||
No surprise first-utterance downloads. Runtime components are installed from
|
||||
their own package index rather than redistributed in our tarball.
|
||||
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
|
||||
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
|
||||
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
|
||||
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
|
||||
`/dev/uinput` permission changes or modifications to unrelated launchers.
|
||||
- No Steam store AppID, Steamworks SDK, root service or network ASR dependency.
|
||||
- Third-party notices for everything we redistribute must be included with a release.
|
||||
|
||||
## Installer lifecycle and safety
|
||||
|
||||
- Detect architecture, libc and prerequisites first. Unsupported hosts fail with a
|
||||
clear explanation; never install an x86 payload silently on ARM64.
|
||||
- Download to a staging directory, check versioned SHA-256 manifests and archive
|
||||
paths, then atomically select the completed version. HTTPS/checksums alone do not
|
||||
authenticate a compromised publisher; use signed release metadata if provided.
|
||||
- Keep configuration across upgrades; retain the previous version for rollback.
|
||||
Refuse or defer replacement while this application's process is running rather
|
||||
than killing arbitrary processes. Never touch SSH or unrelated sessions.
|
||||
- Autostart is opt-in (`--autostart` or explicit settings); do not enable a systemd
|
||||
service or SteamVR autolaunch by default. Do not enable overlay input overrides.
|
||||
- Uninstall removes only owned launcher, manifest registration and install files;
|
||||
model/config deletion is separately explicit. Preserve other SteamVR apps.
|
||||
- No update daemon initially. A deliberate rerun/update command is sufficient.
|
||||
|
||||
## Acceptance before advertising one-command installation
|
||||
|
||||
- Clean supported Frame: install without sudo/compiler/engine checkout/store AppID;
|
||||
launch overlay, load local Redux and type into an owned disposable target.
|
||||
- Normal use after installation needs no network connection or desktop ASR host.
|
||||
- Failed download/hash, unsupported architecture, low disk space and interrupted
|
||||
upgrades leave a usable previous install or a cleanly reported failure.
|
||||
- Reinstall, rollback and uninstall preserve unrelated data and SteamVR entries.
|
||||
- Noninteractive piped invocation never hangs on stdin; inspection-first path works.
|
||||
- Autolaunch remains off unless chosen; uninstall removes only our registration.
|
||||
- Hardware-free installer tests use temporary homes, mocked runtime registration
|
||||
and local fixture artifacts. Never exercise a real user's Steam configuration
|
||||
in ordinary CI/CTest.
|
||||
Vet the **exact** release closure/licenses, ARM64 symbol versions/loader,
|
||||
model attribution and compatible CPU Python environment; establish a tested
|
||||
libc/runtime floor, then publish and authenticate a checksummed archive from a
|
||||
clean tag. On a clean supported Frame, install without a compiler/sudo/store ID,
|
||||
load Redux, type into a disposable owned target and validate rollback/uninstall,
|
||||
foreign-file failures, autolaunch off, and no unintended session changes. Separately
|
||||
validate microphone → review → delivered input and headset comfort. Offline
|
||||
installer fixture tests and a local native build are not those acceptance gates.
|
||||
+89
-31
@@ -2,10 +2,12 @@
|
||||
|
||||
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
|
||||
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
|
||||
Native `--run` wiring in `src/main.cpp` is implemented behind the explicit
|
||||
`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful
|
||||
compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a
|
||||
normal build or test.
|
||||
Native `--run` wiring in `src/main.cpp` is behind the explicit
|
||||
`FRAMEYAP_NATIVE` build option. The Models chooser, offline verification and
|
||||
explicit installer handoff are implemented locally, **not** an accepted Frame
|
||||
install/voice-typing path or a published release. Neither hardware-free tests
|
||||
nor a successful compile establish Frame input, visibility, comfort or text
|
||||
delivery. Launching the runtime is explicit, never part of a normal build or test.
|
||||
|
||||
## Rendering and controls
|
||||
|
||||
@@ -49,7 +51,9 @@ This replaces the raw-upload rendering path; headset flicker acceptance still
|
||||
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
|
||||
|
||||
The header shows local time and date instead of the former on-device/review
|
||||
and current-mount labels. It updates when the displayed minute or date changes,
|
||||
and current-mount labels. Settings explains Hold Quit (hold 0.9 seconds then
|
||||
release) and Lasers anytime (system-wide lasers may affect games). The review
|
||||
tab describes Type and Type + Enter. It updates when the displayed minute or date changes,
|
||||
not every frame. Settings toggles 12/24-hour time and cycles date Off →
|
||||
MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display;
|
||||
mount choices remain in Settings.
|
||||
@@ -58,23 +62,25 @@ The complete transcript preview is paginated by glyph width and four-line
|
||||
height; Previous and Next navigate it without changing the source transcript.
|
||||
Status fits on the single status line; the old bottom detail label is gone.
|
||||
The footer remains available on all tabs: Record (labelled Stop while recording),
|
||||
Cancel, Insert, Submit, Hold Quit. Hold Quit needs a 900 ms press and release on
|
||||
Cancel, Type, Type + Enter, Hold Quit. Hold Quit needs a 900 ms press and release on
|
||||
that same button; its thin progress bar shows the hold. Record can retry after an
|
||||
error; it is disabled while warming/transcribing and until an existing review is
|
||||
inserted or discarded. Cancel can stop worker startup. Insert and Submit are
|
||||
typed or discarded. Cancel can stop worker startup. Type and Type + Enter are
|
||||
disabled during recording and transcription. A pointer action requires
|
||||
a press/release on the same enabled control from the same cursor; focus loss,
|
||||
tab changes, action-state changes and relocation clear pending presses. Submit is *always* a separate
|
||||
deliberate action, not inferred from text. Insert appends a trailing space (without
|
||||
doubling an existing trailing space). Submit inserts any pending review and then
|
||||
queues Enter; with no pending text it queues Enter only. Y opens the quick-chat
|
||||
list over the review area; each further Y press cycles its highlighted choice.
|
||||
Cancel closes the picker without discarding an existing review. Submit sends the
|
||||
selected text *without* a trailing space, then Enter. The choices are short
|
||||
tab changes, action-state changes and relocation clear pending presses. Type + Enter
|
||||
is *always* a separate deliberate action, not inferred from text. Type normally
|
||||
appends a trailing space (without doubling an existing one); a full 4096-byte
|
||||
transcript without room for that suffix is queued unchanged, with no extra error
|
||||
for the missing space. Type + Enter types any pending review and then queues
|
||||
Enter; with no pending text it queues Enter only. Y opens the Quick phrases
|
||||
list over the review area; each further Y press cycles its highlighted choice. Cancel closes the picker without discarding an
|
||||
existing review. Type + Enter sends the selected phrase *without* a trailing
|
||||
space, then Enter. The choices are short
|
||||
single-line literals, not speech commands. A failed text step never proceeds to
|
||||
Enter. Recording never automatically submits. Auto insert, when
|
||||
explicitly enabled, can queue text + space after transcription only under the
|
||||
stable Xwayland focus guard described below.
|
||||
explicitly enabled, can queue text (normally + space) after transcription only
|
||||
under the stable Xwayland focus guard described below.
|
||||
|
||||
### Bindings button
|
||||
|
||||
@@ -107,6 +113,8 @@ installer creates one with defaults on first install. Copy the shipped
|
||||
"input_priority": "normal",
|
||||
"advanced_debug": false,
|
||||
"auto_insert": false,
|
||||
"close_mic_when_idle": false,
|
||||
"backend": "redux",
|
||||
"lock_layout": false,
|
||||
"clock_24h": false,
|
||||
"date_format": "mdy",
|
||||
@@ -149,6 +157,18 @@ config, retaining other fields and formatting; invalid/unwritable configs are
|
||||
left untouched and return failure. The installer backs up original bytes before
|
||||
repairing invalid values, while valid `true` and `false` are retained.
|
||||
|
||||
`close_mic_when_idle` is a separate boolean, default **false**, also available
|
||||
as Settings → **Close mic when idle**. Normally the SDL capture device stays
|
||||
open while Ready and idle samples are discarded. Enabling this toggle closes it
|
||||
between clips and opens it on PTT: this avoids an open idle capture device but
|
||||
can cause an audio spike (observed with per-PTT transitions on Frame), startup
|
||||
latency or first-syllable clipping. Settings persistently displays **OFF: discard
|
||||
idle audio; ON: spike / start latency** below the toggle, as well as an ON/OFF
|
||||
indicator; the status/detail line also explains a change when toggled. The
|
||||
native setting is saved to `config.json`; a failed save applies only for this
|
||||
session and warns. This is not a mute switch for other applications. Quit and
|
||||
failure still close the device.
|
||||
|
||||
`auto_insert` is a separate boolean, default `false`, also available as a
|
||||
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
|
||||
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
||||
@@ -156,41 +176,77 @@ display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
||||
Both properties and focus are rechecked after IME lease acquisition. A watched
|
||||
focus-out, root focus-property change (even if the same window returns), window
|
||||
destruction, held keyboard key, missing X display or any disagreement permanently
|
||||
disarms that clip. The transcript then remains for explicit review/Insert.
|
||||
disarms that clip. The transcript then remains for explicit review/Type.
|
||||
Native Wayland focus and child text-field focus cannot be safely inferred here;
|
||||
those cases fall back to review. No automatic Enter, speech commands or retry.
|
||||
The compositor can still change focus in the gap between the final check and
|
||||
global delivery, and IME commit is not an application receipt. This path has
|
||||
offline synthetic focus tests and a separate owned-target IME fixture; live
|
||||
speech-driven Auto Insert, target coverage and headset acceptance remain
|
||||
speech-driven Auto insert, target coverage and headset acceptance remain
|
||||
unverified. The setting is preserved on upgrade
|
||||
and a failed preference write applies only to the current session.
|
||||
|
||||
`quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings,
|
||||
each at most 64 characters. Edit the JSON file and restart; there is no headset
|
||||
text editor. Inputs are literal (not expanded or interpreted by FrameYap) and
|
||||
are sent to the current Gamescope focus, so check the destination before Submit.
|
||||
are sent to the current Gamescope focus, so check the destination before Type + Enter.
|
||||
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
|
||||
`insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME`
|
||||
button paths. Omitted actions retain their bundled defaults; an empty string
|
||||
disables a mapping, including after an upgrade. The Frame defaults are right
|
||||
X = hold-to-talk, B = Cancel, A = Insert + space, Y = quick chat. Enter has no
|
||||
single-button mapping by default; the left grip double-tap still submits.
|
||||
An existing config mapping `enter` to right Y is migrated to quick chat in memory
|
||||
X = hold-to-talk, B = Cancel, A = Type + space, Y = Quick phrases. Enter has no
|
||||
single-button mapping by default; the left grip double-tap still requests Type + Enter.
|
||||
An existing config mapping `enter` to right Y is migrated to Quick phrases in memory
|
||||
when `quick_chat` is absent; this does not overwrite custom mappings.
|
||||
Existing configs with empty actions retain those disabled mappings; change them explicitly
|
||||
or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized;
|
||||
SteamVR user overrides may still supersede it. On customized launches a generated
|
||||
action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings`
|
||||
(or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The
|
||||
config is read once at launch, not hot-reloaded. On install/upgrade the installer
|
||||
fills missing fields, removes retired keys, and resets invalid entries. It saves
|
||||
the exact prior bytes under `config.json.backup-*` before a repair and refuses
|
||||
symlink/oversized config paths; valid customizations remain intact. The installed
|
||||
launcher no longer pins `--font`, so this selection takes effect. Direct native
|
||||
config is read once at launch, not hot-reloaded (the in-panel backend selection
|
||||
is saved separately). On install/upgrade the installer fills known missing fields,
|
||||
including `close_mic_when_idle` and `backend`, removes retired keys and resets
|
||||
invalid entries. It saves the exact prior bytes under `config.json.backup-*`
|
||||
before a repair and refuses symlink/oversized config paths; valid customizations
|
||||
remain intact. The installed launcher no longer pins `--font`, so this selection takes effect. Direct native
|
||||
launches with bad JSON, colors or button mappings fail startup rather than
|
||||
silently changing input behavior.
|
||||
|
||||
### Models / backends (local implementation)
|
||||
|
||||
Settings → **Models / backends** opens a paginated local chooser. The saved
|
||||
`"backend": "redux"` selects a *manifest ID*, not a model download;
|
||||
`--backend ID` overrides it for that `--run` invocation. Selecting a listed backend
|
||||
restarts the owned worker and closes capture, discarding pending audio, review
|
||||
and any prior focus/delivery authorization. A failed preference write leaves the
|
||||
choice active only for this session. The currently listed Redux model is the
|
||||
only inference implementation supplied; adding a manifest alone does not add
|
||||
an inference runtime. The Models tab labels local checks as checking, missing
|
||||
(`not_installed`), invalid, or installed/verified; for the selected model it
|
||||
also reports loading, ready or failed. An install-in-progress note reports
|
||||
model provisioning, followed by a new offline check; a verified status alone
|
||||
is not proof the CPU runtime loaded. Recording is disabled until the selected
|
||||
files verify offline and the worker warms; missing files never trigger a silent
|
||||
download. Model loading and the default open-while-Ready idle microphone policy
|
||||
are unchanged. The panel's state and successful helper calls do **not** prove
|
||||
microphone transcription, input delivery or headset acceptance.
|
||||
|
||||
For a missing/invalid selected model, **Install** first presents a separate
|
||||
confirmation showing pinned source, approximate download size, license text,
|
||||
attribution and the exact raw manifest SHA-256 fingerprint. Only **Confirm
|
||||
Install** launches the local installer with that fingerprint; leaving the view
|
||||
or changing metadata invalidates consent. The installer rechecks the *installed*
|
||||
manifest bytes under its model lock before creating a download target or using
|
||||
the network, downloads only on the explicit click, hashes pinned files and then
|
||||
the app checks them again offline before enabling recording. Existing invalid
|
||||
or unsafe model files are refused rather than silently overwritten. This
|
||||
installs model files, **not** Python, Torch, moondream, Kestrel or other runtime
|
||||
dependencies.
|
||||
The source-tree UI needs an installed release for installer-backed provisioning;
|
||||
a hand-edited manifest is not an approved artifact. The installer handoff and
|
||||
native UI have not yet been accepted on a clean Frame. See
|
||||
[packaging](packaging.md#consumer) for an inspection-first CLI path.
|
||||
|
||||
### Experimental controller input priority
|
||||
|
||||
There are two independent gates:
|
||||
@@ -348,7 +404,7 @@ recentring and readability still require a separately authorized headset check.
|
||||
|
||||
The opt-in native `--check-controls` probe logs pointer counters and action
|
||||
callbacks to the terminal rather than repainting them on the panel. Its canvas
|
||||
stays static for Record/Cancel/Insert/Enter clicks so those clicks can be checked
|
||||
stays static for Record/Cancel/Type/Type + Enter clicks so those clicks can be checked
|
||||
without diagnostic texture uploads. Switching tabs or mount still updates
|
||||
the visible panel. Diagnostics identify `renderer=Vulkan` and count
|
||||
`textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions.
|
||||
@@ -367,8 +423,10 @@ cmake --build build-native --target frameyap_texture_check
|
||||
|
||||
It does not initialize OpenVR or establish compositor/headset acceptance.
|
||||
|
||||
`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert,
|
||||
Enter. `bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
|
||||
`assets/actions.json` names seven actions: left/right grip, PTT, cancel,
|
||||
Type, Type + Enter and Quick phrases. `insert`, `enter`, and `quick_chat` remain
|
||||
internal binding keys; visible controls read Type, Type + Enter, Quick phrases.
|
||||
`bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
|
||||
the grip bindings remain for optional remapping/diagnosis. In one dashboard
|
||||
probe grips were inactive; a later controls-only probe delivered repeated right
|
||||
X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions
|
||||
@@ -384,7 +442,7 @@ second squeeze **down** within 350 ms starts capture; hold as long as needed
|
||||
action explicitly begins on down and ends on up. On tracking-pose invalidity,
|
||||
action inactivity or overlay focus loss, a held capture emits Cancel, and
|
||||
reconnection requires a neutral observation before any new press. PTT and left
|
||||
Enter require an enabled panel; clickable Record remains available for retry
|
||||
Type + Enter require an enabled panel; clickable Record remains available for retry
|
||||
after an error and Cancel is always available. The action set defaults to normal
|
||||
priority; the experimental config request is described above. Neither priority
|
||||
guarantees delivery while a game or dashboard owns input.
|
||||
|
||||
+89
-26
@@ -11,53 +11,71 @@ build with `FRAMEYAP_NATIVE=ON`, and stage this layout (regular files, no links)
|
||||
|
||||
```
|
||||
bin/frameyap
|
||||
lib/* # compatible bundled native libraries
|
||||
assets/actions.json # and adjacent controller binding JSON
|
||||
bin/install.sh # managed installer helper when required by layout
|
||||
lib/* # explicitly supplied compatible native libraries
|
||||
assets/actions.json # plus binding JSON and backends/redux.json
|
||||
fonts/font.ttf
|
||||
python/frameyap/*.py
|
||||
scripts/model-status.py # offline CLI verifier; also backend-service.py, fetch-model.py
|
||||
licenses/THIRD_PARTY_NOTICES.txt
|
||||
model/* # optional pinned public weights + attribution
|
||||
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
|
||||
```
|
||||
|
||||
For the current **external-runtime** package, `scripts/stage-native-poc.py --help`
|
||||
documents explicit inputs. It invokes `cmake --install` on an existing native build,
|
||||
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
||||
The stage copies the self-contained installer into `bin/` and CMake installs
|
||||
`scripts/model-status.py` alongside the backend manifests. The package allowlist
|
||||
permits `scripts/`; installed `frameyap --list-models` expects the verifier at
|
||||
`../scripts/model-status.py`. **No actual staged archive has been audited/tested
|
||||
from a clean account for publication**; exercise the entire producer pipeline
|
||||
and the installed CLI before treating the payload layout as release-ready.
|
||||
|
||||
For the current **external-runtime** package, `scripts/stage-native.py --help`
|
||||
documents explicit inputs. The old `scripts/stage-native-poc.py` is retained as
|
||||
a deprecated migration wrapper for that command, not the documented or shipped
|
||||
staging interface. The stage invokes `cmake --install` on an existing native
|
||||
build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
||||
not build, download, run the app, or copy an ASR runtime. The native
|
||||
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
||||
libstdc++ and glibc; audit `ldd` on the installed binary.
|
||||
libstdc++ and glibc. Audit the actual staged ARM64 binaries' `NEEDED`,
|
||||
`GLIBC_*`/`GLIBCXX_*` symbol versions, ELF interpreter and notices; then test
|
||||
on a clean target. No compatible libc floor is yet established.
|
||||
SDL/OpenVR resolve inside its own `lib/`, not a producer
|
||||
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
|
||||
|
||||
```sh
|
||||
python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \
|
||||
--version 2026-09-24T162712Z-g417f81c --arch linux-aarch64 \
|
||||
--version 0.1.202609241627 --arch linux-aarch64 \
|
||||
--model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime
|
||||
```
|
||||
|
||||
Use the actual binary's UTC build stamp (`frameyap --version`) for the archive
|
||||
tag, not this illustrative timestamp. CMake generates `YYYY-MM-DDTHHMMSSZ`
|
||||
at configuration time (plus `-gSHORTSHA` for a Git checkout and `-dirty` for
|
||||
uncommitted tracked changes); producers may pin `-DFRAMEYAP_VERSION=...` to
|
||||
embed a vetted release stamp. A timestamp distinguishes same-day archives;
|
||||
the installer still refuses a reused tag whose contents have changed. Historic
|
||||
`v0.1.0-poc*` local artifacts remain valid for reinstall/rollback.
|
||||
Use the actual binary's numeric `MAJOR.MINOR.YYYYMMDDHHMM` UTC version
|
||||
(currently `0.1`), not this illustrative value, for `--version` and filename;
|
||||
tag a vetted clean tree as `vVERSION`. CMake generates this stamp at
|
||||
configuration time (`SOURCE_DATE_EPOCH` may supply it);
|
||||
`-DFRAMEYAP_VERSION=0.1.YYYYMMDDHHMM` can pin it. Development `--version`
|
||||
may print a *separate* `git HASH` line, with `(uncommitted changes)` only if
|
||||
dirty: that line is not part of the version, tag or archive name. The installer
|
||||
rejects a reused tag with different contents; historic local versions may
|
||||
remain selectable for rollback, not as new releases.
|
||||
|
||||
`--external-runtime` refuses a runtime directory and records
|
||||
`runtime: external-authorized-python` in `release.json`. The installer explicitly
|
||||
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
|
||||
in the archive. Staging validation is not an inference test.
|
||||
|
||||
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz`
|
||||
plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal,
|
||||
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing
|
||||
`HASH FILENAME`. Archive extraction rejects traversal,
|
||||
links/special files, duplicate members, oversized metadata/payloads and invalid
|
||||
layout. Checksums detect corruption, not a malicious/compromised publisher;
|
||||
authenticate release metadata independently. No packaging/installation model fetch.
|
||||
|
||||
## Consumer
|
||||
|
||||
Bootstrap: Linux ARM64/glibc, Python 3.12+, curl, sha256sum and tar. **No compiler,
|
||||
sudo, Steam store AppID or engine checkout.** Download/inspect a pinned installer
|
||||
Bootstrap for a binary archive: Linux ARM64/glibc, Python 3.12+ (installer
|
||||
bootstrap, **not** the inference runtime), curl for network release downloads,
|
||||
sha256sum and tar for archive handling. **No compiler, sudo, Steam store AppID
|
||||
or engine checkout in binary mode.** No minimum glibc floor has been certified;
|
||||
preflight alone cannot guarantee compatibility. Download/inspect a pinned installer
|
||||
before running it. Current local artifact route:
|
||||
|
||||
```sh
|
||||
@@ -65,17 +83,62 @@ sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
||||
--sha256 64_HEX_DIGIT_HASH --version VERSION
|
||||
```
|
||||
|
||||
After an actual vetted release exists, `sh install.sh --version TAG` retrieves
|
||||
that GitHub release and its versioned checksum; no `latest` or moving-branch
|
||||
After an actual vetted release exists, use a real numeric version (for example,
|
||||
`sh install.sh --mode binary --version 0.1.202609241627 --yes`); the installer
|
||||
will retrieve tag `v0.1.202609241627` and its versioned checksum; no `latest` or moving-branch
|
||||
lookup. A pipe invocation is supported, never prompts on stdin, and must also
|
||||
pin a real published tag. **There is no functional public download command yet.**
|
||||
|
||||
`--without-model` omits bundled model files from staging, never deletes a current
|
||||
model on rerun, and records the choice. Same digest/version/choice is idempotent
|
||||
and repairs missing managed wrappers. Different digest or model choice for the
|
||||
same version is refused. External model provisioning is always deliberate.
|
||||
`--without-model` omits any model files in the selected archive, never deletes
|
||||
an existing current model on rerun, and records the choice. Same
|
||||
digest/version/choice is idempotent and repairs missing managed wrappers. Different digest or model choice for the
|
||||
same version is refused. External model provisioning is always deliberate: the
|
||||
local installer accepts `sh install.sh --install-model --backend redux --yes`
|
||||
(optional `--model-dir /absolute/path`) to explicitly download and verify files from the
|
||||
installed pinned manifest, not an ASR runtime. Inspect the model/size first
|
||||
with `sh install.sh --install-model --backend redux --print-plan --json`.
|
||||
`--expected-manifest-sha256 HASH` additionally binds consent to the exact raw
|
||||
installed `redux.json` bytes: under the model lock a mismatch fails **before**
|
||||
a model directory is created or any network request. In the locally implemented
|
||||
Models UI, Install displays source, rounded size, license text, attribution and
|
||||
the fingerprint; only the second Confirm Install click passes that fingerprint
|
||||
through the backend helper to the installer. Installation alone does not
|
||||
provision a CPU Python runtime; no in-panel flow has been accepted on Frame.
|
||||
In the source tree, `python3 scripts/model-status.py --list-models` or
|
||||
`--check-model redux --model-dir /absolute/model` hash-checks local files;
|
||||
the native `frameyap --list-models` / `--check-model` entry points use the
|
||||
adjacent installed verifier script; verify this in a staged archive before release. Manifest schema,
|
||||
source, size/hash and attribution live in `assets/backends/redux.json` and are
|
||||
validated by `python/frameyap/model_files.py`.
|
||||
|
||||
The installed launcher defaults to `--run`. For a native-only package, supply
|
||||
For an explicit local **source** install, e.g.:
|
||||
|
||||
```sh
|
||||
sh install.sh --mode source --source /absolute/source --openvr-root /absolute/sdk \
|
||||
--openvr-library /absolute/libopenvr_api.so --openvr-license /absolute/openvr/LICENSE \
|
||||
--sdl-library /absolute/libSDL3.so.0 --sdl-license /absolute/sdl/LICENSE \
|
||||
--version 0.1.202609241627
|
||||
```
|
||||
|
||||
It checks tools and native dependencies, builds/stages/packages in a private
|
||||
workspace, then installs the result. Unlike binary mode, this requires a C++
|
||||
compiler, CMake, SDK, SDL3, Wayland/scanner, libxcb, FreeType, Vulkan development
|
||||
files and producer-supplied
|
||||
licenses; it still does not install Python ASR packages. `--print-plan` performs
|
||||
a read-only plan, `--json` gives machine-readable results/errors (model installs
|
||||
also stream file events), and `--yes` authorizes network downloads. Bare TTY
|
||||
invocation can guide choices and prints equivalent flags; non-TTY runs require
|
||||
explicit arguments and never prompt. `--autolaunch`/`--no-autolaunch` are explicit
|
||||
OpenVR registration choices, off by default; do not pass either during an inert
|
||||
install if a running SteamVR session must remain untouched.
|
||||
|
||||
The installed launcher defaults to `--run` and forwards explicit run flags
|
||||
(including `--backend ID`, `--model-store /absolute/store` and
|
||||
`--manifest-dir /absolute/manifests`) to the binary. These overrides select
|
||||
local metadata/model paths, not a runtime download; the latter two require
|
||||
absolute paths without dot segments. `--backend` selects for that run unless
|
||||
changed in the panel; a saved `config.json` backend is otherwise used. For a
|
||||
native-only package, supply
|
||||
`FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`,
|
||||
or override `--python`/`--model` on an explicit `--run`. For menu launches, create
|
||||
`$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`):
|
||||
@@ -157,7 +220,7 @@ without configured model/runtime cannot transcribe; it should show **Unavailable
|
||||
(possibly after a brief Warming transition), rather than record or infer.
|
||||
Only perform this check on an unconfigured native-only installation: verify
|
||||
that `current/runtime/bin/python3` and `current/model` are absent and no
|
||||
user-local paths are configured; do not press Record, Insert or Enter.
|
||||
user-local paths are configured; do not press Record, Type or Type + Enter.
|
||||
The installer also provides a desktop entry; where Steam's UI supports adding a
|
||||
non-Steam app, the user may select that entry or browse to the installed launcher.
|
||||
Shortcut discovery/persistence after a normal restart is not yet verified. Do
|
||||
|
||||
+82
-74
@@ -1,91 +1,99 @@
|
||||
# Dependency provenance and release boundary
|
||||
|
||||
FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project
|
||||
owner. This does not relicense external models, fonts, protocols or runtimes.
|
||||
There is no dependency on another application's checkout, assets or environment.
|
||||
FrameYap's original code is [MIT licensed](../LICENSE). This does not relicense
|
||||
models, fonts, protocols, native libraries or Python wheels. There is no dependency
|
||||
on another application's checkout, assets or environment. **This is an upstream
|
||||
license inventory, not an audit of a particular release binary. No public release
|
||||
has been published.** Before shipping any archive, inspect the actual staged
|
||||
files, their transitive dependencies and notices, and test on a clean supported
|
||||
host. An external dependency does not make the current archive self-contained.
|
||||
|
||||
## Included source
|
||||
## Included source/assets
|
||||
|
||||
`protocol/gamescope-input-method.xml` is the unmodified public Gamescope
|
||||
**3.16.28** protocol, downloaded from
|
||||
<https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
|
||||
SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`.
|
||||
Its embedded permissive copyright/license notice is preserved. Generated bindings
|
||||
are build outputs, not hand-written wire encoding. The private protocol may change
|
||||
with SteamOS; compatibility must be rechecked.
|
||||
- `protocol/gamescope-input-method.xml`: unmodified public Gamescope **3.16.28**
|
||||
protocol, <https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
|
||||
SHA-256 `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`;
|
||||
embedded permissive copyright/license notice preserved. Generated bindings are
|
||||
build outputs; this private Gamescope extension needs rechecking after updates.
|
||||
- `assets/fonts/Inconsolata-Regular.ttf`: unchanged Inconsolata Regular from
|
||||
<https://github.com/googlefonts/Inconsolata>, copyright 2006 The Inconsolata
|
||||
Project Authors, **SIL OFL 1.1**. SHA-256
|
||||
`e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
|
||||
`assets/fonts/OFL-Inconsolata.txt` retains the license. The font remains OFL,
|
||||
not MIT, and is not sold by itself. An override font requires its own license.
|
||||
- Frame controller bindings and the mint-to-blue CPU-rasterized panel were
|
||||
authored here, using public profile names; no SteamVR driver artwork, MSDF
|
||||
atlas, other app renderer or protected kernels were copied.
|
||||
|
||||
Frame controller bindings were authored here using the observed public input
|
||||
profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code,
|
||||
images, protected kernels or another application's assets were extracted for these bindings.
|
||||
Native staging uses `scripts/stage-native.py`: it includes the chosen font/license
|
||||
and copied SDL3 and OpenVR notices in `licenses/THIRD_PARTY_NOTICES.txt`.
|
||||
It does **not** bundle ASR. A4 is not closed for release: inspect the final
|
||||
notice file, especially FreeType attribution, plus the selected native binary
|
||||
and any bundled wheel/licenses before publishing.
|
||||
|
||||
## Bundled font and UI reference
|
||||
## Native build/runtime inventory
|
||||
|
||||
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular.
|
||||
SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
|
||||
Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**,
|
||||
retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace
|
||||
normalized; license text unchanged). The reviewed OFL permits
|
||||
bundling and redistribution with its copyright/license notice; the font remains
|
||||
OFL, not MIT, and is not sold by itself. Upstream: <https://github.com/googlefonts/Inconsolata>.
|
||||
The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies
|
||||
were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph.
|
||||
| Component | Upstream license / evidence | Current packaging boundary / action |
|
||||
| --- | --- | --- |
|
||||
| Valve OpenVR SDK 2.15.6 | BSD-3-Clause-style license, SDK LICENSE. | Staging copies its loader and explicitly supplied license; verify chosen binary and transitive closure. |
|
||||
| SDL3 (device trial 3.2.16) | zlib, local `/usr/share/licenses/sdl3/LICENSE`. | Staging copies explicit library and license. Confirm exact build options/version and its transitive libraries. |
|
||||
| Wayland client + scanner | MIT/Expat-style, local `/usr/share/licenses/wayland/COPYING`. | Client is linked from system; scanner is build-time. If shipped, include copyright/license and audit closure. |
|
||||
| libxcb | MIT-style with name-use restriction, local `/usr/share/licenses/libxcb/COPYING`. | Xwayland focus guard uses client library at runtime; not bundled by current native stage. Audit exact binary. |
|
||||
| FreeType 2 | **FreeType Project License (FTL) selected** for this project, local `/usr/share/licenses/freetype2/FTL.TXT`; upstream also offers a GPL option. | Current stage uses system library, not bundled. Credit FreeType Team for use; if distributing its binary, meet FTL binary disclaimer/notice obligations and review the precise build. Do not silently substitute GPL terms. |
|
||||
| Vulkan loader, driver, system graphics dependencies | Loader/driver licenses vary by build and vendor. | Current stage depends on system Vulkan loader/driver; no GPU runtime is bundled. Audit the chosen loader if ever bundled. |
|
||||
| Compiler runtime (`libstdc++`, `libgcc_s` when used) | GCC libraries: GPL with **GCC Runtime Library Exception** in upstream distribution; local `/usr/share/licenses/libstdc++/RUNTIME.LIBRARY.EXCEPTION` and `libgcc/...` are exception texts, not a full installed release audit. | Current stage relies on system runtime. Audit dynamic linkage, C++ ABI/`GLIBCXX_*` and exception coverage for *any* bundled compiler libraries; include corresponding complete notices/source obligations as applicable. |
|
||||
| glibc/loader | GNU LGPL-2.1-or-later for core GNU C Library, with component-specific exceptions and other licenses to inspect. | Current stage relies on system libc/loader. Audit exact target binary symbol versions (`GLIBC_*`), ELF interpreter and its transitive closure; no minimum glibc/`GLIBCXX`/kernel floor is certified here. |
|
||||
|
||||
The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted
|
||||
selection) is implemented independently on a single CPU RGBA surface.
|
||||
CMake installs the font and OFL with assets; native staging defaults to that
|
||||
font, places the launcher copy at `fonts/font.ttf`, and includes its license in
|
||||
`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license.
|
||||
|
||||
## Explicit native build inputs (not vendored)
|
||||
|
||||
- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015;
|
||||
retain its LICENSE with redistributed loader binaries.
|
||||
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
|
||||
- Wayland client and scanner: retain upstream MIT-style notices.
|
||||
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
|
||||
Xwayland focus observer. Include it in native runtime dependency checks;
|
||||
no X server is started by normal operation.
|
||||
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
|
||||
- Optional font override: the earlier device check used system Hack Regular.
|
||||
A custom release font must include its own license and assessed glyph coverage;
|
||||
new builds default to the bundled Inconsolata described above.
|
||||
- Compiler runtime, libc minimum and transitive shared libraries require a release
|
||||
dependency audit. Passing a developer build is not a portable-runtime guarantee.
|
||||
The historical Frame CPU trial used Torch **2.8.0+cpu** on a host with **glibc
|
||||
2.39**. Those are *observed trial versions*, **not** minimum compatible versions
|
||||
for FrameYap, Python wheels or a future released artifact. `ldd` on a developer
|
||||
machine alone is not sufficient: inspect the staged ARM64 binaries with `readelf`
|
||||
(`NEEDED`, ELF interpreter, symbol-version requirements), `ldd` on a trusted
|
||||
clean target, actual bundled libraries and notices, and test the final archive on
|
||||
a clean supported Frame. Check the chosen compiler, CPU instruction/kernel,
|
||||
FreeType/Wayland/XCB/SDL/OpenVR/Vulkan ABI, Python/native wheels and licenses.
|
||||
Do not invent a libc floor from a build host's version.
|
||||
|
||||
## Redux weights and inference runtime
|
||||
|
||||
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution:
|
||||
Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3.
|
||||
No modifications to the supplied weights are made. Exact model/config/tokenizer
|
||||
sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`.
|
||||
`fetch-model.py` fetches and retains the original model card alongside the files.
|
||||
No weights are committed to this repository.
|
||||
Model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**.
|
||||
Attribution: Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT
|
||||
0.6B v3. No modifications to the supplied weights are made. Pinned model/config/
|
||||
tokenizer/card sizes and SHA-256 hashes, source and attribution are recorded in
|
||||
`assets/backends/redux.json` (schema validated by `python/frameyap/model_files.py`),
|
||||
not hardcoded in `model_files.py`. `fetch-model.py` fetches and retains the model
|
||||
card alongside the weights on **explicit** request; neither build/tests nor a
|
||||
normal app launch downloads them. `scripts/model-status.py` and the native
|
||||
`--list-models` / `--check-model` interface inspect/hashes local files offline.
|
||||
No weights are committed to Git.
|
||||
|
||||
Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels`
|
||||
dependencies, installed by the user from PyPI into their own environment. Upstream's
|
||||
Kestrel README states: "Local inference is free and requires no API key"
|
||||
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs a Moondream
|
||||
API key and is not used here. FrameYap's release archives do not vendor or bundle
|
||||
these packages; they are installed from PyPI onto the user's machine, either by the
|
||||
user or by the installer at the user's request.
|
||||
The current Redux worker uses separately provisioned `moondream` Python and its
|
||||
`kestrel` / `kestrel-kernels` dependencies, not a bundled runtime. Kestrel's
|
||||
upstream README says “Local inference is free and requires no API key”
|
||||
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs an API
|
||||
key and is **not** this path. The native-only installer **does not run pip**,
|
||||
provision an interpreter, or make a native-only artifact able to transcribe on
|
||||
its own. A person supplying a Python environment must review/authorize its
|
||||
exact dependency closure. No bundled-ASR artifact is licensed/approved by this
|
||||
inventory.
|
||||
|
||||
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need
|
||||
their own notice/license inventory before a bundled release.
|
||||
| Python/native package | Upstream license inventory (not a wheel audit) | Release action |
|
||||
| --- | --- | --- |
|
||||
| PyTorch / Torch CPU | PyTorch project: BSD-3-Clause; third-party components/wheels carry additional notices and dependencies. | No Torch wheels bundled. Pin CPU-only ARM64 wheel if building a distribution; audit its `LICENSE`, `NOTICE`, `third_party`/wheel contents and `NEEDED`/symbol versions. |
|
||||
| NumPy | NumPy core: BSD-3-Clause; dependencies/embedded algorithms carry additional BSD, MIT, 0BSD, zlib, CC0 and other notices depending on wheel. Local `python-numpy` 2.5.3 package metadata (`/usr/lib/python3.14/site-packages/numpy-2.5.3.dist-info/METADATA`) declares `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` with many `License-File` entries (different from the historical trial environment). | Not bundled. Keep *all* license files and inspect the exact target wheel, BLAS/OpenBLAS and runtime closure before redistribution. |
|
||||
| Hugging Face `tokenizers` | Upstream `huggingface/tokenizers` is Apache-2.0; native/Rust crate dependencies need separate inventory. | Not bundled. Confirm actual installed wheel version, package LICENSE/NOTICE and transitive Rust/native code if ever distributed. |
|
||||
| `moondream`, Kestrel/kernels/native, Python interpreter | Distinct packages with distinct license files and native transitive code; Kestrel local-use statement is not a blanket redistribution license. | None bundled. Exact versions, permissions, wheel notices, CPython build and native linkage require review before any runtime bundle. |
|
||||
|
||||
## CPU trial dependency choice
|
||||
Development CPU trial *interfaces*, not package-floor promises: moondream
|
||||
**2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, native **0.1.8**, Python
|
||||
**3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified moondream install
|
||||
initially resolved CUDA-enabled Torch and NVIDIA wheels; the owned trial venv
|
||||
was corrected before measurement (`torch.version.cuda is None`). Do **not**
|
||||
repeat unconstrained `pip install moondream` as a CPU setup recipe.
|
||||
|
||||
Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**,
|
||||
native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified
|
||||
moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels;
|
||||
these were replaced/removed from the owned venv before measurement. The measured
|
||||
Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
|
||||
`pip install moondream` as a CPU setup recipe.
|
||||
|
||||
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
|
||||
was downloaded but not bundled with any ASR runtime:
|
||||
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
||||
A standalone CPython 3.12.14 ARM64 distribution was downloaded for packaging
|
||||
research but **not bundled**: <https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
||||
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
|
||||
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
|
||||
Its included licenses and compatible native dependencies still require review
|
||||
before any release. No public prebuilt FrameYap release has been published.
|
||||
Its own licenses and native compatibility need review before any release.
|
||||
+38
-15
@@ -1,8 +1,10 @@
|
||||
# Offline Redux worker adapter (component, not an installed product)
|
||||
# Offline worker adapter and backend dispatcher
|
||||
|
||||
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
|
||||
threads=2, advanced_debug=false)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for
|
||||
one `WorkerReply` (text or privacy-safe per-request error). One request at a time;
|
||||
threads=2, advanced_debug=false)` explicitly for the legacy Redux script, or
|
||||
supply the optional `backend`, `manifest_dir`, `root` arguments for the manifest
|
||||
dispatcher. Poll until `ready()`, then `submit(id, pcm)` and poll for one
|
||||
`WorkerReply` (text or privacy-safe per-request error). One request at a time;
|
||||
no queue, no capture and no input injection. `stop()` discards pending audio,
|
||||
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
|
||||
and is safe to repeat. Destruction stops it. `start()` returns without waiting
|
||||
@@ -36,13 +38,19 @@ requests reading the fixed clip; `Y` means ready; `F` means load failure
|
||||
missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8
|
||||
text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096
|
||||
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
|
||||
or oversized frame stops the worker. Warmup deadline is 120 s, transcription
|
||||
deadline 60 s; `poll()` must be called regularly to enforce deadlines. It
|
||||
never initializes a headset or starts a recording. There is no auto restart.
|
||||
or oversized frame stops the worker. A correlated `E` reply or an invalid
|
||||
transcript is a **request-level** failure: the Controller drops that clip,
|
||||
keeps the ready model loaded and allows Record to retry. A broken protocol,
|
||||
worker crash or explicit in-flight cancellation is different and can require a
|
||||
reload. Warmup deadline is 120 s, transcription deadline 60 s; `poll()` must
|
||||
be called regularly to enforce deadlines. It never initializes a headset or
|
||||
starts a recording. There is no auto restart after a process failure.
|
||||
|
||||
`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI
|
||||
startup, with HF/Transformers/Datasets offline variables and bounded native
|
||||
thread-pool variables set before import. It uses
|
||||
Parakeet Redux is the **model** (`moondream/parakeet-redux`); `moondream` is
|
||||
its Python inference package, not a second model or cloud endpoint. Kestrel is
|
||||
a native dependency of that local runtime. `python/frameyap/worker.py` lazily
|
||||
imports `moondream` only after explicit CLI startup, with HF/Transformers/Datasets
|
||||
offline variables and bounded native thread-pool variables set before import. It uses
|
||||
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
|
||||
device="cpu", cpu_threads=threads)` and persistent
|
||||
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
|
||||
@@ -51,16 +59,31 @@ moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
|
||||
preinstalled local weights from revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
|
||||
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
|
||||
against `model_files.py` before importing model libraries. Protocol stdout is
|
||||
isolated at the file-descriptor level from third-party diagnostics. Thread limits
|
||||
cover Torch interop/native pools and CUDA is not selected. Offline environment
|
||||
against `assets/backends/redux.json` via the shared, offline
|
||||
`python/frameyap/model_files.py` schema/verifier before importing model libraries.
|
||||
`frameyap --list-models` and `frameyap --check-model redux --model-dir /absolute/model`
|
||||
(or `scripts/model-status.py`) read local manifests and optionally hash local
|
||||
files; they never load the worker or download weights. For native `--run`,
|
||||
`--backend ID`, `--model-store /absolute/store`, and
|
||||
`--manifest-dir /absolute/manifests` are wired through the installed launcher
|
||||
as explicit overrides; default selection is saved in `config.json` (`redux`). The dispatcher
|
||||
`python/frameyap/backend_worker.py` validates a selected manifest and hashes
|
||||
its local model, resolves an in-release relative Python/executable launcher
|
||||
without a shell, then supervises one child over `frameyap-worker-v1` (ready,
|
||||
correlated request/reply and failure frames). The launcher must accept
|
||||
`{model_dir}` and `{clip_dir}` (optional `{threads}`); matching manifest and
|
||||
pinned weights are **not** a runtime or automatically trusted new backend.
|
||||
Only Redux is supplied today. Protocol stdout is isolated at the file-descriptor
|
||||
level from third-party diagnostics. Thread limits cover Torch interop/native
|
||||
pools and CUDA is not selected. Offline environment
|
||||
flags do not prove every third-party internal is unable to access a network.
|
||||
Runtime/build/tests perform no downloads; the separate explicit setup utility
|
||||
`scripts/fetch-model.py` can provision the public pinned weights.
|
||||
|
||||
Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public
|
||||
runtime bundle has been released. Limited ARM64 measurements were taken during development;
|
||||
they are not a claim of complete headset acceptance.
|
||||
Current native-only archives do not bundle or pip-install the runtime; see
|
||||
[third-party notes](third-party.md). No public runtime bundle has been released.
|
||||
Limited ARM64 measurements were taken during development; they are not a claim
|
||||
of complete headset acceptance.
|
||||
|
||||
## Advanced debugging
|
||||
|
||||
|
||||
Reference in new issue
Block a user