mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-06 02:00:06 +02:00
docs: align release scope and track remaining device acceptance gates
This commit is contained in:
1 parent
4498ef5d04
commit
1321ca8316
9 files changed
+622
-527
No files matched your search
@@ -1,138 +1,47 @@
|
|||||||
# FrameYap
|
# FrameYap
|
||||||
|
|
||||||
Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress).
|
Voice typing on Steam Frame, with recognition on the headset rather than a desktop or cloud server.
|
||||||
|
Hold a controller button to record, review the transcript, then deliberately type it into the focused app.
|
||||||
|
Standalone MIT-licensed OpenVR overlay; no Steam store AppID or sudo. First v0.1 release is in progress.
|
||||||
|
|
||||||
Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture,
|
## Requirements
|
||||||
persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope,
|
|
||||||
and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR
|
|
||||||
server, cloud fallback or unrelated application dependency.
|
|
||||||
|
|
||||||
**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility,
|
- Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format. Binary compatibility must be checked against each actual release artifact, not inferred from a developer build.
|
||||||
Gamescope discovery and native-only installation have been exercised on Frame.
|
- For voice recognition, separately provision a compatible **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0 and dependencies) and the pinned local Parakeet Redux model. Neither is bundled or installed with pip by the current native-only installer. There is no fallback ASR service.
|
||||||
Live microphone → reviewed text → real target delivery is **not yet accepted**.
|
- A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](docs/build.md).
|
||||||
|
|
||||||
**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the
|
## Install (local artifacts only)
|
||||||
`moondream` Python package (its Kestrel runtime states that local inference is free
|
|
||||||
and needs no API key). The build and tests never download it; the installer or you
|
|
||||||
install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md).
|
|
||||||
No GitHub release is published yet.
|
|
||||||
|
|
||||||
## Controls
|
**No public release is published.** To install a locally vetted, checksummed ARM64 native-only archive without a compiler:
|
||||||
|
|
||||||
- **Right X (default Frame binding):** hold to record; release to
|
|
||||||
transcribe. Repeated presses reached the controls-only diagnostic on Frame;
|
|
||||||
live mic capture through this shortcut still needs guided acceptance. This
|
|
||||||
PTT action is remappable through SteamVR bindings.
|
|
||||||
- **Right B:** cancel/discard (or close quick chat). **Right A:** insert reviewed text with a trailing space.
|
|
||||||
**Right Y:** open quick chat; press again to cycle its highlighted choice.
|
|
||||||
**Submit** on the overlay (or double-tap left grip when active) sends that choice
|
|
||||||
verbatim then Enter; outside quick chat it inserts pending review + Enter, or
|
|
||||||
sends Enter alone when there is no text. Edit the `quick_inputs` list in
|
|
||||||
`$XDG_CONFIG_HOME/frameyap/config.json` (restart to apply). Nothing submits
|
|
||||||
automatically. Auto Insert is opt-in and off by default.
|
|
||||||
- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Submit and Hold Quit
|
|
||||||
(hold the button for 0.9 seconds before releasing).
|
|
||||||
Review and settings share one Inconsolata/neon-framed surface. The header
|
|
||||||
shows local time/date; Settings selects 12/24-hour time and date format/off.
|
|
||||||
Hold the thin bar below the panel to freely move and rotate it with your
|
|
||||||
controller; release to leave it at that pose. Drag the external lower-right
|
|
||||||
bracket to scale it (world, head or wrist), keeping the upper-left anchored.
|
|
||||||
Both handles use the app's gradient in transparent margins, like Steam's
|
|
||||||
window handles. Settings → **Lock grab/scale** hides and disables both handles;
|
|
||||||
the lock is saved. Position/size changes last for this run only.
|
|
||||||
Wrist mounting fades the panel as its full orientation turns away from an
|
|
||||||
upright viewer-facing pose (60°–75°), hiding interaction past that angle;
|
|
||||||
world and head mounting do not fade.
|
|
||||||
**Bindings** requests SteamVR's binding editor directly. Existing SteamVR overrides may supersede defaults. World-space by
|
|
||||||
default; settings offer left wrist, right wrist and head mounting, plus recenter.
|
|
||||||
Dashboard lasers provide clickable controls. Settings → Lasers anytime is an
|
|
||||||
opt-in, default-off system-wide laser mode while the panel is visible; it may
|
|
||||||
affect games and is separate from experimental input overrides.
|
|
||||||
- **Theme and controls:** optional `$XDG_CONFIG_HOME/frameyap/config.json` selects
|
|
||||||
panel colors, a font path and Frame controller button mappings; missing fonts
|
|
||||||
fall back to bundled Inconsolata. The installer creates/checks this file and
|
|
||||||
backs it up before repairs. See [overlay configuration](docs/overlay.md#user-theme-and-controller-configuration).
|
|
||||||
- **Advanced debugging:** Settings toggle / `"advanced_debug": true` in config.
|
|
||||||
Off by default. Restarts the worker and discards current work; full exceptions,
|
|
||||||
worker output and transcripts go to private, bounded local logs. No raw audio
|
|
||||||
archive. See [diagnostics](docs/worker.md#advanced-debugging).
|
|
||||||
- **Experimental input priority:** set `"input_priority": "experimental"` in
|
|
||||||
that config and enable SteamVR's Developer option **Enable global input from
|
|
||||||
overlays**. FrameYap then requests priority for its bound controller sources.
|
|
||||||
This may consume controls used by games or the dashboard; coexistence on Frame
|
|
||||||
is under test. The default is `"normal"`; restart FrameYap after changing it.
|
|
||||||
- **Review by default:** focus your destination, then press Insert (text + space) or
|
|
||||||
explicitly Enter (text + space, then Enter). Settings → Auto insert is off by default:
|
|
||||||
when enabled, it queues text + space only if Xwayland keyboard focus, active
|
|
||||||
window and Gamescope focus match continuously from recording through delivery.
|
|
||||||
Any uncertainty leaves a preview for manual Insert; it never sends Enter.
|
|
||||||
This is not yet validated for live transcription on Frame. Maximum clip 20 seconds;
|
|
||||||
accidental taps under 200 ms are discarded.
|
|
||||||
- While the native app is Ready, it keeps the mic device open and discards idle
|
|
||||||
audio instead of opening/closing on every PTT. Quit releases the device. Other
|
|
||||||
apps may still hear/transmit your voice; FrameYap does not mute them.
|
|
||||||
|
|
||||||
Physical gesture timing, global bindings during games, mic capture, target-app
|
|
||||||
compatibility and headset comfort still need coordinated validation. Successful
|
|
||||||
API initialization is not delivered input or human acceptance.
|
|
||||||
|
|
||||||
## Build and test offline
|
|
||||||
|
|
||||||
CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests.
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cmake -S . -B build
|
sh install.sh --mode binary --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
||||||
cmake --build build
|
|
||||||
ctest --test-dir build --output-on-failure
|
|
||||||
./build/frameyap --help
|
|
||||||
```
|
|
||||||
|
|
||||||
Default build has no hardware backends. It never downloads packages/models or
|
|
||||||
initializes SteamVR, a microphone or input injection. Native dependencies and
|
|
||||||
explicit launch/check commands are documented in [the build guide](docs/build.md).
|
|
||||||
`--version` uses an ISO-like UTC build timestamp (with Git hash when available),
|
|
||||||
not a numbered release; use that same tag when packaging the binary.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
The real installer accepts a versioned, checksummed prebuilt ARM64 archive:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
|
||||||
--sha256 ARCHIVE_SHA256 --version VERSION
|
--sha256 ARCHIVE_SHA256 --version VERSION
|
||||||
```
|
```
|
||||||
|
|
||||||
This is the **local-artifact command shape**, not an available public download.
|
`VERSION` is numeric `0.1.YYYYMMDDHHMM` for this release line; a future release tag will be `vVERSION`. The installer has an explicit local source-build path with toolchain/dependency inputs, and a read-only `--print-plan`/machine-readable `--json` mode; the producer stage→archive path still needs a clean-account artifact test and release audit. See [packaging](docs/packaging.md). Installation is user-local with rollback, no automatic launch or registration, and no Steam store AppID. Before voice typing, independently supply a licensed Python CPU environment and pinned weights, and configure their absolute paths in `~/.config/frameyap/paths.conf` (or the XDG config equivalent). Model download is opt-in and separate: `sh install.sh --install-model --backend redux --print-plan --json` inspects pinned metadata, while `--yes` explicitly authorizes installation from the *installed* manifest; `--expected-manifest-sha256 HASH` additionally binds consent to its exact bytes. The locally implemented Settings → Models chooser offers selection/restart and a two-click Install/Confirm Install with source, size, license, attribution and manifest SHA-256; it has not been validated as an installed/headset flow. No chooser action installs the Python runtime. Do not mistake a native-only install for working ASR. The pinned GitHub download route must not be advertised as functional until a vetted release exists.
|
||||||
Installation is user-local, retains rollback, refuses active-app upgrades and
|
|
||||||
foreign files, and does not launch or register automatically. Registration uses
|
|
||||||
OpenVR application key `local.frameyap.overlay`, not a Steam store AppID.
|
|
||||||
Autolaunch is opt-in. An installed desktop entry can be selected manually as a
|
|
||||||
non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the
|
|
||||||
panel on one Frame; registration alone did not show an entry in the first checked
|
|
||||||
dashboard menu. See [packaging and lifecycle](docs/packaging.md).
|
|
||||||
For a native-only install, menu-driven inference needs a Python runtime you
|
|
||||||
provide and the pinned model. Configure their absolute paths in
|
|
||||||
`~/.config/frameyap/paths.conf` as described in the packaging guide; they are
|
|
||||||
never fetched or bundled implicitly.
|
|
||||||
|
|
||||||
A pinned GitHub one-command route is implemented in `install.sh --version TAG`,
|
## Controls (default Frame binding)
|
||||||
but **do not advertise or run it as a working public installation until a vetted
|
|
||||||
release exists**. Native-only artifacts support the overlay/checks without an
|
|
||||||
end-user compiler; a bundled-ASR distribution is not yet offered.
|
|
||||||
|
|
||||||
## Project map
|
| Button / control | Action |
|
||||||
|
| --- | --- |
|
||||||
|
| Right X, hold / release | Record while held; release to transcribe. |
|
||||||
|
| Right B | Cancel/discard, or close the Quick phrases picker. |
|
||||||
|
| Right A | **Type:** queue reviewed text, normally with a trailing space. |
|
||||||
|
| Right Y | Open **Quick phrases**; press again to cycle the selection. |
|
||||||
|
| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. |
|
||||||
|
| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. |
|
||||||
|
| Overlay Type / Type + Enter | The same deliberate text / Enter actions. |
|
||||||
|
| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). |
|
||||||
|
|
||||||
- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests.
|
Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Check the focused destination before Type or Type + Enter. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](docs/overlay.md).
|
||||||
- [Design](docs/design.md): full target design; some features remain proposed.
|
|
||||||
- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md).
|
|
||||||
- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md),
|
|
||||||
[dependency/license inventory](docs/third-party.md).
|
|
||||||
- [TODO](TODO.md): release plan and open work items.
|
|
||||||
|
|
||||||
Dated device-test records are kept locally (untracked) and are not authority for
|
## Status / not yet validated
|
||||||
current device availability. Live microphone → reviewed text → target delivery has
|
|
||||||
not been formally accepted; see Status above.
|
|
||||||
|
|
||||||
No recordings, transcripts, private logs, model weights or runtime binaries are
|
Native ARM64 build, CPU inference on a public clip, overlay visibility, Gamescope API discovery, a controls-only Right X probe and native-only installation have been exercised on Frame. **Live microphone → reviewed transcript → real target delivery has not been formally accepted.** Focus guard and insertion have offline/owned-target checks, not general app compatibility or human headset acceptance. Global bindings, physical gesture feel, Auto insert with speech, latency, battery/thermal cost and controller coexistence still require opt-in headset testing. Front-prefix loss on repeated submissions (P1) remains under separate investigation; do not treat it as fixed. An old-code fixture crashed the Gamescope session, not the OS, and does not prove the new delivery path. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it.
|
||||||
committed. The worker boundary is intentionally small for forks experimenting
|
|
||||||
with other models/APIs; the default remains local-only Redux.
|
The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](docs/overlay.md), [worker](docs/worker.md), [design](docs/design.md).
|
||||||
|
|
||||||
|
Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.<timestamp>` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](docs/third-party.md); [TODO](TODO.md).
|
||||||
|
|
||||||
|
No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection.
|
||||||
@@ -6,95 +6,110 @@ M ≈ a day, L ≈ multi-day.
|
|||||||
|
|
||||||
Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
|
Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
|
||||||
downloads, no automatic Enter, small verified commits, docs must state implemented
|
downloads, no automatic Enter, small verified commits, docs must state implemented
|
||||||
vs proposed behavior honestly.
|
vs proposed behavior honestly. Checked source-work items below mean the local
|
||||||
|
implementation is present, **not** a shipped or headset-accepted release.
|
||||||
|
|
||||||
|
## Implemented locally; offline tested; installed/headset validation pending
|
||||||
|
|
||||||
|
The current working tree includes the source changes for A3, A5–A7, B1–B4,
|
||||||
|
C1–C3, D1–D2, E2–E3 and F1–F2. Offline checks passed 23/23 default,
|
||||||
|
24/24 strict UI, and 28/28 on a private native ARM64 snapshot. Later review
|
||||||
|
fixes require a fresh final native build before deployment.
|
||||||
|
These checkboxes close the *source tasks*, not their empirical acceptance gates.
|
||||||
|
C1's second backend is a fake executable fixture, **not** a second shipped ASR
|
||||||
|
engine. C2's two-click model consent, SHA-bound installer handoff, and D1/D2's
|
||||||
|
source/attended installer paths still need an audited native archive and an
|
||||||
|
installed clean-account/Frame exercise. A4 (exact artifact ABI/license closure),
|
||||||
|
D3 (publication and clean-account acceptance), P1 (real-target delivery), G1
|
||||||
|
(browser), and H (live headset acceptance) remain open. No new build is claimed
|
||||||
|
deployed to Frame; local code/tests cannot establish a fixed delivery regression.
|
||||||
|
|
||||||
|
## Priority regression (reported during implementation)
|
||||||
|
|
||||||
|
- [ ] **P1. Repeated delivery loses the beginning of later submissions.** After
|
||||||
|
the first Type + Enter, later text reportedly loses a dozen to a few dozen leading
|
||||||
|
bytes. Investigate preview versus destination loss, retain the full bounded
|
||||||
|
literal transcript, and add repeated/long/Unicode delivery regression tests.
|
||||||
|
Do not assume a larger buffer fixes it or retry uncertain delivery automatically.
|
||||||
|
An old-code delivery fixture crashed the Gamescope session, **not** the OS;
|
||||||
|
this is not evidence of a fix. Real-target confirmation remains required after
|
||||||
|
a tested fix is deployed.
|
||||||
|
|
||||||
## A. First release (v0.1) blockers
|
## A. First release (v0.1) blockers
|
||||||
|
|
||||||
- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
|
- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
|
||||||
states local inference is free; blocker text removed from README/docs/scripts and
|
states local inference is free; blocker text removed from README/docs/scripts and
|
||||||
replaced with a neutral dependency note in `docs/third-party.md`.
|
replaced with a neutral dependency note in `docs/third-party.md`.
|
||||||
- [x] **A2. Remove references to the unrelated app (kouseki).** (S)
|
- [x] **A2. Remove references to the unrelated app.** (S)
|
||||||
Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a
|
Tracked references were removed while preserving Inconsolata/OFL attribution,
|
||||||
final grep. Keep the Inconsolata/OFL attribution and cite the
|
font SHA-256 and upstream googlefonts/Inconsolata source. A tracked,
|
||||||
upstream font source (googlefonts/Inconsolata) instead.
|
case-insensitive grep for the former app name must remain empty.
|
||||||
*Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains.
|
- [x] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
|
||||||
- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
|
"FrameYap"). (S) Done in baseline checkpoint `07c03ea`.
|
||||||
"FrameYap"). (S)
|
|
||||||
- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M)
|
- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M)
|
||||||
Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL
|
Upstream inventory and the FreeType FTL choice are documented, but the exact
|
||||||
option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt
|
staged ARM64 native/runtime binaries, transitive wheel/library notices, symbol
|
||||||
archive that includes any of them.
|
versions, loader and libc floor still require artifact-specific review before
|
||||||
- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README,
|
publishing any prebuilt archive.
|
||||||
`scripts/stage-native-poc.py`, `CMakeLists.txt` messages,
|
- [x] **A5. Drop "POC" from the shipped surface.** (S) Public help, README,
|
||||||
installer strings. v0.1 is a first small release.
|
CMake and installer use the release name. `scripts/stage-native-poc.py` remains
|
||||||
- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install,
|
a deprecated compatibility wrapper for `scripts/stage-native.py`, not the
|
||||||
controls **table** (button → action), then a "Status / not yet validated" section.
|
documented or shipped staging entry point. v0.1 is a first small release.
|
||||||
Today it reads as a lab notebook and Controls is a wall of text.
|
- [x] **A6. Rewrite the README front.** (M) Pitch, requirements, local-only install,
|
||||||
|
controls table and explicit "Status / not yet validated" section are present.
|
||||||
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
|
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
|
||||||
to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
|
to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
|
||||||
Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and
|
Current design/overlay/packaging docs distinguish implemented local behavior
|
||||||
hedging language before v0.1.
|
from proposed and unaccepted headset/release behavior.
|
||||||
|
|
||||||
## B. Correctness / robustness
|
## B. Correctness / robustness
|
||||||
|
|
||||||
- [ ] **B1. A malformed transcript must not kill the worker.** (S)
|
- [x] **B1. Keep malformed transcripts request-local.** (S)
|
||||||
`src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control
|
`Controller::tick()` catches a correlated bad UTF-8/control reply and fails the
|
||||||
characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and
|
request without stopping the ready worker; hardware-free fakes test retry.
|
||||||
`session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a
|
- [x] **B2. Use engine-neutral C++ worker errors.** (S)
|
||||||
request-level error (like the `E` path: keep the worker, show "transcription
|
`F`/`I` messages no longer name Redux's Python engine.
|
||||||
failed", allow retry). Add a test with a control-character reply asserting the
|
- [x] **B3. "Close mic when idle" setting, default OFF.** (M)
|
||||||
worker stays ready.
|
Config/Settings and fake-backed mic-lifetime tests cover default idle draining
|
||||||
- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes
|
versus opt-in close/reopen; the physical spike/latency tradeoff is documented.
|
||||||
"moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a
|
- [x] **B4. Extract/test interaction logic from `run()`.** (L)
|
||||||
worker-supplied message code. Prerequisite for C1.
|
`Controller` receives injectable audio/worker/focus/delivery interfaces; offline
|
||||||
- [ ] **B3. "Close mic when idle" setting, default OFF.** (M)
|
tests cover PTT, cancel, phrases, Auto insert and error/retry transitions.
|
||||||
Default keeps the mic open while Ready (opening/closing per PTT causes an audio
|
|
||||||
spike on the physical hardware, and it avoids first-syllable clipping). The
|
|
||||||
setting closes it between clips for people who don't want a live device. Document
|
|
||||||
the tradeoff (spike/latency) next to the toggle, in Settings and in the docs.
|
|
||||||
Persist in `config.json`; add to the panel Settings tab.
|
|
||||||
- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L)
|
|
||||||
`run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no
|
|
||||||
tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`)
|
|
||||||
so PTT, cancel, quick phrases, auto-insert and error transitions are testable
|
|
||||||
without hardware. B1 is the first regression test.
|
|
||||||
|
|
||||||
## C. Backends and model management
|
## C. Backends and model management
|
||||||
|
|
||||||
- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L)
|
- [x] **C1. Manifest-driven worker backends (source capability).** (L)
|
||||||
Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper,
|
Redux's pinned manifest and a generic local dispatcher implement the existing
|
||||||
sherpa-onnx Parakeet, …) as separate worker executables speaking the existing
|
`Y`/`T`/`R`/`E` framing. An offline fake second executable backend works without
|
||||||
`Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name,
|
edits to C++ worker/runtime; only Redux has a shipped inference engine. New
|
||||||
launcher, pinned model files + hashes + attribution, license text, CPU/GPU
|
backends still require license/runtime and actual inference validation.
|
||||||
requirements) so nothing is hardcoded in C++ or `model_files.py`.
|
- [x] **C2. Model/backend state and chooser (local source).** (L)
|
||||||
*Done when:* Redux is expressed as a manifest, and a second backend can be added
|
Settings lists manifest-backed model status and active/loading/ready/failure
|
||||||
without touching `worker.cpp`/`runtime.cpp`.
|
states; selection persists and invalidates/restarts worker, clip and review.
|
||||||
- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4.
|
Install → Confirm Install displays pinned source, size, license, attribution and
|
||||||
Settings page listing backends/models with state (not installed / installed and
|
manifest SHA-256; confirmation launches an owned installer helper with that
|
||||||
verified / loading / ready / failed), the active one marked, and selection that
|
digest and offline rechecks afterward. Full consent metadata is paginated;
|
||||||
restarts the worker. Target user is non-technical: an **Install** button on the
|
the panel shows bounded per-file download/verification events and sanitized
|
||||||
panel runs the installer's machine-readable mode (D2) as a child process and shows
|
errors. Installer output alone never proves success. No implicit downloads or
|
||||||
progress/errors in the panel, so nobody needs a terminal. The click is the consent;
|
runtime install. **Installed chooser/consent behavior on Frame remains unvalidated.**
|
||||||
there are still no implicit or background downloads, and the panel states what
|
- [x] **C3. Offline model status CLI.** (S) `frameyap --list-models` and
|
||||||
will be downloaded and how large it is before it starts. Persist the choice in
|
`--check-model ID` dispatch pinned local manifest/hash checks, without ASR or
|
||||||
`config.json`.
|
downloads. Installed archive CLI still needs clean-account verification.
|
||||||
- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID`
|
|
||||||
(offline, hash-verifies installed files) so the UI and installer share one
|
|
||||||
implementation.
|
|
||||||
|
|
||||||
## D. Installer
|
## D. Installer
|
||||||
|
|
||||||
- [ ] **D1. Installer with a binary-or-source choice.** (L)
|
- [x] **D1. Local binary-or-source installer paths.** (L)
|
||||||
`install.sh` offers *prebuilt archive* (checksummed) or *build from source*
|
`install.sh` has a checksummed binary route and explicit local source/toolchain
|
||||||
(checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then
|
preflight/build/stage/install route, with rollback and no default registration.
|
||||||
continues automatically through install after the user's choices. Retain rollback,
|
A native archive was exercised in private HOME/XDG roots on Frame: install,
|
||||||
idempotency, no sudo, no Steam AppID, opt-in autolaunch.
|
installed model-status CLI, idempotency, wrapper repair and uninstall. This
|
||||||
- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M)
|
same-user/system-library fixture is not clean-account acceptance.
|
||||||
Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`,
|
- [x] **D2. Attended-or-unattended model-agnostic installer interface.** (M)
|
||||||
`--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`,
|
TTY choices have equivalent flags; `--mode binary|source`, `--backend ID`,
|
||||||
`--json` output) so a model/agent can run it non-interactively; interactive
|
`--model-dir`, `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`,
|
||||||
prompts only run on a TTY and print the equivalent flags they chose. Exit codes
|
`--print-plan` and `--json` support offline plans and structured outcomes;
|
||||||
and messages must be machine-readable.
|
explicit model installs use installed pinned manifests. Tested with local
|
||||||
|
fixtures only, not a released archive or an installed Frame UI handoff.
|
||||||
- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the
|
- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the
|
||||||
release checklist in `docs/packaging.md`; do not advertise the one-command route
|
release checklist in `docs/packaging.md`; do not advertise the one-command route
|
||||||
until the archive and its checksum are actually published and tested from a clean
|
until the archive and its checksum are actually published and tested from a clean
|
||||||
@@ -106,32 +121,24 @@ vs proposed behavior honestly.
|
|||||||
says what it is; no rename churn before the first tag. If a hardware-neutral
|
says what it is; no rename churn before the first tag. If a hardware-neutral
|
||||||
project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
|
project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
|
||||||
front end), decide it before the cross-window work in G, not now.
|
front end), decide it before the cross-window work in G, not now.
|
||||||
- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text +
|
- [x] **E2. User-facing Type / Type + Enter / Quick phrases labels.** (S)
|
||||||
space) and **Type + Enter**; align overlay buttons, `--check-controls` output,
|
Overlay, diagnostics, README/help and overlay docs align on these controls;
|
||||||
README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only
|
`insert`, `enter`, `quick_chat` remain internal binding/API names. Settings
|
||||||
if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings
|
explains Hold Quit and Lasers anytime; worker docs distinguish the Redux model
|
||||||
explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux"
|
from its `moondream` Python inference package.
|
||||||
vs `moondream` naming once in `docs/worker.md`.
|
- [x] **E3. Numeric `MAJOR.MINOR.YYYYMMDDHHMM` version source work.** (S)
|
||||||
- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S)
|
CMake, CLI/tests, package producer and installer use numeric release versions;
|
||||||
e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts
|
dev git info is a separate `--version` line. `SOURCE_DATE_EPOCH` can supply
|
||||||
correctly, keeps the build date visible, URL/filename-safe. The version field is
|
the configuration timestamp. A clean release tag/archive still needs D3.
|
||||||
just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted
|
|
||||||
changes) suffixes were only for telling developer builds apart, so they move out
|
|
||||||
of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev
|
|
||||||
build adds a second line like `git abc12345 (uncommitted changes)`. Release
|
|
||||||
archives are built from a clean tag, so users never see it. Update the CMake
|
|
||||||
version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`,
|
|
||||||
installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the
|
|
||||||
timestamp. Tag releases `v0.1.<timestamp>`.
|
|
||||||
|
|
||||||
## F. Code structure (non-urgent)
|
## F. Code structure (non-urgent)
|
||||||
|
|
||||||
- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of
|
- [x] **F1. Move `--check-*` diagnostics out of `main.cpp`.** (S)
|
||||||
inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a
|
`src/check.cpp` owns native checks; `src/cli.cpp` provides the table-driven
|
||||||
table-driven option parser.
|
parser. Device behavior remains separately gated.
|
||||||
- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag
|
- [x] **F2. Split `overlay.cpp`'s `Impl`.** (M)
|
||||||
state, save-failure flags, counters, pose caches): separate drag, persistence and
|
Drag state, persistence/save failures and diagnostics now have separate grouped
|
||||||
diagnostics.
|
owners; native snapshot build and offline panel/drag tests cover the refactor.
|
||||||
|
|
||||||
## G. Other windows (post-release)
|
## G. Other windows (post-release)
|
||||||
|
|
||||||
@@ -158,7 +165,7 @@ this app is future design and out of scope for v0.1.
|
|||||||
across a normal restart (without restarting sessions just for the test). Confirm no
|
across a normal restart (without restarting sessions just for the test). Confirm no
|
||||||
runtime/model environment override before future live checks.
|
runtime/model environment override before future live checks.
|
||||||
- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery,
|
- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery,
|
||||||
Auto Insert with speech, physical resize.
|
Auto insert with speech, physical resize and a verified deployed version.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -179,4 +186,5 @@ this app is future design and out of scope for v0.1.
|
|||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
None currently blocking.
|
Release artifact compatibility/license audit, publication, P1 real-target behavior
|
||||||
|
and live headset validation are unresolved gates, not implied by checked source tasks.
|
||||||
+55
-25
@@ -11,8 +11,9 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
|||||||
mode while FrameYap is visible; it may affect games, and is not an input override.
|
mode while FrameYap is visible; it may affect games, and is not an input override.
|
||||||
- Remappable SteamVR actions. The default Steam Frame binding maps right X
|
- Remappable SteamVR actions. The default Steam Frame binding maps right X
|
||||||
(hold to record, release to transcribe) to the existing PTT action using the
|
(hold to record, release to transcribe) to the existing PTT action using the
|
||||||
observed `frame_controller` profile. Right B cancels, A inserts + space, and Y
|
observed `frame_controller` profile. Right B cancels, A requests Type (text +
|
||||||
inserts pending text + Enter (or Enter only with no preview). The Bindings button
|
space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay
|
||||||
|
control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button
|
||||||
requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
|
requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
|
||||||
actions were inactive in the observed dashboard check; do not rely on them.
|
actions were inactive in the observed dashboard check; do not rely on them.
|
||||||
If left grip becomes active, two short taps request explicit Enter.
|
If left grip becomes active, two short taps request explicit Enter.
|
||||||
@@ -22,27 +23,41 @@ initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
|||||||
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
|
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
|
||||||
20 second maximum. In the native app the microphone stream opens after model
|
20 second maximum. In the native app the microphone stream opens after model
|
||||||
warm-up, stays running between utterances, and discards idle samples; PTT does
|
warm-up, stays running between utterances, and discards idle samples; PTT does
|
||||||
not open/pause/close the device. Quit, worker restart, device failure or capture
|
not open/pause/close the device **by default**. Settings → Close mic when idle
|
||||||
failure closes it. This avoids repeated capture-device transitions but does not
|
(`"close_mic_when_idle": true`, default **false**) closes it between clips and
|
||||||
promise glitch-free playback on every audio stack. Other apps may still
|
reopens on PTT; repeated transitions caused an audio spike on Frame and may
|
||||||
|
add latency or clip the first syllable. Quit, worker restart, device failure
|
||||||
|
or capture failure also closes it. Keeping it open does not promise glitch-free
|
||||||
|
playback on every audio stack. Other apps may still
|
||||||
transmit your voice: this app does **not** mute VRChat or any other app.
|
transmit your voice: this app does **not** mute VRChat or any other app.
|
||||||
|
- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed
|
||||||
|
backend restarts the worker and invalidates pending clip/review/focus authority;
|
||||||
|
a missing or invalid model disables recording. A separate Install then Confirm
|
||||||
|
Install exposes pinned source/size/license/attribution and exact manifest-byte
|
||||||
|
SHA-256 before installer handoff. These are local implementations, not a tested
|
||||||
|
installed UI or an approved additional inference engine. Only Redux is supplied.
|
||||||
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
|
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
|
||||||
cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
|
cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
|
||||||
Model imports are lazy and loading is offline. Normal repeats, request-local
|
Model imports are lazy and loading is offline. A correlated bad transcript/
|
||||||
transcription failures and microphone failures retain the loaded model;
|
`E` reply is a request-level failure: Record can retry without dropping the
|
||||||
|
ready model. Normal repeats and microphone failures retain the loaded model;
|
||||||
microphone device failure releases the stream for explicit retry. A cancelled
|
microphone device failure releases the stream for explicit retry. A cancelled
|
||||||
in-flight request or broken worker protocol may require reloading. No
|
in-flight request or broken worker protocol may require reloading. No
|
||||||
cloud/desktop fallback.
|
cloud/desktop fallback.
|
||||||
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
|
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
|
||||||
handling, UTF-8/control validation and explicit Submit action for Enter.
|
handling, UTF-8/control validation and explicit Type + Enter action.
|
||||||
Insert ensures a trailing space without doubling an existing one. Enter
|
Type ensures a trailing space without doubling an existing one. Type + Enter
|
||||||
first inserts pending review, releases the text lease, then acquires a fresh
|
first types pending review, releases the text lease, then acquires a fresh
|
||||||
lease for Submit. Failed/uncertain text never proceeds to Submit; failed
|
lease for Enter. Failed/uncertain text never proceeds to Enter; failed
|
||||||
Submit acquisition never replays text. A full 4096-byte transcript without
|
Enter acquisition never replays text. If a validated transcript fills the
|
||||||
room for a space is preserved with an error, never silently truncated.
|
4096-byte bound and lacks a trailing space, Type preserves all its bytes and
|
||||||
- Idempotent user-local release-archive installer: SHA-256, safe extraction,
|
queues it **without** the usual space; it does not signal a separate error.
|
||||||
atomic current-version selection, retained rollback, runtime/install lock,
|
Destination consumption and repeated-delivery behavior remain unaccepted.
|
||||||
foreign-file refusal and explicit unregister-before-uninstall acknowledgement.
|
- User-local installer with checked binary-archive or explicitly provisioned
|
||||||
|
source-build mode, safe extraction, atomic current-version selection, retained
|
||||||
|
rollback, runtime/install lock, foreign-file refusal and explicit
|
||||||
|
unregister-before-uninstall acknowledgement. Source mode needs a local
|
||||||
|
compiler, SDK, libraries and license inputs; no runtime is pip-installed.
|
||||||
|
|
||||||
## Deliberately not claimed
|
## Deliberately not claimed
|
||||||
|
|
||||||
@@ -63,15 +78,26 @@ fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip binding
|
|||||||
are not guaranteed globally active in every scene/dashboard state, and the app
|
are not guaranteed globally active in every scene/dashboard state, and the app
|
||||||
never enables SteamVR's experimental overlay overrides on your behalf.
|
never enables SteamVR's experimental overlay overrides on your behalf.
|
||||||
|
|
||||||
This is a compact prototype panel, not yet the proposed polished miniature status
|
This is one compact panel, not a separate miniature status chip. Front-prefix
|
||||||
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
|
loss on repeated submissions (P1) is under separate investigation, **not fixed**
|
||||||
impact and target application compatibility require further headset work.
|
by the chooser/worker documentation or any speculative delivery change. Font
|
||||||
|
coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact
|
||||||
|
and target application compatibility require further headset work.
|
||||||
|
|
||||||
## Inference runtime
|
## Inference runtime
|
||||||
|
|
||||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference
|
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Pinned
|
||||||
runs through the `moondream` Python package and its Kestrel runtime, which you
|
file sizes/hashes and attribution live in `assets/backends/redux.json`; offline
|
||||||
install in your own environment; builds/tests never fetch it. See
|
`--list-models`/`--check-model redux --model-dir /absolute/model` (or
|
||||||
|
`scripts/model-status.py`) verify without inference or downloads. Manifest
|
||||||
|
schema/verification are in `python/frameyap/model_files.py`. The local generic
|
||||||
|
dispatcher resolves a manifest's in-release Python/executable launcher and
|
||||||
|
checks request/reply correlation; a new manifest still needs its own licensed,
|
||||||
|
compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and
|
||||||
|
`--manifest-dir /absolute/manifests` are wired through
|
||||||
|
the installed launcher as an explicit override, not a provisioning command.
|
||||||
|
Inference uses the `moondream` Python package and Kestrel runtime, separately
|
||||||
|
provisioned in your own environment; builds/tests/installer do not pip-install them. See
|
||||||
[third-party notes](third-party.md).
|
[third-party notes](third-party.md).
|
||||||
|
|
||||||
## Developer native build
|
## Developer native build
|
||||||
@@ -142,12 +168,16 @@ kestrel 0.8.0) and local weights:
|
|||||||
```
|
```
|
||||||
|
|
||||||
Use a disposable text destination first. `--run` loads the model but does not
|
Use a disposable text destination first. `--run` loads the model but does not
|
||||||
record until an explicit recording control. Click Insert only after focusing your
|
record until an explicit recording control. Click Type only after focusing your
|
||||||
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
|
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
|
||||||
and terminates only the owned worker. Installer upgrades refuse an active app.
|
and terminates only the owned worker. Installer upgrades refuse an active app.
|
||||||
|
|
||||||
The worker API is intentionally small: a fork can replace the worker implementation
|
The `Controller` receives injected `ControllerAudio`, `ControllerWorker`,
|
||||||
or add its own model/API integration without changing overlay and delivery code.
|
`ControllerFocus` and delivery factory interfaces; hardware-free fakes can test
|
||||||
|
state, retry, authorization and mic lifetime without initializing OpenVR or
|
||||||
|
recording speech. The worker API is intentionally small: a fork can replace the
|
||||||
|
worker implementation or add its own model/API integration without changing
|
||||||
|
overlay and delivery code.
|
||||||
The default product remains local-only Redux; extending a fork does not authorize
|
The default product remains local-only Redux; extending a fork does not authorize
|
||||||
sending existing users' audio to a service.
|
sending existing users' audio to a service.
|
||||||
|
|
||||||
|
|||||||
+61
-39
@@ -1,11 +1,16 @@
|
|||||||
# FrameYap native dictation overlay — proposal
|
# FrameYap native dictation overlay — design and remaining goals
|
||||||
|
|
||||||
Standalone project design. This document is the full target design, not a blanket
|
Standalone project design. This document is the full target design, not a blanket
|
||||||
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
|
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
|
Redux adapter, review-first Gamescope insertion and a user-local archive installer
|
||||||
|
with explicit source-build option. Redux's pinned manifest, offline status CLI,
|
||||||
|
manifest-driven dispatcher and in-panel Models selection/confirmation are
|
||||||
|
implemented locally. The installed chooser/installer handoff is **not yet
|
||||||
|
end-to-end verified on Frame**; the native-only archive needs an external CPU
|
||||||
|
runtime for speech.
|
||||||
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
|
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
|
||||||
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
|
accepted for live automatic typing on Frame. A separate status chip and full hardware
|
||||||
acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md).
|
acceptance remain possible future work. See [build and scope](build.md) and [third-party notes](third-party.md).
|
||||||
Future work is tracked in [TODO.md](../TODO.md).
|
Future work is tracked in [TODO.md](../TODO.md).
|
||||||
|
|
||||||
## Recommendation
|
## Recommendation
|
||||||
@@ -17,7 +22,9 @@ as explicit fallbacks. No desktop ASR server, network hop, LLM cleanup, scene
|
|||||||
renderer, avatar, desktop capture or root service is needed in the primary path.
|
renderer, avatar, desktop capture or root service is needed in the primary path.
|
||||||
|
|
||||||
A first-class product goal is a **one-command GitHub install without a Steam store
|
A first-class product goal is a **one-command GitHub install without a Steam store
|
||||||
AppID**. Package a prebuilt native executable and isolated CPU runtime; use a normal
|
AppID**. Package a prebuilt native executable; the current native-only archive requires a
|
||||||
|
separately supplied, compatible CPU runtime (no automatic pip install). A future
|
||||||
|
isolated runtime bundle requires its own license and compatibility audit. Use a normal
|
||||||
OpenVR application key for registration, not Steamworks. Installation must remain
|
OpenVR application key for registration, not Steamworks. Installation must remain
|
||||||
user-local with opt-in autolaunch. See [installation design](install-design.md).
|
user-local with opt-in autolaunch. See [installation design](install-design.md).
|
||||||
|
|
||||||
@@ -31,12 +38,14 @@ one persistent local Parakeet Redux CPU worker
|
|||||||
correlated literal transcript → focus/delivery policy
|
correlated literal transcript → focus/delivery policy
|
||||||
↓
|
↓
|
||||||
Gamescope IME set_string + commit → focused Frame application
|
Gamescope IME set_string + commit → focused Frame application
|
||||||
↘ overlay preview / error / explicit Insert when delivery is unsafe
|
↘ overlay preview / error / explicit Type when delivery is unsafe
|
||||||
```
|
```
|
||||||
|
|
||||||
The CPU worker is local process isolation, not remote inference or a service
|
The CPU worker is local process isolation, not remote inference or a service
|
||||||
framework. Implement a small transport owned by this repository. Keep the model
|
framework. The locally implemented manifest dispatcher uses a small bounded
|
||||||
loaded between utterances; do not spawn Python/load 178 MB for every release.
|
transport owned by this repository; only Redux inference is shipped. Keep the
|
||||||
|
model loaded between utterances; do not spawn Python/load 178 MB for every release.
|
||||||
|
A correlated bad transcript is a request-level failure, not a model unload.
|
||||||
|
|
||||||
## Minimal interaction
|
## Minimal interaction
|
||||||
|
|
||||||
@@ -53,27 +62,29 @@ armed remains a possible refinement, not a second implemented overlay. See
|
|||||||
Recording… 00:04 [ Cancel ]
|
Recording… 00:04 [ Cancel ]
|
||||||
|
|
||||||
"The recognized text appears here."
|
"The recognized text appears here."
|
||||||
[ Insert + space ] [ Discard ] [ Insert + Enter — explicit ]
|
[ Type + space ] [ Discard ] [ Type + Enter — explicit ]
|
||||||
```
|
```
|
||||||
|
|
||||||
- States: disabled, warming, ready, recording, transcribing, review, inserted,
|
- States: warming, ready, recording, transcribing, review, input queued,
|
||||||
unavailable/error. Recording uses visible icon + text, not color alone.
|
unavailable/error. Input queued is not an application receipt. Recording uses
|
||||||
|
visible icon + text, not color alone.
|
||||||
- Default Frame bindings: hold right X to speak, release to finish; B cancels,
|
- Default Frame bindings: hold right X to speak, release to finish; B cancels,
|
||||||
A inserts with a trailing space, Y opens/cycles the quick-chat selection.
|
A types with a trailing space, Y opens/cycles the Quick phrases selection.
|
||||||
Overlay Submit or left-grip double-tap submits the selected literal + Enter,
|
Overlay Type + Enter or left-grip double-tap submits the selected literal + Enter,
|
||||||
or pending review + Enter, or Enter alone if neither is present.
|
or pending review (normally + space) then Enter, or Enter alone if neither is present.
|
||||||
The alternate right-grip tap-then-hold gesture and left-grip double-tap Submit
|
The alternate right-grip tap-then-hold gesture and left-grip double-tap Type + Enter
|
||||||
remain available. All are remappable through the Bindings button's SteamVR editor.
|
remain available. All are remappable through the Bindings button's SteamVR editor.
|
||||||
A click-to-start/stop overlay button provides
|
A click-to-start/stop overlay button provides
|
||||||
a binding-independent alternative. Bound recording to 20 seconds; discard
|
a binding-independent alternative. Bound recording to 20 seconds; discard
|
||||||
accidental taps (initial threshold: 200 ms).
|
accidental taps (initial threshold: 200 ms).
|
||||||
- **Quick typing (opt-in, locally implemented):** insert on completion only when
|
- **Auto insert (opt-in, locally implemented):** type on completion only when
|
||||||
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
|
uninterrupted Xwayland target observation remains valid. **Review mode (default):**
|
||||||
wait for Insert. Bring up review on uncertainty rather than silently losing a
|
wait for Type. Bring up review on uncertainty rather than silently losing a
|
||||||
transcript or typing into a new target. Live Frame acceptance remains open.
|
transcript or typing into a new target. Live Frame acceptance remains open.
|
||||||
- Submit requires its own explicit control activation: insert pending review with
|
- Type + Enter requires its own explicit control activation: type pending review
|
||||||
a trailing space, then queue Enter only if the text step succeeds. With no
|
(normally with a trailing space), then queue Enter only if the text step succeeds.
|
||||||
review, it queues only Enter. Never interpret "submit", "delete" or other speech
|
At the 4096-byte bound, preserve the entire transcript without a suffix if no
|
||||||
|
space fits. With no review, it queues only Enter. Never interpret "submit", "delete" or other speech
|
||||||
as commands. Transcription completion never auto-submits.
|
as commands. Transcription completion never auto-submits.
|
||||||
- No generic "undo last dictation" initially: another application's edits/cursor
|
- No generic "undo last dictation" initially: another application's edits/cursor
|
||||||
cannot be reliably rolled back by a guessed number of backspaces.
|
cannot be reliably rolled back by a guessed number of backspaces.
|
||||||
@@ -109,7 +120,7 @@ license-reviewed extraction, never a runtime path into another project's checkou
|
|||||||
|
|
||||||
### Controller bindings
|
### Controller bindings
|
||||||
|
|
||||||
Expose PTT, cancel and explicit insert/Enter as named SteamVR actions; let the
|
Expose PTT, cancel and explicit Type/Type + Enter as named SteamVR actions; let the
|
||||||
user bind them. Do not assume a scene app's left-bumper mapping works globally or
|
user bind them. Do not assume a scene app's left-bumper mapping works globally or
|
||||||
silently takes a game's button away. Check action activity and neutral rearm.
|
silently takes a game's button away. Check action activity and neutral rearm.
|
||||||
|
|
||||||
@@ -143,7 +154,7 @@ The server implements text through synthetic key events and a temporary keymap,
|
|||||||
not a guaranteed rich-text/IME edit operation in every app. Verify actual target
|
not a guaranteed rich-text/IME edit operation in every app. Verify actual target
|
||||||
toolkits and games. Respect singleton/unavailable handling and Steam-keyboard
|
toolkits and games. Respect singleton/unavailable handling and Steam-keyboard
|
||||||
coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe:
|
coexistence. Never use the installed `gamescope-type` CLI as a transcript pipe:
|
||||||
its inspected sample loop is byte-oriented and interprets newline as Submit.
|
its inspected sample loop is byte-oriented and interprets newline as Enter.
|
||||||
|
|
||||||
### 2. Explicit fallbacks, not a framework built up front
|
### 2. Explicit fallbacks, not a framework built up front
|
||||||
|
|
||||||
@@ -183,11 +194,11 @@ Gamescope exposes focus-display/window root properties, but their encoding and
|
|||||||
relationship to seat focus need implementation-specific validation. Do not infer
|
relationship to seat focus need implementation-specific validation. Do not infer
|
||||||
that X display `:0` is always the destination, or that an X focus observation
|
that X display `:0` is always the destination, or that an X focus observation
|
||||||
identifies a native Wayland text field. For unobservable native Wayland focus,
|
identifies a native Wayland text field. For unobservable native Wayland focus,
|
||||||
require explicit review/Insert; do not advertise safe auto-targeting.
|
require explicit review/Type; do not advertise safe auto-targeting.
|
||||||
|
|
||||||
If focus changes, keep the result in review. A fresh Insert explicitly approves
|
If focus changes, keep the result in review. A fresh Type explicitly approves
|
||||||
the current destination and creates a new delivery authorization. Recheck again
|
the current destination and creates a new delivery authorization. Recheck again
|
||||||
at insertion. This minimizes stale delivery but does **not** eliminate a race
|
at typing. This minimizes stale delivery but does **not** eliminate a race
|
||||||
between the final check and global input processing; do not claim otherwise.
|
between the final check and global input processing; do not claim otherwise.
|
||||||
A Wayland roundtrip means compositor processing, not application consumption.
|
A Wayland roundtrip means compositor processing, not application consumption.
|
||||||
Report `input queued`, never `message sent`.
|
Report `input queued`, never `message sent`.
|
||||||
@@ -221,9 +232,11 @@ that tiny dataset; keep transcript visibility and a cheap retry.
|
|||||||
|
|
||||||
Implement a small independent audio/worker adapter with explicit capture,
|
Implement a small independent audio/worker adapter with explicit capture,
|
||||||
single-request bounds, owner-only runtime files, correlated replies and cancellation.
|
single-request bounds, owner-only runtime files, correlated replies and cancellation.
|
||||||
The worker should be implemented independently; do not link, vendor or import
|
The worker is independently implemented; do not link, vendor or import another
|
||||||
another application's speech code. Avoid a generic provider framework: one
|
application's speech code. A narrow manifest dispatcher now selects a checked
|
||||||
explicit Redux worker is enough for the first version.
|
local backend launcher; this is not a general cloud/provider framework and only
|
||||||
|
Redux is supplied with a runtime implementation. Other manifests require their
|
||||||
|
own audited offline engine, launcher and tests.
|
||||||
|
|
||||||
Suggested ownership, introduced only as implementation needs it:
|
Suggested ownership, introduced only as implementation needs it:
|
||||||
|
|
||||||
@@ -243,12 +256,20 @@ processing timeout. No shell commands in IPC and no input authority in the worke
|
|||||||
native kernel pools. Choose measured latency versus compositor contention,
|
native kernel pools. Choose measured latency versus compositor contention,
|
||||||
not the desktop's thread count by habit. No real-time scheduling or permanent
|
not the desktop's thread count by habit. No real-time scheduling or permanent
|
||||||
CPU pinning initially; inspect runtime affinity behaviour during measurement.
|
CPU pinning initially; inspect runtime affinity behaviour during measurement.
|
||||||
- Explicit local model path and offline loading. Missing runtime/weights produces
|
- Explicit local model path and offline loading. Pinned files and attribution live
|
||||||
an actionable error, not an unsolicited download/network fallback.
|
in `assets/backends/redux.json`; `python/frameyap/model_files.py` validates its
|
||||||
- Optional explicit enable/warm-up before first PTT; warming must not record audio.
|
schema and hashes. The CLI's offline `--list-models`/`--check-model` checks
|
||||||
Expose that lifecycle explicitly rather than warming on import or construction.
|
never start inference. Settings → Models selects/saves the manifest ID and
|
||||||
Otherwise display first-use loading honestly. Keep
|
restarts the worker, invalidating any audio/review/delivery authorization.
|
||||||
model reuse after normal completion; cancellation may restart the owned worker.
|
Install requires a separate confirmation showing source, size, license,
|
||||||
|
attribution and exact-manifest SHA-256; only that click hands off to the
|
||||||
|
installer for pinned model files. Missing runtime/weights produces an
|
||||||
|
actionable error, not an unsolicited download/network fallback.
|
||||||
|
- Native `--run` explicitly starts offline verification then warms a verified
|
||||||
|
selected local model before PTT; warming must not record audio. Missing or
|
||||||
|
unverified files disable Record rather than prompting a background download.
|
||||||
|
Keep model reuse after normal completion and request-local transcript errors;
|
||||||
|
cancellation may restart the owned worker.
|
||||||
- Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded
|
- Use a private owner-only directory under `$XDG_RUNTIME_DIR` for bounded
|
||||||
tmpfs-backed clips; remove them on completion, error, cancellation and shutdown.
|
tmpfs-backed clips; remove them on completion, error, cancellation and shutdown.
|
||||||
Avoid persistent audio/transcripts by default. Local IPC is not a network hop;
|
Avoid persistent audio/transcripts by default. Local IPC is not a network hop;
|
||||||
@@ -278,14 +299,14 @@ These are proposed implementation gates, **not completed acceptance**:
|
|||||||
confirm dashboard/hand placement, input events, close/reopen and no scene-focus
|
confirm dashboard/hand placement, input events, close/reopen and no scene-focus
|
||||||
takeover. Validate global PTT separately rather than blocking the clickable
|
takeover. Validate global PTT separately rather than blocking the clickable
|
||||||
prototype on experimental override support.
|
prototype on experimental override support.
|
||||||
3. **Real dictation path:** microphone → local Redux → preview → explicit insert
|
3. **Real dictation path:** microphone → local Redux → preview → explicit Type
|
||||||
into a disposable target; then enable quick typing after target tracking tests.
|
into a disposable target; then validate opt-in Auto insert after target tracking tests.
|
||||||
Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate
|
Test Unicode, punctuation, long bounded clips, silence, cancellation, duplicate
|
||||||
replies, lost mic, worker crash and missing model without persisting speech.
|
replies, lost mic, worker crash and missing model without persisting speech.
|
||||||
4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game
|
4. **Target matrix:** Xwayland terminal/browser and selected native/Proton game
|
||||||
text fields; Steam keyboard coexistence; native Wayland targets separately.
|
text fields; Steam keyboard coexistence; native Wayland targets separately.
|
||||||
Test focus changes during capture/inference, rapid loss/regain, held modifiers,
|
Test focus changes during capture/inference, rapid loss/regain, held modifiers,
|
||||||
explicit Insert retargeting, no hidden Enter, and no second delivery.
|
explicit Type retargeting, no hidden Enter, and no second delivery.
|
||||||
5. **In-headset acceptance:** readable feedback and comfortable PTT; measured
|
5. **In-headset acceptance:** readable feedback and comfortable PTT; measured
|
||||||
release-to-insert latency and compositor timing while an actual scene runs;
|
release-to-insert latency and compositor timing while an actual scene runs;
|
||||||
no noticeable sustained thermal/battery regression. Set numeric budgets after
|
no noticeable sustained thermal/battery regression. Set numeric budgets after
|
||||||
@@ -301,8 +322,9 @@ without moving recognition off Frame. It is not part of this initial design.
|
|||||||
The default hardware-free build needs only CMake and a C++20 compiler (Python
|
The default hardware-free build needs only CMake and a C++20 compiler (Python
|
||||||
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
|
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
|
||||||
SDL3, FreeType and Wayland client/generated protocol bindings. A separately
|
SDL3, FreeType and Wayland client/generated protocol bindings. A separately
|
||||||
authorized Python Redux environment is explicitly supplied at launch. Pin revisions
|
authorized Python Redux environment is explicitly supplied at launch; the
|
||||||
and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery.
|
native-only installer neither bundles it nor pip-installs one. Pin revisions
|
||||||
|
and review licenses for each introduced dependency. No automatic fetch/install in configure or normal tests; no external checkout discovery.
|
||||||
|
|
||||||
Hardware-free tests should cover state transitions, bounded PCM/transcripts,
|
Hardware-free tests should cover state transitions, bounded PCM/transcripts,
|
||||||
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
|
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
|
||||||
|
|||||||
+73
-99
@@ -1,108 +1,82 @@
|
|||||||
# Installation and distribution goal
|
# Installation goal and current boundary
|
||||||
|
|
||||||
**User goal:** install from GitHub with a `curl … | bash`-style command, without a
|
Goal: a one-command, pinned GitHub release install for Steam Frame without a Steam
|
||||||
Steam store AppID. An idempotent archive installer and native-only local artifacts
|
store AppID, sudo or end-user compiler. **No public archive or verified clean
|
||||||
are now implemented/tested; no public release is published. A bundled-ASR
|
install is published. Do not advertise a `curl | sh` command as functional.**
|
||||||
experience is not yet offered. This document retains the target
|
The local installer has binary-archive and explicitly provisioned source-build
|
||||||
design; see [current packaging](packaging.md) and [third-party notes](third-party.md).
|
modes, machine-readable plans/results and an attended TTY path. The current
|
||||||
Planned installer work (binary-or-source choice, flag-driven operation) is in
|
native-only artifact does not include or pip-install an ASR runtime; it is not
|
||||||
[TODO.md](../TODO.md).
|
a one-command voice-typing experience. See [packaging](packaging.md) for exact
|
||||||
|
flags and [third-party inventory](third-party.md) for open license/ABI audits.
|
||||||
|
|
||||||
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable
|
`scripts/install-preflight.sh` is a read-only Linux ARM64/glibc/bootstrap check;
|
||||||
for the **proposed** Linux ARM64 glibc package format and has the expected basic
|
`--source` adds toolchain/library checks. It does not download, install, register,
|
||||||
bootstrap utilities. It reports system Python, Git and uv, but none is required
|
or certify model/runtime compatibility or a minimum libc version. Its report of
|
||||||
for the intended bundled release. A read-only check on one Frame observed
|
Python/Git/uv availability is not a runtime guarantee: `install.sh` currently
|
||||||
Python 3.12.3 and Git, but not uv; availability may change. This script
|
needs Python 3.12+ **for its bootstrap**, while native-only voice inference needs
|
||||||
is not an installer or a model/runtime compatibility test; it has no downloads,
|
a separately provisioned compatible CPU Python environment and pinned weights.
|
||||||
registration, SteamVR initialization or persistent changes. No glibc minimum can
|
|
||||||
be certified until release artifacts are chosen and tested.
|
|
||||||
|
|
||||||
## Non-Steam overlay identity
|
## OpenVR identity
|
||||||
|
|
||||||
OpenVR overlay applications do not require a Steam store AppID or Steamworks.
|
`local.frameyap.overlay` is a string OpenVR application key, **not** a Steam
|
||||||
The native executable initializes as `VRApplication_Overlay`. For discoverability
|
store AppID. The installer creates a manifest/desktop launcher user-locally but
|
||||||
and optional autolaunch, register an OpenVR application manifest with a stable,
|
does not register/launch the app by default. Explicit `frameyap --register
|
||||||
project-owned **string application key** (proposed: `local.frameyap.overlay`).
|
/absolute/manifest/path` uses the OpenVR registration API; registration alone
|
||||||
That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam
|
did not reveal a launcher in the first checked dashboard menu. On one Frame the
|
||||||
library shortcut should be necessary for the normal route.
|
user opened the panel from the **Non-Steam** section and quit; shortcut discovery
|
||||||
|
and persistence after a normal restart are still unverified. Optional
|
||||||
|
`--autolaunch`/`--no-autolaunch` on the installer explicitly request OpenVR
|
||||||
|
registration/autolaunch choices; no SteamVR settings or sessions are changed
|
||||||
|
without that request. Unregister explicitly before uninstall. SteamVR must
|
||||||
|
already be available for registration; never start/restart it for installation.
|
||||||
|
|
||||||
The manifest identifies the installed executable; the installer/registration
|
## Installation workflows
|
||||||
helper should use `IVRApplications::AddApplicationManifest` and the corresponding
|
|
||||||
remove operation, not hand-edit Steam's internal JSON. Autolaunch uses the OpenVR
|
|
||||||
application setting only when explicitly requested. Validate the exact manifest,
|
|
||||||
launch behaviour, registration persistence and uninstall on native Frame before
|
|
||||||
claiming this route works end to end. Existing probes established overlay client
|
|
||||||
initialization, not manifest installation.
|
|
||||||
|
|
||||||
SteamVR/OpenVR must already be installed and usable. If registration needs a
|
- **Binary mode** (no compiler): verify a locally supplied ARM64 archive digest,
|
||||||
running runtime, defer it to the first explicit launch rather than starting or
|
or, after a vetted release actually exists, explicitly approve retrieval of
|
||||||
restarting SteamVR behind the user's back.
|
a pinned `v0.1.YYYYMMDDHHMM` GitHub tag and checksum. No moving `latest` tag.
|
||||||
|
`--mode binary --archive FILE --sha256 HASH --version 0.1.YYYYMMDDHHMM`
|
||||||
|
selects the local-artifact path.
|
||||||
|
- **Source mode**: requires explicit local source, SDK, SDL/OpenVR libraries and
|
||||||
|
their notices, CMake/C++20, native build dependencies and a version. It builds,
|
||||||
|
stages, packages and continues through local installation; it does **not**
|
||||||
|
provision ASR packages or bypass producer license obligations. See
|
||||||
|
[packaging](packaging.md) for all flags.
|
||||||
|
- **Model**: only an explicit `--install-model --backend redux --yes` fetches
|
||||||
|
pinned public files for the *already installed* backend. Inspect the read-only
|
||||||
|
`--print-plan --json` first; it includes model size, attribution and license
|
||||||
|
metadata from `assets/backends/redux.json`. `--expected-manifest-sha256 HASH`
|
||||||
|
binds consent to the exact installed manifest bytes and fails before model
|
||||||
|
directory creation/network if they changed. The local in-panel chooser shows
|
||||||
|
source, size, license text, attribution and manifest digest, then requires a
|
||||||
|
second **Confirm Install** click; the installed/native UI route still needs
|
||||||
|
clean-target and headset acceptance. `--without-model` permits an
|
||||||
|
archive install without bundled model files. Neither operation installs Torch,
|
||||||
|
moondream, Kestrel or an interpreter. Launch paths to an independently
|
||||||
|
authorized runtime/model can be set in `paths.conf`.
|
||||||
|
- **Noninteractive**: supply flags and `--yes` for network/model consent.
|
||||||
|
`--print-plan` is read-only; `--json` provides structured results/errors and
|
||||||
|
progress events for a model download. No prompt reads stdin in a pipe. An
|
||||||
|
empty TTY invocation offers a local menu and prints equivalent flags.
|
||||||
|
|
||||||
## Intended user experience
|
Installation is user-local under XDG data/config paths with a managed launcher,
|
||||||
|
retained rollback, SHA-256/path validation, foreign-file refusal and a lock
|
||||||
|
shared with the app. No OS package changes, udev rule, root service, Steam store
|
||||||
|
listing, unrelated application dependency, microphone recording, input injection
|
||||||
|
or automatic update daemon. Hashes detect accidental/unauthorized alteration
|
||||||
|
of a downloaded artifact but do not authenticate a compromised publisher;
|
||||||
|
release metadata needs independent trust. No installer operation silently runs
|
||||||
|
pip or launches inference. Native-only archives support overlay checks but need
|
||||||
|
an externally provisioned runtime/weights before voice typing. Selecting an
|
||||||
|
uninstalled backend does not authorize a download or supply its inference engine.
|
||||||
|
|
||||||
1. Run one documented command from the eventual GitHub repository/release.
|
## Gate before publishing the goal as fulfilled
|
||||||
2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and
|
|
||||||
explains/downloads the application, compatible CPU runtime and pinned model.
|
|
||||||
3. User-local installation provides a simple `frameyap` launcher, desktop
|
|
||||||
entry where supported, and an OpenVR manifest. No compiler, engine checkout,
|
|
||||||
Python dependency troubleshooting or separate ASR server for ordinary users.
|
|
||||||
4. The user explicitly launches/enables dictation. No installation-time microphone
|
|
||||||
recording, input injection, inference benchmark or overlay takeover.
|
|
||||||
|
|
||||||
Illustrative command shape only; `OWNER`, `REPO` and `VERSION` are placeholders:
|
Vet the **exact** release closure/licenses, ARM64 symbol versions/loader,
|
||||||
|
model attribution and compatible CPU Python environment; establish a tested
|
||||||
```sh
|
libc/runtime floor, then publish and authenticate a checksummed archive from a
|
||||||
curl --fail --silent --show-error --location \
|
clean tag. On a clean supported Frame, install without a compiler/sudo/store ID,
|
||||||
https://raw.githubusercontent.com/OWNER/REPO/VERSION/install.sh | bash
|
load Redux, type into a disposable owned target and validate rollback/uninstall,
|
||||||
```
|
foreign-file failures, autolaunch off, and no unintended session changes. Separately
|
||||||
|
validate microphone → review → delivered input and headset comfort. Offline
|
||||||
Also document a download-inspect-run path for users who do not want to pipe remote
|
installer fixture tests and a local native build are not those acceptance gates.
|
||||||
code into a shell. Pin a release/tag instead of executing a moving branch by default.
|
|
||||||
Offer explicit version selection and noninteractive flags; do not read interactive
|
|
||||||
confirmation from stdin while the installer itself is arriving through that pipe.
|
|
||||||
|
|
||||||
## Packaging boundary
|
|
||||||
|
|
||||||
- Prebuilt ARM64 executable plus a known-compatible, isolated CPU inference runtime;
|
|
||||||
no external application libraries/assets and no system Python modification.
|
|
||||||
- Model fetched during explicit installation/setup, with pinned revision/hash and
|
|
||||||
attribution. A documented `--without-model` option can defer the large download.
|
|
||||||
No surprise first-utterance downloads. Runtime components are installed from
|
|
||||||
their own package index rather than redistributed in our tarball.
|
|
||||||
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
|
|
||||||
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
|
|
||||||
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
|
|
||||||
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
|
|
||||||
`/dev/uinput` permission changes or modifications to unrelated launchers.
|
|
||||||
- No Steam store AppID, Steamworks SDK, root service or network ASR dependency.
|
|
||||||
- Third-party notices for everything we redistribute must be included with a release.
|
|
||||||
|
|
||||||
## Installer lifecycle and safety
|
|
||||||
|
|
||||||
- Detect architecture, libc and prerequisites first. Unsupported hosts fail with a
|
|
||||||
clear explanation; never install an x86 payload silently on ARM64.
|
|
||||||
- Download to a staging directory, check versioned SHA-256 manifests and archive
|
|
||||||
paths, then atomically select the completed version. HTTPS/checksums alone do not
|
|
||||||
authenticate a compromised publisher; use signed release metadata if provided.
|
|
||||||
- Keep configuration across upgrades; retain the previous version for rollback.
|
|
||||||
Refuse or defer replacement while this application's process is running rather
|
|
||||||
than killing arbitrary processes. Never touch SSH or unrelated sessions.
|
|
||||||
- Autostart is opt-in (`--autostart` or explicit settings); do not enable a systemd
|
|
||||||
service or SteamVR autolaunch by default. Do not enable overlay input overrides.
|
|
||||||
- Uninstall removes only owned launcher, manifest registration and install files;
|
|
||||||
model/config deletion is separately explicit. Preserve other SteamVR apps.
|
|
||||||
- No update daemon initially. A deliberate rerun/update command is sufficient.
|
|
||||||
|
|
||||||
## Acceptance before advertising one-command installation
|
|
||||||
|
|
||||||
- Clean supported Frame: install without sudo/compiler/engine checkout/store AppID;
|
|
||||||
launch overlay, load local Redux and type into an owned disposable target.
|
|
||||||
- Normal use after installation needs no network connection or desktop ASR host.
|
|
||||||
- Failed download/hash, unsupported architecture, low disk space and interrupted
|
|
||||||
upgrades leave a usable previous install or a cleanly reported failure.
|
|
||||||
- Reinstall, rollback and uninstall preserve unrelated data and SteamVR entries.
|
|
||||||
- Noninteractive piped invocation never hangs on stdin; inspection-first path works.
|
|
||||||
- Autolaunch remains off unless chosen; uninstall removes only our registration.
|
|
||||||
- Hardware-free installer tests use temporary homes, mocked runtime registration
|
|
||||||
and local fixture artifacts. Never exercise a real user's Steam configuration
|
|
||||||
in ordinary CI/CTest.
|
|
||||||
+89
-31
@@ -2,10 +2,12 @@
|
|||||||
|
|
||||||
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
|
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
|
||||||
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
|
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
|
||||||
Native `--run` wiring in `src/main.cpp` is implemented behind the explicit
|
Native `--run` wiring in `src/main.cpp` is behind the explicit
|
||||||
`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful
|
`FRAMEYAP_NATIVE` build option. The Models chooser, offline verification and
|
||||||
compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a
|
explicit installer handoff are implemented locally, **not** an accepted Frame
|
||||||
normal build or test.
|
install/voice-typing path or a published release. Neither hardware-free tests
|
||||||
|
nor a successful compile establish Frame input, visibility, comfort or text
|
||||||
|
delivery. Launching the runtime is explicit, never part of a normal build or test.
|
||||||
|
|
||||||
## Rendering and controls
|
## Rendering and controls
|
||||||
|
|
||||||
@@ -49,7 +51,9 @@ This replaces the raw-upload rendering path; headset flicker acceptance still
|
|||||||
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
|
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
|
||||||
|
|
||||||
The header shows local time and date instead of the former on-device/review
|
The header shows local time and date instead of the former on-device/review
|
||||||
and current-mount labels. It updates when the displayed minute or date changes,
|
and current-mount labels. Settings explains Hold Quit (hold 0.9 seconds then
|
||||||
|
release) and Lasers anytime (system-wide lasers may affect games). The review
|
||||||
|
tab describes Type and Type + Enter. It updates when the displayed minute or date changes,
|
||||||
not every frame. Settings toggles 12/24-hour time and cycles date Off →
|
not every frame. Settings toggles 12/24-hour time and cycles date Off →
|
||||||
MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display;
|
MM/DD/YYYY → DD/MM/YYYY → YYYY-MM-DD → Off. These only affect display;
|
||||||
mount choices remain in Settings.
|
mount choices remain in Settings.
|
||||||
@@ -58,23 +62,25 @@ The complete transcript preview is paginated by glyph width and four-line
|
|||||||
height; Previous and Next navigate it without changing the source transcript.
|
height; Previous and Next navigate it without changing the source transcript.
|
||||||
Status fits on the single status line; the old bottom detail label is gone.
|
Status fits on the single status line; the old bottom detail label is gone.
|
||||||
The footer remains available on all tabs: Record (labelled Stop while recording),
|
The footer remains available on all tabs: Record (labelled Stop while recording),
|
||||||
Cancel, Insert, Submit, Hold Quit. Hold Quit needs a 900 ms press and release on
|
Cancel, Type, Type + Enter, Hold Quit. Hold Quit needs a 900 ms press and release on
|
||||||
that same button; its thin progress bar shows the hold. Record can retry after an
|
that same button; its thin progress bar shows the hold. Record can retry after an
|
||||||
error; it is disabled while warming/transcribing and until an existing review is
|
error; it is disabled while warming/transcribing and until an existing review is
|
||||||
inserted or discarded. Cancel can stop worker startup. Insert and Submit are
|
typed or discarded. Cancel can stop worker startup. Type and Type + Enter are
|
||||||
disabled during recording and transcription. A pointer action requires
|
disabled during recording and transcription. A pointer action requires
|
||||||
a press/release on the same enabled control from the same cursor; focus loss,
|
a press/release on the same enabled control from the same cursor; focus loss,
|
||||||
tab changes, action-state changes and relocation clear pending presses. Submit is *always* a separate
|
tab changes, action-state changes and relocation clear pending presses. Type + Enter
|
||||||
deliberate action, not inferred from text. Insert appends a trailing space (without
|
is *always* a separate deliberate action, not inferred from text. Type normally
|
||||||
doubling an existing trailing space). Submit inserts any pending review and then
|
appends a trailing space (without doubling an existing one); a full 4096-byte
|
||||||
queues Enter; with no pending text it queues Enter only. Y opens the quick-chat
|
transcript without room for that suffix is queued unchanged, with no extra error
|
||||||
list over the review area; each further Y press cycles its highlighted choice.
|
for the missing space. Type + Enter types any pending review and then queues
|
||||||
Cancel closes the picker without discarding an existing review. Submit sends the
|
Enter; with no pending text it queues Enter only. Y opens the Quick phrases
|
||||||
selected text *without* a trailing space, then Enter. The choices are short
|
list over the review area; each further Y press cycles its highlighted choice. Cancel closes the picker without discarding an
|
||||||
|
existing review. Type + Enter sends the selected phrase *without* a trailing
|
||||||
|
space, then Enter. The choices are short
|
||||||
single-line literals, not speech commands. A failed text step never proceeds to
|
single-line literals, not speech commands. A failed text step never proceeds to
|
||||||
Enter. Recording never automatically submits. Auto insert, when
|
Enter. Recording never automatically submits. Auto insert, when
|
||||||
explicitly enabled, can queue text + space after transcription only under the
|
explicitly enabled, can queue text (normally + space) after transcription only
|
||||||
stable Xwayland focus guard described below.
|
under the stable Xwayland focus guard described below.
|
||||||
|
|
||||||
### Bindings button
|
### Bindings button
|
||||||
|
|
||||||
@@ -107,6 +113,8 @@ installer creates one with defaults on first install. Copy the shipped
|
|||||||
"input_priority": "normal",
|
"input_priority": "normal",
|
||||||
"advanced_debug": false,
|
"advanced_debug": false,
|
||||||
"auto_insert": false,
|
"auto_insert": false,
|
||||||
|
"close_mic_when_idle": false,
|
||||||
|
"backend": "redux",
|
||||||
"lock_layout": false,
|
"lock_layout": false,
|
||||||
"clock_24h": false,
|
"clock_24h": false,
|
||||||
"date_format": "mdy",
|
"date_format": "mdy",
|
||||||
@@ -149,6 +157,18 @@ config, retaining other fields and formatting; invalid/unwritable configs are
|
|||||||
left untouched and return failure. The installer backs up original bytes before
|
left untouched and return failure. The installer backs up original bytes before
|
||||||
repairing invalid values, while valid `true` and `false` are retained.
|
repairing invalid values, while valid `true` and `false` are retained.
|
||||||
|
|
||||||
|
`close_mic_when_idle` is a separate boolean, default **false**, also available
|
||||||
|
as Settings → **Close mic when idle**. Normally the SDL capture device stays
|
||||||
|
open while Ready and idle samples are discarded. Enabling this toggle closes it
|
||||||
|
between clips and opens it on PTT: this avoids an open idle capture device but
|
||||||
|
can cause an audio spike (observed with per-PTT transitions on Frame), startup
|
||||||
|
latency or first-syllable clipping. Settings persistently displays **OFF: discard
|
||||||
|
idle audio; ON: spike / start latency** below the toggle, as well as an ON/OFF
|
||||||
|
indicator; the status/detail line also explains a change when toggled. The
|
||||||
|
native setting is saved to `config.json`; a failed save applies only for this
|
||||||
|
session and warns. This is not a mute switch for other applications. Quit and
|
||||||
|
failure still close the device.
|
||||||
|
|
||||||
`auto_insert` is a separate boolean, default `false`, also available as a
|
`auto_insert` is a separate boolean, default `false`, also available as a
|
||||||
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
|
Settings toggle. Only a **new** recording arms it. It observes the Xwayland
|
||||||
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
||||||
@@ -156,41 +176,77 @@ display selected by `DISPLAY`; its root `_NET_ACTIVE_WINDOW` and
|
|||||||
Both properties and focus are rechecked after IME lease acquisition. A watched
|
Both properties and focus are rechecked after IME lease acquisition. A watched
|
||||||
focus-out, root focus-property change (even if the same window returns), window
|
focus-out, root focus-property change (even if the same window returns), window
|
||||||
destruction, held keyboard key, missing X display or any disagreement permanently
|
destruction, held keyboard key, missing X display or any disagreement permanently
|
||||||
disarms that clip. The transcript then remains for explicit review/Insert.
|
disarms that clip. The transcript then remains for explicit review/Type.
|
||||||
Native Wayland focus and child text-field focus cannot be safely inferred here;
|
Native Wayland focus and child text-field focus cannot be safely inferred here;
|
||||||
those cases fall back to review. No automatic Enter, speech commands or retry.
|
those cases fall back to review. No automatic Enter, speech commands or retry.
|
||||||
The compositor can still change focus in the gap between the final check and
|
The compositor can still change focus in the gap between the final check and
|
||||||
global delivery, and IME commit is not an application receipt. This path has
|
global delivery, and IME commit is not an application receipt. This path has
|
||||||
offline synthetic focus tests and a separate owned-target IME fixture; live
|
offline synthetic focus tests and a separate owned-target IME fixture; live
|
||||||
speech-driven Auto Insert, target coverage and headset acceptance remain
|
speech-driven Auto insert, target coverage and headset acceptance remain
|
||||||
unverified. The setting is preserved on upgrade
|
unverified. The setting is preserved on upgrade
|
||||||
and a failed preference write applies only to the current session.
|
and a failed preference write applies only to the current session.
|
||||||
|
|
||||||
`quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings,
|
`quick_inputs` is an editable list of 1–6 nonempty, printable ASCII strings,
|
||||||
each at most 64 characters. Edit the JSON file and restart; there is no headset
|
each at most 64 characters. Edit the JSON file and restart; there is no headset
|
||||||
text editor. Inputs are literal (not expanded or interpreted by FrameYap) and
|
text editor. Inputs are literal (not expanded or interpreted by FrameYap) and
|
||||||
are sent to the current Gamescope focus, so check the destination before Submit.
|
are sent to the current Gamescope focus, so check the destination before Type + Enter.
|
||||||
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
|
`buttons` maps named OpenVR actions (`left_grip`, `right_grip`, `ptt`, `cancel`,
|
||||||
`insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME`
|
`insert`, `enter`, `quick_chat`) to Frame physical `/user/hand/{left|right}/input/NAME`
|
||||||
button paths. Omitted actions retain their bundled defaults; an empty string
|
button paths. Omitted actions retain their bundled defaults; an empty string
|
||||||
disables a mapping, including after an upgrade. The Frame defaults are right
|
disables a mapping, including after an upgrade. The Frame defaults are right
|
||||||
X = hold-to-talk, B = Cancel, A = Insert + space, Y = quick chat. Enter has no
|
X = hold-to-talk, B = Cancel, A = Type + space, Y = Quick phrases. Enter has no
|
||||||
single-button mapping by default; the left grip double-tap still submits.
|
single-button mapping by default; the left grip double-tap still requests Type + Enter.
|
||||||
An existing config mapping `enter` to right Y is migrated to quick chat in memory
|
An existing config mapping `enter` to right Y is migrated to Quick phrases in memory
|
||||||
when `quick_chat` is absent; this does not overwrite custom mappings.
|
when `quick_chat` is absent; this does not overwrite custom mappings.
|
||||||
Existing configs with empty actions retain those disabled mappings; change them explicitly
|
Existing configs with empty actions retain those disabled mappings; change them explicitly
|
||||||
or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized;
|
or use SteamVR's binding editor. Paths must be distinct. Only the Frame binding is customized;
|
||||||
SteamVR user overrides may still supersede it. On customized launches a generated
|
SteamVR user overrides may still supersede it. On customized launches a generated
|
||||||
action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings`
|
action manifest and adjacent bindings are placed in `$XDG_CACHE_HOME/frameyap/bindings`
|
||||||
(or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The
|
(or `~/.cache/frameyap/bindings`); the bundled manifest remains unchanged. The
|
||||||
config is read once at launch, not hot-reloaded. On install/upgrade the installer
|
config is read once at launch, not hot-reloaded (the in-panel backend selection
|
||||||
fills missing fields, removes retired keys, and resets invalid entries. It saves
|
is saved separately). On install/upgrade the installer fills known missing fields,
|
||||||
the exact prior bytes under `config.json.backup-*` before a repair and refuses
|
including `close_mic_when_idle` and `backend`, removes retired keys and resets
|
||||||
symlink/oversized config paths; valid customizations remain intact. The installed
|
invalid entries. It saves the exact prior bytes under `config.json.backup-*`
|
||||||
launcher no longer pins `--font`, so this selection takes effect. Direct native
|
before a repair and refuses symlink/oversized config paths; valid customizations
|
||||||
|
remain intact. The installed launcher no longer pins `--font`, so this selection takes effect. Direct native
|
||||||
launches with bad JSON, colors or button mappings fail startup rather than
|
launches with bad JSON, colors or button mappings fail startup rather than
|
||||||
silently changing input behavior.
|
silently changing input behavior.
|
||||||
|
|
||||||
|
### Models / backends (local implementation)
|
||||||
|
|
||||||
|
Settings → **Models / backends** opens a paginated local chooser. The saved
|
||||||
|
`"backend": "redux"` selects a *manifest ID*, not a model download;
|
||||||
|
`--backend ID` overrides it for that `--run` invocation. Selecting a listed backend
|
||||||
|
restarts the owned worker and closes capture, discarding pending audio, review
|
||||||
|
and any prior focus/delivery authorization. A failed preference write leaves the
|
||||||
|
choice active only for this session. The currently listed Redux model is the
|
||||||
|
only inference implementation supplied; adding a manifest alone does not add
|
||||||
|
an inference runtime. The Models tab labels local checks as checking, missing
|
||||||
|
(`not_installed`), invalid, or installed/verified; for the selected model it
|
||||||
|
also reports loading, ready or failed. An install-in-progress note reports
|
||||||
|
model provisioning, followed by a new offline check; a verified status alone
|
||||||
|
is not proof the CPU runtime loaded. Recording is disabled until the selected
|
||||||
|
files verify offline and the worker warms; missing files never trigger a silent
|
||||||
|
download. Model loading and the default open-while-Ready idle microphone policy
|
||||||
|
are unchanged. The panel's state and successful helper calls do **not** prove
|
||||||
|
microphone transcription, input delivery or headset acceptance.
|
||||||
|
|
||||||
|
For a missing/invalid selected model, **Install** first presents a separate
|
||||||
|
confirmation showing pinned source, approximate download size, license text,
|
||||||
|
attribution and the exact raw manifest SHA-256 fingerprint. Only **Confirm
|
||||||
|
Install** launches the local installer with that fingerprint; leaving the view
|
||||||
|
or changing metadata invalidates consent. The installer rechecks the *installed*
|
||||||
|
manifest bytes under its model lock before creating a download target or using
|
||||||
|
the network, downloads only on the explicit click, hashes pinned files and then
|
||||||
|
the app checks them again offline before enabling recording. Existing invalid
|
||||||
|
or unsafe model files are refused rather than silently overwritten. This
|
||||||
|
installs model files, **not** Python, Torch, moondream, Kestrel or other runtime
|
||||||
|
dependencies.
|
||||||
|
The source-tree UI needs an installed release for installer-backed provisioning;
|
||||||
|
a hand-edited manifest is not an approved artifact. The installer handoff and
|
||||||
|
native UI have not yet been accepted on a clean Frame. See
|
||||||
|
[packaging](packaging.md#consumer) for an inspection-first CLI path.
|
||||||
|
|
||||||
### Experimental controller input priority
|
### Experimental controller input priority
|
||||||
|
|
||||||
There are two independent gates:
|
There are two independent gates:
|
||||||
@@ -348,7 +404,7 @@ recentring and readability still require a separately authorized headset check.
|
|||||||
|
|
||||||
The opt-in native `--check-controls` probe logs pointer counters and action
|
The opt-in native `--check-controls` probe logs pointer counters and action
|
||||||
callbacks to the terminal rather than repainting them on the panel. Its canvas
|
callbacks to the terminal rather than repainting them on the panel. Its canvas
|
||||||
stays static for Record/Cancel/Insert/Enter clicks so those clicks can be checked
|
stays static for Record/Cancel/Type/Type + Enter clicks so those clicks can be checked
|
||||||
without diagnostic texture uploads. Switching tabs or mount still updates
|
without diagnostic texture uploads. Switching tabs or mount still updates
|
||||||
the visible panel. Diagnostics identify `renderer=Vulkan` and count
|
the visible panel. Diagnostics identify `renderer=Vulkan` and count
|
||||||
`textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions.
|
`textureUploads`; raw/file `ImageLoaded` events are not GPU upload completions.
|
||||||
@@ -367,8 +423,10 @@ cmake --build build-native --target frameyap_texture_check
|
|||||||
|
|
||||||
It does not initialize OpenVR or establish compositor/headset acceptance.
|
It does not initialize OpenVR or establish compositor/headset acceptance.
|
||||||
|
|
||||||
`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert,
|
`assets/actions.json` names seven actions: left/right grip, PTT, cancel,
|
||||||
Enter. `bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
|
Type, Type + Enter and Quick phrases. `insert`, `enter`, and `quick_chat` remain
|
||||||
|
internal binding keys; visible controls read Type, Type + Enter, Quick phrases.
|
||||||
|
`bindings_frame_controller.json` maps right X click to hold-to-talk PTT;
|
||||||
the grip bindings remain for optional remapping/diagnosis. In one dashboard
|
the grip bindings remain for optional remapping/diagnosis. In one dashboard
|
||||||
probe grips were inactive; a later controls-only probe delivered repeated right
|
probe grips were inactive; a later controls-only probe delivered repeated right
|
||||||
X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions
|
X PTT BeginRecord/EndRecord callbacks. The wearer reports controller actions
|
||||||
@@ -384,7 +442,7 @@ second squeeze **down** within 350 ms starts capture; hold as long as needed
|
|||||||
action explicitly begins on down and ends on up. On tracking-pose invalidity,
|
action explicitly begins on down and ends on up. On tracking-pose invalidity,
|
||||||
action inactivity or overlay focus loss, a held capture emits Cancel, and
|
action inactivity or overlay focus loss, a held capture emits Cancel, and
|
||||||
reconnection requires a neutral observation before any new press. PTT and left
|
reconnection requires a neutral observation before any new press. PTT and left
|
||||||
Enter require an enabled panel; clickable Record remains available for retry
|
Type + Enter require an enabled panel; clickable Record remains available for retry
|
||||||
after an error and Cancel is always available. The action set defaults to normal
|
after an error and Cancel is always available. The action set defaults to normal
|
||||||
priority; the experimental config request is described above. Neither priority
|
priority; the experimental config request is described above. Neither priority
|
||||||
guarantees delivery while a game or dashboard owns input.
|
guarantees delivery while a game or dashboard owns input.
|
||||||
|
|||||||
+89
-26
@@ -11,53 +11,71 @@ build with `FRAMEYAP_NATIVE=ON`, and stage this layout (regular files, no links)
|
|||||||
|
|
||||||
```
|
```
|
||||||
bin/frameyap
|
bin/frameyap
|
||||||
lib/* # compatible bundled native libraries
|
bin/install.sh # managed installer helper when required by layout
|
||||||
assets/actions.json # and adjacent controller binding JSON
|
lib/* # explicitly supplied compatible native libraries
|
||||||
|
assets/actions.json # plus binding JSON and backends/redux.json
|
||||||
fonts/font.ttf
|
fonts/font.ttf
|
||||||
python/frameyap/*.py
|
python/frameyap/*.py
|
||||||
|
scripts/model-status.py # offline CLI verifier; also backend-service.py, fetch-model.py
|
||||||
licenses/THIRD_PARTY_NOTICES.txt
|
licenses/THIRD_PARTY_NOTICES.txt
|
||||||
model/* # optional pinned public weights + attribution
|
model/* # optional pinned public weights + attribution
|
||||||
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
|
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
|
||||||
```
|
```
|
||||||
|
|
||||||
For the current **external-runtime** package, `scripts/stage-native-poc.py --help`
|
The stage copies the self-contained installer into `bin/` and CMake installs
|
||||||
documents explicit inputs. It invokes `cmake --install` on an existing native build,
|
`scripts/model-status.py` alongside the backend manifests. The package allowlist
|
||||||
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
permits `scripts/`; installed `frameyap --list-models` expects the verifier at
|
||||||
|
`../scripts/model-status.py`. **No actual staged archive has been audited/tested
|
||||||
|
from a clean account for publication**; exercise the entire producer pipeline
|
||||||
|
and the installed CLI before treating the payload layout as release-ready.
|
||||||
|
|
||||||
|
For the current **external-runtime** package, `scripts/stage-native.py --help`
|
||||||
|
documents explicit inputs. The old `scripts/stage-native-poc.py` is retained as
|
||||||
|
a deprecated migration wrapper for that command, not the documented or shipped
|
||||||
|
staging interface. The stage invokes `cmake --install` on an existing native
|
||||||
|
build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
||||||
not build, download, run the app, or copy an ASR runtime. The native
|
not build, download, run the app, or copy an ASR runtime. The native
|
||||||
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
||||||
libstdc++ and glibc; audit `ldd` on the installed binary.
|
libstdc++ and glibc. Audit the actual staged ARM64 binaries' `NEEDED`,
|
||||||
|
`GLIBC_*`/`GLIBCXX_*` symbol versions, ELF interpreter and notices; then test
|
||||||
|
on a clean target. No compatible libc floor is yet established.
|
||||||
SDL/OpenVR resolve inside its own `lib/`, not a producer
|
SDL/OpenVR resolve inside its own `lib/`, not a producer
|
||||||
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
|
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \
|
python3 scripts/package-release.py --stage /path/to/stage --output /existing/output \
|
||||||
--version 2026-09-24T162712Z-g417f81c --arch linux-aarch64 \
|
--version 0.1.202609241627 --arch linux-aarch64 \
|
||||||
--model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime
|
--model-revision fad622f25f303105c20d70e201bcc477c88b620c --external-runtime
|
||||||
```
|
```
|
||||||
|
|
||||||
Use the actual binary's UTC build stamp (`frameyap --version`) for the archive
|
Use the actual binary's numeric `MAJOR.MINOR.YYYYMMDDHHMM` UTC version
|
||||||
tag, not this illustrative timestamp. CMake generates `YYYY-MM-DDTHHMMSSZ`
|
(currently `0.1`), not this illustrative value, for `--version` and filename;
|
||||||
at configuration time (plus `-gSHORTSHA` for a Git checkout and `-dirty` for
|
tag a vetted clean tree as `vVERSION`. CMake generates this stamp at
|
||||||
uncommitted tracked changes); producers may pin `-DFRAMEYAP_VERSION=...` to
|
configuration time (`SOURCE_DATE_EPOCH` may supply it);
|
||||||
embed a vetted release stamp. A timestamp distinguishes same-day archives;
|
`-DFRAMEYAP_VERSION=0.1.YYYYMMDDHHMM` can pin it. Development `--version`
|
||||||
the installer still refuses a reused tag whose contents have changed. Historic
|
may print a *separate* `git HASH` line, with `(uncommitted changes)` only if
|
||||||
`v0.1.0-poc*` local artifacts remain valid for reinstall/rollback.
|
dirty: that line is not part of the version, tag or archive name. The installer
|
||||||
|
rejects a reused tag with different contents; historic local versions may
|
||||||
|
remain selectable for rollback, not as new releases.
|
||||||
|
|
||||||
`--external-runtime` refuses a runtime directory and records
|
`--external-runtime` refuses a runtime directory and records
|
||||||
`runtime: external-authorized-python` in `release.json`. The installer explicitly
|
`runtime: external-authorized-python` in `release.json`. The installer explicitly
|
||||||
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
|
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
|
||||||
in the archive. Staging validation is not an inference test.
|
in the archive. Staging validation is not an inference test.
|
||||||
|
|
||||||
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz`
|
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing
|
||||||
plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal,
|
`HASH FILENAME`. Archive extraction rejects traversal,
|
||||||
links/special files, duplicate members, oversized metadata/payloads and invalid
|
links/special files, duplicate members, oversized metadata/payloads and invalid
|
||||||
layout. Checksums detect corruption, not a malicious/compromised publisher;
|
layout. Checksums detect corruption, not a malicious/compromised publisher;
|
||||||
authenticate release metadata independently. No packaging/installation model fetch.
|
authenticate release metadata independently. No packaging/installation model fetch.
|
||||||
|
|
||||||
## Consumer
|
## Consumer
|
||||||
|
|
||||||
Bootstrap: Linux ARM64/glibc, Python 3.12+, curl, sha256sum and tar. **No compiler,
|
Bootstrap for a binary archive: Linux ARM64/glibc, Python 3.12+ (installer
|
||||||
sudo, Steam store AppID or engine checkout.** Download/inspect a pinned installer
|
bootstrap, **not** the inference runtime), curl for network release downloads,
|
||||||
|
sha256sum and tar for archive handling. **No compiler, sudo, Steam store AppID
|
||||||
|
or engine checkout in binary mode.** No minimum glibc floor has been certified;
|
||||||
|
preflight alone cannot guarantee compatibility. Download/inspect a pinned installer
|
||||||
before running it. Current local artifact route:
|
before running it. Current local artifact route:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -65,17 +83,62 @@ sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
|||||||
--sha256 64_HEX_DIGIT_HASH --version VERSION
|
--sha256 64_HEX_DIGIT_HASH --version VERSION
|
||||||
```
|
```
|
||||||
|
|
||||||
After an actual vetted release exists, `sh install.sh --version TAG` retrieves
|
After an actual vetted release exists, use a real numeric version (for example,
|
||||||
that GitHub release and its versioned checksum; no `latest` or moving-branch
|
`sh install.sh --mode binary --version 0.1.202609241627 --yes`); the installer
|
||||||
|
will retrieve tag `v0.1.202609241627` and its versioned checksum; no `latest` or moving-branch
|
||||||
lookup. A pipe invocation is supported, never prompts on stdin, and must also
|
lookup. A pipe invocation is supported, never prompts on stdin, and must also
|
||||||
pin a real published tag. **There is no functional public download command yet.**
|
pin a real published tag. **There is no functional public download command yet.**
|
||||||
|
|
||||||
`--without-model` omits bundled model files from staging, never deletes a current
|
`--without-model` omits any model files in the selected archive, never deletes
|
||||||
model on rerun, and records the choice. Same digest/version/choice is idempotent
|
an existing current model on rerun, and records the choice. Same
|
||||||
and repairs missing managed wrappers. Different digest or model choice for the
|
digest/version/choice is idempotent and repairs missing managed wrappers. Different digest or model choice for the
|
||||||
same version is refused. External model provisioning is always deliberate.
|
same version is refused. External model provisioning is always deliberate: the
|
||||||
|
local installer accepts `sh install.sh --install-model --backend redux --yes`
|
||||||
|
(optional `--model-dir /absolute/path`) to explicitly download and verify files from the
|
||||||
|
installed pinned manifest, not an ASR runtime. Inspect the model/size first
|
||||||
|
with `sh install.sh --install-model --backend redux --print-plan --json`.
|
||||||
|
`--expected-manifest-sha256 HASH` additionally binds consent to the exact raw
|
||||||
|
installed `redux.json` bytes: under the model lock a mismatch fails **before**
|
||||||
|
a model directory is created or any network request. In the locally implemented
|
||||||
|
Models UI, Install displays source, rounded size, license text, attribution and
|
||||||
|
the fingerprint; only the second Confirm Install click passes that fingerprint
|
||||||
|
through the backend helper to the installer. Installation alone does not
|
||||||
|
provision a CPU Python runtime; no in-panel flow has been accepted on Frame.
|
||||||
|
In the source tree, `python3 scripts/model-status.py --list-models` or
|
||||||
|
`--check-model redux --model-dir /absolute/model` hash-checks local files;
|
||||||
|
the native `frameyap --list-models` / `--check-model` entry points use the
|
||||||
|
adjacent installed verifier script; verify this in a staged archive before release. Manifest schema,
|
||||||
|
source, size/hash and attribution live in `assets/backends/redux.json` and are
|
||||||
|
validated by `python/frameyap/model_files.py`.
|
||||||
|
|
||||||
The installed launcher defaults to `--run`. For a native-only package, supply
|
For an explicit local **source** install, e.g.:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sh install.sh --mode source --source /absolute/source --openvr-root /absolute/sdk \
|
||||||
|
--openvr-library /absolute/libopenvr_api.so --openvr-license /absolute/openvr/LICENSE \
|
||||||
|
--sdl-library /absolute/libSDL3.so.0 --sdl-license /absolute/sdl/LICENSE \
|
||||||
|
--version 0.1.202609241627
|
||||||
|
```
|
||||||
|
|
||||||
|
It checks tools and native dependencies, builds/stages/packages in a private
|
||||||
|
workspace, then installs the result. Unlike binary mode, this requires a C++
|
||||||
|
compiler, CMake, SDK, SDL3, Wayland/scanner, libxcb, FreeType, Vulkan development
|
||||||
|
files and producer-supplied
|
||||||
|
licenses; it still does not install Python ASR packages. `--print-plan` performs
|
||||||
|
a read-only plan, `--json` gives machine-readable results/errors (model installs
|
||||||
|
also stream file events), and `--yes` authorizes network downloads. Bare TTY
|
||||||
|
invocation can guide choices and prints equivalent flags; non-TTY runs require
|
||||||
|
explicit arguments and never prompt. `--autolaunch`/`--no-autolaunch` are explicit
|
||||||
|
OpenVR registration choices, off by default; do not pass either during an inert
|
||||||
|
install if a running SteamVR session must remain untouched.
|
||||||
|
|
||||||
|
The installed launcher defaults to `--run` and forwards explicit run flags
|
||||||
|
(including `--backend ID`, `--model-store /absolute/store` and
|
||||||
|
`--manifest-dir /absolute/manifests`) to the binary. These overrides select
|
||||||
|
local metadata/model paths, not a runtime download; the latter two require
|
||||||
|
absolute paths without dot segments. `--backend` selects for that run unless
|
||||||
|
changed in the panel; a saved `config.json` backend is otherwise used. For a
|
||||||
|
native-only package, supply
|
||||||
`FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`,
|
`FRAMEYAP_PYTHON=/absolute/authorized/python` and `FRAMEYAP_MODEL=/absolute/model`,
|
||||||
or override `--python`/`--model` on an explicit `--run`. For menu launches, create
|
or override `--python`/`--model` on an explicit `--run`. For menu launches, create
|
||||||
`$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`):
|
`$XDG_CONFIG_HOME/frameyap/paths.conf` (default `~/.config/frameyap/paths.conf`):
|
||||||
@@ -157,7 +220,7 @@ without configured model/runtime cannot transcribe; it should show **Unavailable
|
|||||||
(possibly after a brief Warming transition), rather than record or infer.
|
(possibly after a brief Warming transition), rather than record or infer.
|
||||||
Only perform this check on an unconfigured native-only installation: verify
|
Only perform this check on an unconfigured native-only installation: verify
|
||||||
that `current/runtime/bin/python3` and `current/model` are absent and no
|
that `current/runtime/bin/python3` and `current/model` are absent and no
|
||||||
user-local paths are configured; do not press Record, Insert or Enter.
|
user-local paths are configured; do not press Record, Type or Type + Enter.
|
||||||
The installer also provides a desktop entry; where Steam's UI supports adding a
|
The installer also provides a desktop entry; where Steam's UI supports adding a
|
||||||
non-Steam app, the user may select that entry or browse to the installed launcher.
|
non-Steam app, the user may select that entry or browse to the installed launcher.
|
||||||
Shortcut discovery/persistence after a normal restart is not yet verified. Do
|
Shortcut discovery/persistence after a normal restart is not yet verified. Do
|
||||||
|
|||||||
+82
-74
@@ -1,91 +1,99 @@
|
|||||||
# Dependency provenance and release boundary
|
# Dependency provenance and release boundary
|
||||||
|
|
||||||
FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project
|
FrameYap's original code is [MIT licensed](../LICENSE). This does not relicense
|
||||||
owner. This does not relicense external models, fonts, protocols or runtimes.
|
models, fonts, protocols, native libraries or Python wheels. There is no dependency
|
||||||
There is no dependency on another application's checkout, assets or environment.
|
on another application's checkout, assets or environment. **This is an upstream
|
||||||
|
license inventory, not an audit of a particular release binary. No public release
|
||||||
|
has been published.** Before shipping any archive, inspect the actual staged
|
||||||
|
files, their transitive dependencies and notices, and test on a clean supported
|
||||||
|
host. An external dependency does not make the current archive self-contained.
|
||||||
|
|
||||||
## Included source
|
## Included source/assets
|
||||||
|
|
||||||
`protocol/gamescope-input-method.xml` is the unmodified public Gamescope
|
- `protocol/gamescope-input-method.xml`: unmodified public Gamescope **3.16.28**
|
||||||
**3.16.28** protocol, downloaded from
|
protocol, <https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
|
||||||
<https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
|
SHA-256 `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`;
|
||||||
SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`.
|
embedded permissive copyright/license notice preserved. Generated bindings are
|
||||||
Its embedded permissive copyright/license notice is preserved. Generated bindings
|
build outputs; this private Gamescope extension needs rechecking after updates.
|
||||||
are build outputs, not hand-written wire encoding. The private protocol may change
|
- `assets/fonts/Inconsolata-Regular.ttf`: unchanged Inconsolata Regular from
|
||||||
with SteamOS; compatibility must be rechecked.
|
<https://github.com/googlefonts/Inconsolata>, copyright 2006 The Inconsolata
|
||||||
|
Project Authors, **SIL OFL 1.1**. SHA-256
|
||||||
|
`e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
|
||||||
|
`assets/fonts/OFL-Inconsolata.txt` retains the license. The font remains OFL,
|
||||||
|
not MIT, and is not sold by itself. An override font requires its own license.
|
||||||
|
- Frame controller bindings and the mint-to-blue CPU-rasterized panel were
|
||||||
|
authored here, using public profile names; no SteamVR driver artwork, MSDF
|
||||||
|
atlas, other app renderer or protected kernels were copied.
|
||||||
|
|
||||||
Frame controller bindings were authored here using the observed public input
|
Native staging uses `scripts/stage-native.py`: it includes the chosen font/license
|
||||||
profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code,
|
and copied SDL3 and OpenVR notices in `licenses/THIRD_PARTY_NOTICES.txt`.
|
||||||
images, protected kernels or another application's assets were extracted for these bindings.
|
It does **not** bundle ASR. A4 is not closed for release: inspect the final
|
||||||
|
notice file, especially FreeType attribution, plus the selected native binary
|
||||||
|
and any bundled wheel/licenses before publishing.
|
||||||
|
|
||||||
## Bundled font and UI reference
|
## Native build/runtime inventory
|
||||||
|
|
||||||
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular.
|
| Component | Upstream license / evidence | Current packaging boundary / action |
|
||||||
SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
|
| --- | --- | --- |
|
||||||
Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**,
|
| Valve OpenVR SDK 2.15.6 | BSD-3-Clause-style license, SDK LICENSE. | Staging copies its loader and explicitly supplied license; verify chosen binary and transitive closure. |
|
||||||
retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace
|
| SDL3 (device trial 3.2.16) | zlib, local `/usr/share/licenses/sdl3/LICENSE`. | Staging copies explicit library and license. Confirm exact build options/version and its transitive libraries. |
|
||||||
normalized; license text unchanged). The reviewed OFL permits
|
| Wayland client + scanner | MIT/Expat-style, local `/usr/share/licenses/wayland/COPYING`. | Client is linked from system; scanner is build-time. If shipped, include copyright/license and audit closure. |
|
||||||
bundling and redistribution with its copyright/license notice; the font remains
|
| libxcb | MIT-style with name-use restriction, local `/usr/share/licenses/libxcb/COPYING`. | Xwayland focus guard uses client library at runtime; not bundled by current native stage. Audit exact binary. |
|
||||||
OFL, not MIT, and is not sold by itself. Upstream: <https://github.com/googlefonts/Inconsolata>.
|
| FreeType 2 | **FreeType Project License (FTL) selected** for this project, local `/usr/share/licenses/freetype2/FTL.TXT`; upstream also offers a GPL option. | Current stage uses system library, not bundled. Credit FreeType Team for use; if distributing its binary, meet FTL binary disclaimer/notice obligations and review the precise build. Do not silently substitute GPL terms. |
|
||||||
The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies
|
| Vulkan loader, driver, system graphics dependencies | Loader/driver licenses vary by build and vendor. | Current stage depends on system Vulkan loader/driver; no GPU runtime is bundled. Audit the chosen loader if ever bundled. |
|
||||||
were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph.
|
| Compiler runtime (`libstdc++`, `libgcc_s` when used) | GCC libraries: GPL with **GCC Runtime Library Exception** in upstream distribution; local `/usr/share/licenses/libstdc++/RUNTIME.LIBRARY.EXCEPTION` and `libgcc/...` are exception texts, not a full installed release audit. | Current stage relies on system runtime. Audit dynamic linkage, C++ ABI/`GLIBCXX_*` and exception coverage for *any* bundled compiler libraries; include corresponding complete notices/source obligations as applicable. |
|
||||||
|
| glibc/loader | GNU LGPL-2.1-or-later for core GNU C Library, with component-specific exceptions and other licenses to inspect. | Current stage relies on system libc/loader. Audit exact target binary symbol versions (`GLIBC_*`), ELF interpreter and its transitive closure; no minimum glibc/`GLIBCXX`/kernel floor is certified here. |
|
||||||
|
|
||||||
The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted
|
The historical Frame CPU trial used Torch **2.8.0+cpu** on a host with **glibc
|
||||||
selection) is implemented independently on a single CPU RGBA surface.
|
2.39**. Those are *observed trial versions*, **not** minimum compatible versions
|
||||||
CMake installs the font and OFL with assets; native staging defaults to that
|
for FrameYap, Python wheels or a future released artifact. `ldd` on a developer
|
||||||
font, places the launcher copy at `fonts/font.ttf`, and includes its license in
|
machine alone is not sufficient: inspect the staged ARM64 binaries with `readelf`
|
||||||
`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license.
|
(`NEEDED`, ELF interpreter, symbol-version requirements), `ldd` on a trusted
|
||||||
|
clean target, actual bundled libraries and notices, and test the final archive on
|
||||||
## Explicit native build inputs (not vendored)
|
a clean supported Frame. Check the chosen compiler, CPU instruction/kernel,
|
||||||
|
FreeType/Wayland/XCB/SDL/OpenVR/Vulkan ABI, Python/native wheels and licenses.
|
||||||
- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015;
|
Do not invent a libc floor from a build host's version.
|
||||||
retain its LICENSE with redistributed loader binaries.
|
|
||||||
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
|
|
||||||
- Wayland client and scanner: retain upstream MIT-style notices.
|
|
||||||
- libxcb (X11 protocol client): MIT-style license; used only by the opt-in
|
|
||||||
Xwayland focus observer. Include it in native runtime dependency checks;
|
|
||||||
no X server is started by normal operation.
|
|
||||||
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
|
|
||||||
- Optional font override: the earlier device check used system Hack Regular.
|
|
||||||
A custom release font must include its own license and assessed glyph coverage;
|
|
||||||
new builds default to the bundled Inconsolata described above.
|
|
||||||
- Compiler runtime, libc minimum and transitive shared libraries require a release
|
|
||||||
dependency audit. Passing a developer build is not a portable-runtime guarantee.
|
|
||||||
|
|
||||||
## Redux weights and inference runtime
|
## Redux weights and inference runtime
|
||||||
|
|
||||||
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
Model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
||||||
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution:
|
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**.
|
||||||
Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3.
|
Attribution: Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT
|
||||||
No modifications to the supplied weights are made. Exact model/config/tokenizer
|
0.6B v3. No modifications to the supplied weights are made. Pinned model/config/
|
||||||
sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`.
|
tokenizer/card sizes and SHA-256 hashes, source and attribution are recorded in
|
||||||
`fetch-model.py` fetches and retains the original model card alongside the files.
|
`assets/backends/redux.json` (schema validated by `python/frameyap/model_files.py`),
|
||||||
No weights are committed to this repository.
|
not hardcoded in `model_files.py`. `fetch-model.py` fetches and retains the model
|
||||||
|
card alongside the weights on **explicit** request; neither build/tests nor a
|
||||||
|
normal app launch downloads them. `scripts/model-status.py` and the native
|
||||||
|
`--list-models` / `--check-model` interface inspect/hashes local files offline.
|
||||||
|
No weights are committed to Git.
|
||||||
|
|
||||||
Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels`
|
The current Redux worker uses separately provisioned `moondream` Python and its
|
||||||
dependencies, installed by the user from PyPI into their own environment. Upstream's
|
`kestrel` / `kestrel-kernels` dependencies, not a bundled runtime. Kestrel's
|
||||||
Kestrel README states: "Local inference is free and requires no API key"
|
upstream README says “Local inference is free and requires no API key”
|
||||||
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs a Moondream
|
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs an API
|
||||||
API key and is not used here. FrameYap's release archives do not vendor or bundle
|
key and is **not** this path. The native-only installer **does not run pip**,
|
||||||
these packages; they are installed from PyPI onto the user's machine, either by the
|
provision an interpreter, or make a native-only artifact able to transcribe on
|
||||||
user or by the installer at the user's request.
|
its own. A person supplying a Python environment must review/authorize its
|
||||||
|
exact dependency closure. No bundled-ASR artifact is licensed/approved by this
|
||||||
|
inventory.
|
||||||
|
|
||||||
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need
|
| Python/native package | Upstream license inventory (not a wheel audit) | Release action |
|
||||||
their own notice/license inventory before a bundled release.
|
| --- | --- | --- |
|
||||||
|
| PyTorch / Torch CPU | PyTorch project: BSD-3-Clause; third-party components/wheels carry additional notices and dependencies. | No Torch wheels bundled. Pin CPU-only ARM64 wheel if building a distribution; audit its `LICENSE`, `NOTICE`, `third_party`/wheel contents and `NEEDED`/symbol versions. |
|
||||||
|
| NumPy | NumPy core: BSD-3-Clause; dependencies/embedded algorithms carry additional BSD, MIT, 0BSD, zlib, CC0 and other notices depending on wheel. Local `python-numpy` 2.5.3 package metadata (`/usr/lib/python3.14/site-packages/numpy-2.5.3.dist-info/METADATA`) declares `BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0` with many `License-File` entries (different from the historical trial environment). | Not bundled. Keep *all* license files and inspect the exact target wheel, BLAS/OpenBLAS and runtime closure before redistribution. |
|
||||||
|
| Hugging Face `tokenizers` | Upstream `huggingface/tokenizers` is Apache-2.0; native/Rust crate dependencies need separate inventory. | Not bundled. Confirm actual installed wheel version, package LICENSE/NOTICE and transitive Rust/native code if ever distributed. |
|
||||||
|
| `moondream`, Kestrel/kernels/native, Python interpreter | Distinct packages with distinct license files and native transitive code; Kestrel local-use statement is not a blanket redistribution license. | None bundled. Exact versions, permissions, wheel notices, CPython build and native linkage require review before any runtime bundle. |
|
||||||
|
|
||||||
## CPU trial dependency choice
|
Development CPU trial *interfaces*, not package-floor promises: moondream
|
||||||
|
**2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, native **0.1.8**, Python
|
||||||
|
**3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified moondream install
|
||||||
|
initially resolved CUDA-enabled Torch and NVIDIA wheels; the owned trial venv
|
||||||
|
was corrected before measurement (`torch.version.cuda is None`). Do **not**
|
||||||
|
repeat unconstrained `pip install moondream` as a CPU setup recipe.
|
||||||
|
|
||||||
Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**,
|
A standalone CPython 3.12.14 ARM64 distribution was downloaded for packaging
|
||||||
native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified
|
research but **not bundled**: <https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
||||||
moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels;
|
|
||||||
these were replaced/removed from the owned venv before measurement. The measured
|
|
||||||
Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
|
|
||||||
`pip install moondream` as a CPU setup recipe.
|
|
||||||
|
|
||||||
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
|
|
||||||
was downloaded but not bundled with any ASR runtime:
|
|
||||||
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
|
||||||
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
|
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
|
||||||
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
|
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
|
||||||
Its included licenses and compatible native dependencies still require review
|
Its own licenses and native compatibility need review before any release.
|
||||||
before any release. No public prebuilt FrameYap release has been published.
|
|
||||||
+38
-15
@@ -1,8 +1,10 @@
|
|||||||
# Offline Redux worker adapter (component, not an installed product)
|
# Offline worker adapter and backend dispatcher
|
||||||
|
|
||||||
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
|
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
|
||||||
threads=2, advanced_debug=false)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for
|
threads=2, advanced_debug=false)` explicitly for the legacy Redux script, or
|
||||||
one `WorkerReply` (text or privacy-safe per-request error). One request at a time;
|
supply the optional `backend`, `manifest_dir`, `root` arguments for the manifest
|
||||||
|
dispatcher. Poll until `ready()`, then `submit(id, pcm)` and poll for one
|
||||||
|
`WorkerReply` (text or privacy-safe per-request error). One request at a time;
|
||||||
no queue, no capture and no input injection. `stop()` discards pending audio,
|
no queue, no capture and no input injection. `stop()` discards pending audio,
|
||||||
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
|
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
|
||||||
and is safe to repeat. Destruction stops it. `start()` returns without waiting
|
and is safe to repeat. Destruction stops it. `start()` returns without waiting
|
||||||
@@ -36,13 +38,19 @@ requests reading the fixed clip; `Y` means ready; `F` means load failure
|
|||||||
missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8
|
missing Python dependency, `D` for runtime/model load failure); `R` + ID + UTF-8
|
||||||
text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096
|
text and `E` + ID + privacy-safe UTF-8 error are replies. Text is at most 4096
|
||||||
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
|
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
|
||||||
or oversized frame stops the worker. Warmup deadline is 120 s, transcription
|
or oversized frame stops the worker. A correlated `E` reply or an invalid
|
||||||
deadline 60 s; `poll()` must be called regularly to enforce deadlines. It
|
transcript is a **request-level** failure: the Controller drops that clip,
|
||||||
never initializes a headset or starts a recording. There is no auto restart.
|
keeps the ready model loaded and allows Record to retry. A broken protocol,
|
||||||
|
worker crash or explicit in-flight cancellation is different and can require a
|
||||||
|
reload. Warmup deadline is 120 s, transcription deadline 60 s; `poll()` must
|
||||||
|
be called regularly to enforce deadlines. It never initializes a headset or
|
||||||
|
starts a recording. There is no auto restart after a process failure.
|
||||||
|
|
||||||
`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI
|
Parakeet Redux is the **model** (`moondream/parakeet-redux`); `moondream` is
|
||||||
startup, with HF/Transformers/Datasets offline variables and bounded native
|
its Python inference package, not a second model or cloud endpoint. Kestrel is
|
||||||
thread-pool variables set before import. It uses
|
a native dependency of that local runtime. `python/frameyap/worker.py` lazily
|
||||||
|
imports `moondream` only after explicit CLI startup, with HF/Transformers/Datasets
|
||||||
|
offline variables and bounded native thread-pool variables set before import. It uses
|
||||||
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
|
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
|
||||||
device="cpu", cpu_threads=threads)` and persistent
|
device="cpu", cpu_threads=threads)` and persistent
|
||||||
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
|
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
|
||||||
@@ -51,16 +59,31 @@ moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
|
|||||||
preinstalled local weights from revision
|
preinstalled local weights from revision
|
||||||
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
|
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
|
||||||
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
|
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
|
||||||
against `model_files.py` before importing model libraries. Protocol stdout is
|
against `assets/backends/redux.json` via the shared, offline
|
||||||
isolated at the file-descriptor level from third-party diagnostics. Thread limits
|
`python/frameyap/model_files.py` schema/verifier before importing model libraries.
|
||||||
cover Torch interop/native pools and CUDA is not selected. Offline environment
|
`frameyap --list-models` and `frameyap --check-model redux --model-dir /absolute/model`
|
||||||
|
(or `scripts/model-status.py`) read local manifests and optionally hash local
|
||||||
|
files; they never load the worker or download weights. For native `--run`,
|
||||||
|
`--backend ID`, `--model-store /absolute/store`, and
|
||||||
|
`--manifest-dir /absolute/manifests` are wired through the installed launcher
|
||||||
|
as explicit overrides; default selection is saved in `config.json` (`redux`). The dispatcher
|
||||||
|
`python/frameyap/backend_worker.py` validates a selected manifest and hashes
|
||||||
|
its local model, resolves an in-release relative Python/executable launcher
|
||||||
|
without a shell, then supervises one child over `frameyap-worker-v1` (ready,
|
||||||
|
correlated request/reply and failure frames). The launcher must accept
|
||||||
|
`{model_dir}` and `{clip_dir}` (optional `{threads}`); matching manifest and
|
||||||
|
pinned weights are **not** a runtime or automatically trusted new backend.
|
||||||
|
Only Redux is supplied today. Protocol stdout is isolated at the file-descriptor
|
||||||
|
level from third-party diagnostics. Thread limits cover Torch interop/native
|
||||||
|
pools and CUDA is not selected. Offline environment
|
||||||
flags do not prove every third-party internal is unable to access a network.
|
flags do not prove every third-party internal is unable to access a network.
|
||||||
Runtime/build/tests perform no downloads; the separate explicit setup utility
|
Runtime/build/tests perform no downloads; the separate explicit setup utility
|
||||||
`scripts/fetch-model.py` can provision the public pinned weights.
|
`scripts/fetch-model.py` can provision the public pinned weights.
|
||||||
|
|
||||||
Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public
|
Current native-only archives do not bundle or pip-install the runtime; see
|
||||||
runtime bundle has been released. Limited ARM64 measurements were taken during development;
|
[third-party notes](third-party.md). No public runtime bundle has been released.
|
||||||
they are not a claim of complete headset acceptance.
|
Limited ARM64 measurements were taken during development; they are not a claim
|
||||||
|
of complete headset acceptance.
|
||||||
|
|
||||||
## Advanced debugging
|
## Advanced debugging
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user