diff --git a/README.md b/README.md index 178fa27..b3b705f 100644 --- a/README.md +++ b/README.md @@ -1,53 +1,84 @@ -# Frame Dictation +# FrameYap -Standalone, on-device voice typing for Steam Frame. +Standalone, on-device voice typing POC for Steam Frame. **MIT licensed.** -**Status: project scaffold and design only.** The executable prints help/version; -it does not yet render an overlay, open a microphone, load a model or type text. +Implemented: native OpenVR overlay, remappable grip controls, bounded SDL3 capture, +persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope, +and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR +server, cloud fallback or unrelated application dependency. -Planned path: **OpenVR overlay → local Parakeet Redux CPU inference → Gamescope -Unicode input**. No external application checkout, library, submodule, desktop -inference server or running scene host is required. +**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility, +Gamescope discovery and native-only installation have been exercised on Frame. +Live microphone → reviewed text → real target delivery is **not yet accepted**. -**Distribution goal:** one-command installation from GitHub releases, entirely -user-local, with no Steam store AppID. See the [installer design](docs/install-design.md). -There is no working installer or published release yet. +**Runtime licensing remains unresolved.** Redux weights are CC-BY-4.0, but the +observed Kestrel kernel runtime requires separate permission. We are checking with +the vendor. Current native-only packages do **not** bundle or download that +runtime; an independently authorized Python environment must be supplied. +See [third-party notes](docs/third-party.md). No GitHub release is published yet. -For a read-only check of proposed release prerequisites on a Frame, run -`sh scripts/install-preflight.sh`. It checks Linux AArch64/glibc and bootstrap -tools, reports system Python (3.12+ for a possible source worker), Git and uv. -Git and uv are not required for the planned bundled release. This check downloads -and installs nothing; passing it does **not** mean an install or dictation works. +## Controls -## Build the scaffold +- **Right grip:** short tap, then hold the second squeeze to record; release to + transcribe. Remappable through SteamVR bindings; separate hold-to-talk action too. +- **Left grip:** double-tap to explicitly send Enter. Never inferred from speech. +- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Enter and Quit. + Dashboard lasers provide clickable controls without forcing global laser mode. +- **Review-first:** focus your destination, then press Insert. No automatic insertion + or submission. Maximum clip 20 seconds; accidental taps under 200 ms are discarded. +- Other apps may still hear/transmit your voice. FrameYap does not mute them. -Requirements: CMake 3.20+ and a C++20 compiler. No third-party packages or downloads. +Physical gesture timing, global bindings during games, mic capture, target-app +compatibility and headset comfort still need coordinated validation. Successful +API initialization is not delivered input or human acceptance. + +## Build and test offline + +CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests. ```sh cmake -S . -B build cmake --build build ctest --test-dir build --output-on-failure -./build/frame-dictation --help +./build/frameyap --help ``` -This is a host-native scaffold build, not an ARM64 deployment or headset test. -Future OpenVR, audio and inference integrations must be explicitly configured; -configuration/build/tests must never install packages or launch SteamVR implicitly. +Default build has no hardware backends. It never downloads packages/models or +initializes SteamVR, a microphone or input injection. Native dependencies and +explicit launch/check commands are documented in [the POC guide](docs/poc.md). + +## Installation + +The real installer accepts a versioned, checksummed prebuilt ARM64 archive: + +```sh +sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \ + --sha256 ARCHIVE_SHA256 --version VERSION +``` + +This is the **local-artifact command shape**, not an available public download. +Installation is user-local, retains rollback, refuses active-app upgrades and +foreign files, and does not launch or register automatically. Registration uses +OpenVR application key `local.frameyap.overlay`, not a Steam store AppID. +Autolaunch is opt-in. See [packaging and lifecycle](docs/packaging.md). + +A pinned GitHub one-command route is implemented in `install.sh --version TAG`, +but **do not advertise or run it as a working public installation until a vetted +release exists**. Native-only artifacts support the overlay/checks without an +end-user compiler; full bundled-ASR distribution awaits runtime permission. ## Project map -- [Design](docs/design.md): UI, local inference, input backend and acceptance gates. -- [Frame API evidence](docs/evidence/frame-dictation-apis-2026-09-24.md): successful - mixed-script Unicode test and overlay-client initialization; known limits. -- [Installer design](docs/install-design.md): GitHub install, standalone OpenVR - identity, packaging, upgrades and uninstall. -- [Provenance](docs/provenance.md): origin of the investigation, model/runtime pins. -- `src/main.cpp`: inert CLI entry point. -- `tests/cli.cmake`: hardware-free scaffold smoke test. -- `scripts/install-preflight.sh`: read-only proposed packaging prerequisite check. -- [Agent guidance](AGENTS.md): project boundaries and safe validation. +- [POC guide](docs/poc.md): implemented boundaries, build, controls, explicit tests. +- [Design](docs/design.md): full target design; some features remain proposed. +- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md). +- [Current POC observations](docs/evidence/poc-cpu-overlay-2026-09-24.md): measured + CPU behavior and native installation checks, with acceptance limits. +- [Earlier API evidence](docs/evidence/frameyap-apis-2026-09-24.md) and + [provenance](docs/provenance.md): historical investigation, not live authority. +- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md), + [dependency/license inventory](docs/third-party.md). -First implementation gate: an isolated ARM64 CPU trial of the pinned Redux model. -Then a minimal overlay and explicit insertion into a disposable target. Neither -inference nor an overlay is implemented here yet. No model weights, recordings, -credentials, engine assets or runtime binaries are included. +No recordings, transcripts, private logs, model weights or runtime binaries are +committed. The worker boundary is intentionally small for forks experimenting +with other models/APIs; the default remains local-only Redux. diff --git a/docs/design.md b/docs/design.md index 41e2e2d..e0bccf0 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,8 +1,13 @@ -# Native Frame dictation overlay — proposal +# FrameYap native dictation overlay — proposal Standalone project design; see -[provenance](provenance.md) and [historical Frame probes](evidence/frame-dictation-apis-2026-09-24.md). -Only the inert build scaffold is implemented. This document describes proposed runtime behavior. +[provenance](provenance.md) and [historical Frame probes](evidence/frameyap-apis-2026-09-24.md). +This document is the full target design, not a blanket implementation claim. +The native POC now implements overlay/actions, bounded SDL3 capture, a persistent +Redux adapter, review-first Gamescope insertion and a user-local archive installer. +Quick typing/focus-generation tracking, polished status-chip UX and full hardware +acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md) +and [runtime licensing boundary](third-party.md). ## Recommendation @@ -52,7 +57,10 @@ Recording… 00:04 [ Cancel ] - States: disabled, warming, ready, recording, transcribing, review, inserted, unavailable/error. Recording uses visible icon + text, not color alone. -- Hold to speak, release to finish. A click-to-start/stop overlay button provides +- Default POC binding: tap right grip briefly, then hold the second squeeze to + speak; release to finish. Double-tap left grip is a separate explicit Enter. + Both are remappable, with a separate named hold-to-talk action available. + A click-to-start/stop overlay button provides a binding-independent alternative. Bound recording to 20 seconds; discard accidental taps (initial threshold: 200 ms). - **Quick typing:** insert on completion only when the explicitly armed target @@ -67,7 +75,7 @@ Recording… 00:04 [ Cancel ] ## Host and rendering boundary -Implement the standalone `frame-dictation` executable, initialized +Implement the standalone `frameyap` executable, initialized with `VRApplication_Overlay`. Frame accepted that application type and `IVROverlay_028` in the probe. Use `CreateOverlay`, tracked-device-relative transform, `ShowOverlay`/`HideOverlay`, and `PollNextOverlayEvent` for the panel. @@ -213,7 +221,7 @@ Suggested ownership, introduced only as implementation needs it: - `src/audio.*`: explicit SDL capture and bounded mono PCM. - `src/worker.*`: one local child process, bounded requests/replies, timeout/reaping. - `src/text_input.*`: Gamescope protocol, focus observation and delivery policy. -- `python/frame_dictation/`: persistent CPU Redux worker, no external application imports. +- `python/frameyap/`: persistent CPU Redux worker, no external application imports. The worker reads a fixed private clip and returns an ID-correlated literal string; one request at a time, 64 KiB framed messages, 4096-byte transcript, and a bounded @@ -280,10 +288,11 @@ without moving recognition off Frame. It is not part of this initial design. ## Standalone dependency and test policy -The scaffold currently needs only CMake and a C++20 compiler. Later integrations -will explicitly select OpenVR, SDL3, Wayland client/protocol bindings and a Python -Redux environment. Pin revisions and review licenses when introduced. No automatic -fetch/install in configure or normal tests; no external checkout discovery. +The default hardware-free build needs only CMake and a C++20 compiler (Python +runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR, +SDL3, FreeType and Wayland client/generated protocol bindings. A separately +authorized Python Redux environment is explicitly supplied at launch. Pin revisions +and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery. Hardware-free tests should cover state transitions, bounded PCM/transcripts, worker framing/timeout/cancellation, duplicate/stale replies and focus generations diff --git a/docs/evidence/frame-dictation-apis-2026-09-24.md b/docs/evidence/frameyap-apis-2026-09-24.md similarity index 100% rename from docs/evidence/frame-dictation-apis-2026-09-24.md rename to docs/evidence/frameyap-apis-2026-09-24.md diff --git a/docs/evidence/poc-cpu-overlay-2026-09-24.md b/docs/evidence/poc-cpu-overlay-2026-09-24.md new file mode 100644 index 0000000..1290c94 --- /dev/null +++ b/docs/evidence/poc-cpu-overlay-2026-09-24.md @@ -0,0 +1,103 @@ +# FrameYap POC observations — 2026-09-24 + +Dated observations on one user-authorized Steam Frame, not proof of future device +availability, full dictation acceptance or runtime redistribution rights. No audio, +weights, private transcript/log files, credentials or runtime binaries are included. +The final native package below was built from source commit **`d60d8d2`**. + +## CPU compatibility trial (before runtime-license review) + +Isolated user-local environment: ARM64, glibc 2.39, Python 3.12.3, +moondream 2.4.0 / kestrel 0.8.0 / kernels 0.7.0 / native 0.1.8, +Torch **2.8.0+cpu** (`torch.version.cuda is None`). Model weights/config/tokenizer +verified against pinned Redux revision `fad622f25f303105c20d70e201bcc477c88b620c`. +No dense model substitution or desktop/cloud inference was used. + +A one-second synthetic silence trial loaded in 5.645 s; three decodes took +0.187 / 0.149 / 0.167 s and returned no text. Peak process RSS in that trial was +989,844 KiB (~967 MiB). This is not peak RSS for arbitrary 20-second speech clips. + +Public 11.0-second JFK speech fixture: +. +Source FLAC SHA-256 `63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715`. +Converted to PCM16/mono/16 kHz WAV with ffmpeg; WAV SHA-256 +`0c397b12d7dbdd89e4fd0d9a0840c14a2c5c3707758562755cda96a931b25a91`. +The persistent worker used bounded file/pipe IPC; no microphone was opened. + +| CPU threads | Worker load | Five request times (seconds) | Median | RTF | Audio/compute ratio | +| --- | --- | --- | --- | --- | --- | +| 2 | 5.190 s | 1.550, 1.619, 1.587, 1.619, 1.659 | 1.619 s | 0.147 | 6.8× realtime | +| 4 | 4.655 s | 1.196, 1.158, 1.143, 1.249, 1.502 | 1.196 s | 0.109 | 9.2× realtime | + +The first result reproduced the fixture's known sentence; all five replies in +each run were correlated and 108 UTF-8 bytes. These are medians/maxima from one +public clip, not representative accuracy/WER, p95, streaming realtime performance, +or compositor/thermal acceptance. Load time is excluded from RTF. Two threads +remain the default pending scene-contention measurements. + +An unconstrained dependency install initially selected CUDA Torch/NVIDIA wheels; +these were replaced/removed from the owned venv before measurement. No system +packages were changed. Later inspection found the proprietary kernel license's +separate-agreement requirement. Permission was declared unresolved; further +inference and proprietary-runtime packaging were paused pending vendor clarification. +See [license boundary](../third-party.md). The acquired development environment +remains user-local and is not included in the native-only installation. + +## Native runtime / controller / overlay checks + +- Native CMake build succeeded on Frame using standalone OpenVR SDK v2.15.6, + explicitly built SDL 3.2.16, system Wayland and FreeType. No unrelated app tree + or environment was linked/copied. No sudo or system/session restart. +- Installed executable connected to `gamescope-0`, bound the IME and received + ready/done state; each discovery check disconnected **without text or Enter**. +- OpenVR accepted the raw panel. The user reported seeing the five-second window. + This confirms visibility, not readability/comfort or successful pointer clicks. +- Device controller properties identified `frame_controller` and its input profile. + The profile exposes grip `click`; authored default bindings were added. A + controls-only run reported both grip actions tracked/active. **No physical + gesture delivery was observed in that run.** Double-tap/hold/loss behavior is + currently verified by deterministic unit tests, not a human controller trial. +- The ARM loader initially tried `/data/work/openvrpaths.vrpath`. Selecting the + existing standard user registry fixed initialization; code now does so only + when no explicit override is supplied. +- Registration initially returned API success but did not install the app. + Frame's manifest parser requires **`binary_path_linux_arm`**. Adding it fixed + registration. Code now verifies `IsApplicationInstalled`, not return status alone. +- Repeated Add, Remove and re-Add of **only** `local.frameyap.overlay` succeeded; + autolaunch queried **off**. The final installed panel check also succeeded after + explicit identification with that registered key. Menu-driven launch and cold + SteamVR startup were not exercised. + +## Installer and final state + +Native-only artifacts `v0.1.0-poc1`/`poc2` exercised installation, same-version +rerun, upgrade, rollback and forward selection. A producer-prefix SDL RUNPATH leak +was found and removed; final ELF RUNPATH is only `$ORIGIN/../lib` and bundled +SDL/OpenVR resolve inside the installed tree. + +Explicit OpenVR unregister followed by `--uninstall --unregistered` succeeded. +The final `v0.1.0-poc3` artifact was then installed twice and registered with +autolaunch off. It includes native code, SDL/OpenVR, Hack font/notices and the +worker adapter, **no ASR runtime or weights**. Installed size: ~8.1 MiB. + +Local archive SHA-256: +`90aab107feea2cb501bf815806bdc1d04d84bb6f87f7c795b60ad43c7fdc0634`. +The artifact is retained only in the device's project-owned development directory; +no GitHub release/upload was performed. Installed launcher is `~/.local/bin/frameyap`. +Actual native CLI and installer both refused an independently held install lock. +At the final check, no owned `frameyap` process or transient `frameyap-*` runtime +directory remained. Installation/registration remain; nothing was autostarted. + +Final verification on source `d60d8d2`: default local build **8/8 CTests**; +optional native local build **9/9**; native ARM64 build **9/9**. Suites include +13 installer cases, worker framing/hash/cancellation/deadlines, gestures, state, +instance locking, and an isolated fake Wayland server. Those tests do not connect +to the live compositor or record audio. `git diff --check` passed locally. + +## Still open + +Runtime permission and any public distribution; live microphone quality/device +selection; physical tap/hold/buttons; actual text/Enter delivery into disposable +and real target apps; focus/held-modifier behavior; quick typing's focus observer; +scene/dashboard collisions; compositor cost, battery/thermal and headset comfort. +No live microphone recording or input injection occurred in this POC pass. diff --git a/docs/install-design.md b/docs/install-design.md index d549747..dadba07 100644 --- a/docs/install-design.md +++ b/docs/install-design.md @@ -1,8 +1,11 @@ # Installation and distribution goal **User goal:** install from GitHub with a `curl … | bash`-style command, without a -Steam store AppID. This is a requirement for the future release, not a working -installer. No release artifacts or functional installer are published yet. +Steam store AppID. An idempotent archive installer and native-only local artifacts +are now implemented/tested; no public release is published. The full bundled-ASR +experience remains blocked on runtime permission. This document retains the target +design; see [current packaging](packaging.md), [POC evidence](evidence/poc-cpu-overlay-2026-09-24.md) +and [third-party boundary](third-party.md). The read-only `scripts/install-preflight.sh` checks whether a host appears suitable for the **proposed** Linux ARM64 glibc package format and has the expected basic @@ -18,7 +21,7 @@ be certified until release artifacts are chosen and tested. OpenVR overlay applications do not require a Steam store AppID or Steamworks. The native executable initializes as `VRApplication_Overlay`. For discoverability and optional autolaunch, register an OpenVR application manifest with a stable, -project-owned **string application key** (proposed: `local.frame-dictation.overlay`). +project-owned **string application key** (proposed: `local.frameyap.overlay`). That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam library shortcut should be necessary for the normal route. @@ -39,7 +42,7 @@ restarting SteamVR behind the user's back. 1. Run one documented command from the eventual GitHub repository/release. 2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and explains/downloads the application, compatible CPU runtime and pinned model. -3. User-local installation provides a simple `frame-dictation` launcher, desktop +3. User-local installation provides a simple `frameyap` launcher, desktop entry where supported, and an OpenVR manifest. No compiler, engine checkout, Python dependency troubleshooting or separate ASR server for ordinary users. 4. The user explicitly launches/enables dictation. No installation-time microphone @@ -65,8 +68,8 @@ confirmation from stdin while the installer itself is arriving through that pipe attribution. A documented `--without-model` option can defer the large download. No surprise first-utterance downloads. Licensing may require obtaining particular runtime components from their vendor instead of redistributing them in our tarball. -- Install under `$XDG_DATA_HOME/frame-dictation` (default `~/.local/share/...`), - configuration under `$XDG_CONFIG_HOME/frame-dictation`, optional launcher in +- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`), + configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in `~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory. - No sudo, OS read-only-root changes, package-manager installs, udev changes, `/dev/uinput` permission changes or modifications to unrelated launchers. diff --git a/docs/overlay.md b/docs/overlay.md new file mode 100644 index 0000000..ca1bd53 --- /dev/null +++ b/docs/overlay.md @@ -0,0 +1,71 @@ +# Native OpenVR POC panel + +`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel +only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion. +Native `--run` wiring in `src/main.cpp` is implemented behind the explicit +`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful +compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a +normal build or test. + +## Rendering and controls + +Explicit development dependencies: Valve OpenVR SDK v2.15.6 and FreeType 2. +Configure/build must not fetch them. An installed font is passed explicitly; +Unicode coverage depends on that font. The 900×500 RGBA, 0.85 m panel uses +`SetOverlayRaw` only when its status, detail, transcript, recording timer, or +preview page changes. The caller may call `draw(Panel)` at 10 ms intervals. +The complete transcript preview is paginated by glyph width and three-line +height; Prev and Next controls navigate it. Long status/detail messages display +a prefix and a visible truncation marker rather than disappearing silently. +`hand=true` attaches to the validly tracked left controller and falls back to +head-relative placement otherwise. Overlay pointer controls are Record +(click-to-start/stop), Cancel, Insert, Enter, Quit and Prev, plus Next in the +transcript area. Record is available to retry during startup/error; Cancel can +stop worker startup. Enter is *always* a separate deliberate action, not +inferred from text. Recording never automatically submits Enter. + +`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert, +Enter. `bindings_frame_controller.json` uses the observed Frame profile's grip +click paths; both bound actions were reported tracked/active in the device check. +This does not prove physical gesture delivery. `bindings_knuckles.json` is an +additional **Index/knuckles example only**. Collisions with scene actions require +separate on-device validation. Left grip double tap +(releases <=250 ms, second press within 350 ms) requests explicit Enter only +when enabled. Right grip: first short squeeze and release (<=250 ms), then +second squeeze **down** within 350 ms starts capture; hold as long as needed +(up to runtime's clip bound), second **release** ends capture. The named PTT +action explicitly begins on down and ends on up. On tracking-pose invalidity, +action inactivity or overlay focus loss, a held capture emits Cancel, and +reconnection requires a neutral observation before any new press. PTT and left +Enter require an enabled panel; the clickable Record/Cancel controls remain +available while disabled. The action set has normal priority: global input +while a scene is active is not guaranteed, and experimental overlay overrides +are not switched on automatically. + +The UI cannot itself guarantee a capture started when a BeginRecord action +arrives: the owning runtime checks worker readiness. `Cancel` invalidates +capture/worker work; a failed startup can be retried with Record. The +`UiAction::Toggle` enumerator remains for caller ABI compatibility but the +panel no longer emits it: Prev is navigation only. A pointer Record click +emits `Record` to toggle capture, unlike controller PTT edges. + +## Registration + +`assets/application.vrmanifest.in` is a **template**, not a runnable manifest. +At install, substitute absolute executable and action JSON paths and store +it under the user-owned install directory, keeping bindings adjacent to action +JSON. Register `local.frameyap.overlay` using `registration(path,false,false)`; +autolaunch is opt-in. Unregister before deleting the installed manifest. +The app key is not a Steam store AppID. Registration uses OpenVR Utility init +only when explicitly invoked, then verifies `IsApplicationInstalled`. The device +required `binary_path_linux_arm` in the manifest (a generic or Linux-only path +was silently skipped despite successful AddApplicationManifest return). +Registration, repeated registration and removal were tested against the running +Frame runtime, with autolaunch verified off. The overlay explicitly identifies +its process with the registered app key before setting its action manifest. +Cold-runtime behavior and actual SteamVR-menu launch still need validation. +See [packaging](packaging.md); no published release is claimed here. + +A live headset check must be opt-in and distinguish overlay API discovery from +controller delivery, actual transcription, insertion into a disposable target, +and human comfort/acceptance. diff --git a/docs/poc.md b/docs/poc.md new file mode 100644 index 0000000..916bb43 --- /dev/null +++ b/docs/poc.md @@ -0,0 +1,136 @@ +# FrameYap POC: implementation and validation + +This is a standalone native application, not a plugin. Default builds/tests never +initialize OpenVR, open a microphone, run ASR, download files or inject input. + +## Implemented + +- Opt-in OpenVR RGBA overlay with head/left-hand placement, status, recording timer, + paginated UTF-8 preview, Record/Cancel/Insert/Enter/Quit controls. +- Remappable SteamVR actions. Default Steam Frame grip bindings use the observed + `frame_controller` profile. Right: short tap, then hold the second squeeze to + record; release to transcribe. Left: two short taps request explicit Enter. + First squeeze <=250 ms; second squeeze begins <=350 ms after first release. + Activity/tracking loss cancels a held recording and requires neutral rearm. +- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum, + 20 second maximum. Microphone is closed outside actual capture. Other apps may + still transmit your voice: this app does **not** mute VRChat or any other app. +- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips, + cancellation/reaping and deadlines; exact pinned model SHA-256 verification. + Model imports are lazy and loading is offline. No cloud/desktop fallback. +- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable + handling, UTF-8/control validation and explicit separate Submit action for Enter. +- Idempotent user-local release-archive installer: SHA-256, safe extraction, + atomic current-version selection, retained rollback, runtime/install lock, + foreign-file refusal and explicit unregister-before-uninstall acknowledgement. + +## Deliberately not claimed + +**Review-first only.** No automatic insertion or inferred commands. A transcript +must be explicitly inserted into the *current* focused destination. We do not yet +implement the proposed Xwayland focus-generation observer or safe quick typing. +There remains a race with focus changes after user approval. Text delivery is +reported as **input queued**, not application consumption or message delivery. +A request is consumed once even if transport completion is uncertain; no retries. +Unavailable IME acquisition leaves the preview intact. + +No VAD, always-listening mode, desktop transcription service, keyboard emulation +fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip bindings +are not guaranteed globally active in every scene/dashboard state, and the app +never enables SteamVR's experimental overlay overrides on your behalf. + +This is a compact prototype panel, not yet the proposed polished miniature status +chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery +impact and target application compatibility require further headset work. + +## Critical runtime licensing boundary + +Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. +The installed **kestrel-kernels 0.7.0** license is different: proprietary, requiring +an M87 Labs written agreement for use; copying/redistribution depends on that +agreement. PyPI availability is not permission. See [third-party notes](third-party.md). + +Do not publish a bundled Redux runtime, imply a public release is ready, or rerun +inference while applicable permission is unresolved. Our adapter is implemented; +that does not resolve distribution rights. An alternative runtime would be a +separately scoped and independently licensed implementation—not a silent model swap. + +## Developer native build + +Requirements: Linux, CMake/C++20, SDL3 >=3.2, Wayland client + scanner, FreeType, +and a deliberately provisioned standalone OpenVR **v2.15.6** SDK. No CMake fetches. + +```sh +cmake -S . -B build-native -DFRAMEYAP_NATIVE=ON \ + -DOPENVR_ROOT=/absolute/path/to/openvr +cmake --build build-native -j2 +ctest --test-dir build-native --output-on-failure +``` + +On ARM64, select the SDK's `lib/linuxarm64/libopenvr_api.so` explicitly with +`-DOPENVR_LIBRARY=...` if necessary. Do not use an x86 library or another +application's build/runtime. Installable binary has `$ORIGIN/../lib` RUNPATH; +producer must audit and bundle its compatible dependency closure. + +The device development trial built SDL **3.2.16** under a project-owned user +prefix with audio enabled and video/render/GPU/joystick/haptic/sensor/camera/tests +examples disabled (`SDL_UNIX_CONSOLE_BUILD=ON`). No OS package modifications. +That is a producer workflow, not an end-user compiler requirement. + +## Explicit hardware checks (not CTest) + +```sh +./build-native/frameyap --check-input --socket gamescope-0 +./build-native/frameyap --check-overlay --assets "$PWD/assets" --font /path/font.ttf --head +./build-native/frameyap --check-controls --assets "$PWD/assets" --font /path/font.ttf --head +``` + +The first acquires/releases an IME without text/actions. The second displays a +five-second inert panel. The third displays a 30-second diagnostic panel and +reports gestures, **without microphone or input injection**. Active action handles +are not proof that gestures were delivered. Checks must be explicitly launched +while the user expects the panel. Normal CLI/help/version remain inert. + +The ARM64 OpenVR loader's default `/data/work/openvrpaths.vrpath` failed on the +observed device. FrameYap respects `VR_PATHREG_OVERRIDE`; when absent, it selects +an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or +`~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files +are edited and no runtime/session restart is performed. + +## Deliberate launch, once runtime permission is settled + +Explicit setup downloads only the pinned, openly licensed model: + +```sh +python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model" +``` + +`fetch-model.py` is idempotent for matching hashes, rejects existing mismatched +files, and is never invoked by build/tests or first utterance. + +Provide your independently authorized CPU Python environment (moondream 2.4.0, +kestrel 0.8.0) and local weights: + +```sh +./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \ + --python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \ + --model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head +``` + +Use a disposable text destination first. `--run` loads the model but does not +record until an explicit recording control. Click Insert only after focusing your +intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery +and terminates only the owned worker. Installer upgrades refuse an active app. + +The worker API is intentionally small: a fork can replace the worker implementation +or add its own model/API integration without changing overlay and delivery code. +The default product remains local-only Redux; extending a fork does not authorize +sending existing users' audio to a service. + +## Benchmarks + +`scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV; +no recording, device input or automatic download. `--show-text` is a separate +explicit disclosure of that fixture's transcript. Use only with an authorized +runtime. See the dated [POC record](evidence/poc-cpu-overlay-2026-09-24.md) for +measurements and their limits; they are not headset/performance acceptance. diff --git a/docs/provenance.md b/docs/provenance.md index e97a1e9..d4d28a9 100644 --- a/docs/provenance.md +++ b/docs/provenance.md @@ -1,9 +1,10 @@ # Technical provenance and limits -Frame Dictation is a standalone scaffold and proposal. No external application -source, build tree, assets, Python environment, model weights or binaries are -included. The dated [device probe](evidence/frame-dictation-apis-2026-09-24.md) -records limited historical observations; its fixture code and private logs are +FrameYap is a standalone native POC. No unrelated application's source, build +tree, assets or Python environment is included. No model weights or runtime +binaries are in Git. The included Gamescope XML's provenance and separately +licensed build inputs are recorded in [third-party notes](third-party.md). +The dated [device probe](evidence/frameyap-apis-2026-09-24.md) records limited historical observations; its fixture code and private logs are not part of this repository. No current device availability or release install can be inferred from those observations. @@ -20,7 +21,9 @@ A preliminary, unpublished desktop benchmark used Python 3.12.13, moondream median/p95 were 61/137 ms and raw WER 11.11% on just 25 reviewed clips / 189 words. This is not a general accuracy estimate or Frame latency prediction. The CPU-selected desktop process also occupied GPU memory; GPU-free operation was -not established. No Frame inference benchmark exists yet. +not established in that earlier desktop trial. A later isolated ARM64 CPU-only +public-clip trial is recorded in [POC observations](evidence/poc-cpu-overlay-2026-09-24.md); +it does not establish live-microphone/headset acceptance or distribution permission. ## Public platform references diff --git a/docs/third-party.md b/docs/third-party.md new file mode 100644 index 0000000..63ae884 --- /dev/null +++ b/docs/third-party.md @@ -0,0 +1,77 @@ +# Dependency provenance and release boundary + +FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project +owner. This does not relicense external models, fonts, protocols or runtimes. +There is no dependency on another application's checkout, assets or environment. + +## Included source + +`protocol/gamescope-input-method.xml` is the unmodified public Gamescope +**3.16.28** protocol, downloaded from +. +SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`. +Its embedded permissive copyright/license notice is preserved. Generated bindings +are build outputs, not hand-written wire encoding. The private protocol may change +with SteamOS; compatibility must be rechecked. + +Frame controller bindings were authored here using the observed public input +profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code, +images, protected kernels or another application's assets were extracted. + +## Explicit native build inputs (not vendored) + +- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015; + retain its LICENSE with redistributed loader binaries. +- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix. +- Wayland client and scanner: retain upstream MIT-style notices. +- FreeType: choose and comply with its applicable FTL/GPL licensing option. +- Font: explicit user-supplied path; observed device check used system Hack Regular. + A release must include the selected font's own license and assess glyph coverage. +- Compiler runtime, libc minimum and transitive shared libraries require a release + dependency audit. Passing a developer build is not a portable-runtime guarantee. + +## Redux weights and proprietary runtime are different + +Public model: , exact revision +`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution: +Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3. +No modifications to the supplied weights are made. Exact model/config/tokenizer +sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`. +`fetch-model.py` fetches and retains the original model card alongside the files. +No weights are committed to this repository. + +The inspected `kestrel-kernels==0.7.0` wheel license identifies it as proprietary +M87 Labs software and says use requires a separate written agreement. Copying and +redistribution are restricted by that agreement. This includes its protected CPU +payload, not just CUDA. The Python wrapper and model card do not override those +terms. No attempt was made to unpack/decrypt/reverse-engineer protected kernels. + +**Release blocker:** permission covering use and redistribution has not been +established here. The initial on-device compatibility measurements preceded this +license review; further inference/bundling was paused when the issue was found. +The acquired packages remain isolated under the device's project-owned development +directory, not in Git or a published artifact. Do not advertise the GitHub runtime +bundle as available or automatically download/install those packages for end users. +Resolve permission with the vendor, or separately scope an independently licensed +runtime for the same weights. Do not silently substitute a dense/heavier model. + +Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) also +need their own notice/license inventory before publication. The public model's +small size is neither total runtime size nor redistribution permission. + +## CPU trial dependency choice + +Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**, +native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified +moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels; +these were replaced/removed from the owned venv before measurement. The measured +Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained +`pip install moondream` as a CPU setup recipe. + +For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution +was downloaded but not bundled with the proprietary runtime: +, +`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`, +SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`. +Its included licenses and compatible native dependencies still require review +before any release. No public prebuilt FrameYap release has been published. diff --git a/docs/worker.md b/docs/worker.md new file mode 100644 index 0000000..27833a3 --- /dev/null +++ b/docs/worker.md @@ -0,0 +1,64 @@ +# Offline Redux worker adapter (component, not an installed product) + +`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model, +threads=2)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for +one `WorkerReply` (text or generic per-request error). One request at a time; +no queue, no capture and no input injection. `stop()` discards pending audio, +terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL), +and is safe to repeat. Destruction stops it. `start()` returns without waiting +for model load; `poll()` reports warmup/load/crash/timeout/protocol errors by +throwing, then stops. Launch uses `posix_spawn`, safe with the host's OpenVR/SDL +threads, rather than running Python setup in a forked multithreaded child. +Caller must discard stale authorization/results after +cancellation; this component does not implement focus or delivery policy. + +The adapter requires an existing absolute, owner-private `$XDG_RUNTIME_DIR` +(no symlink at the final component), creates its own 0700 `mkdtemp` directory, +and writes only `clip.raw` with `O_EXCL|O_NOFOLLOW`, mode 0600. Clips are +3200..320000 finite float samples, mono 16 kHz, stored as little-endian IEEE +float32 (0.2..20 s). Files are unlinked after replies or shutdown, and the +private directory is removed. Private clips are not encrypted against the +account owner/root; do not use an untrusted runtime directory. The caller +should pass a trusted interpreter and script. Neither audio nor transcripts +are logged; child stderr is redirected to `/dev/null`, so worker diagnostics +are deliberately generic. + +The private pipes use unsigned LE32 payload lengths (1..65536), a one-byte +message type and, for requests/replies, unsigned LE64 request ID. `T` + ID +requests reading the fixed clip; `Y` means ready; `F` means load failure +(optional `M` for missing model, `D` for runtime failure); `R` + ID + UTF-8 +text and `E` + ID + generic UTF-8 error are replies. Text is at most 4096 +bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe +or oversized frame stops the worker. Warmup deadline is 120 s, transcription +deadline 60 s; `poll()` must be called regularly to enforce deadlines. It +never initializes a headset or starts a recording. There is no auto restart. + +`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI +startup, with HF/Transformers/Datasets offline variables and bounded native +thread-pool variables set before import. It uses +`md.photon("moondream/parakeet-redux", model_path=, +device="cpu", cpu_threads=threads)` and persistent +`transcribe(audio=, sample_rate=16000)["text"]`. +Install an **isolated** Python runtime with the separately reviewed +moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide +preinstalled local weights from revision +`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory +explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer +against `model_files.py` before importing model libraries. Protocol stdout is +isolated at the file-descriptor level from third-party diagnostics. Thread limits +cover Torch interop/native pools and CUDA is not selected. Offline environment +flags do not prove every third-party internal is unable to access a network. +Runtime/build/tests perform no downloads; the separate explicit setup utility +`scripts/fetch-model.py` can provision the public pinned weights. + +**Licensing blocker:** the observed kestrel-kernels 0.7.0 license requires a +separate M87 Labs agreement; do not treat wheel availability as permission for use +or bundling. See [third-party notes](third-party.md). No public runtime bundle has +been released. Limited ARM64 measurements are in the [POC record](evidence/poc-cpu-overlay-2026-09-24.md), +not a claim of complete headset acceptance. + +Hardware-free tests run through CTest, including fake-child cancellation, short +injected warmup/request deadlines, duplicate/stale replies, malformed frames, +missing/hash-mismatched model files and symlink refusal. Default production +deadlines remain 120/60 seconds. Python tests never import actual model libraries, +record a microphone, download assets or initialize OpenVR.