mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-06 01:00:04 +02:00
Document native Frame POC evidence and unresolved runtime permission
This commit is contained in:
baketnk committed
1 parent
d60d8d2f5d
commit
26113906f0
10 files changed
+553
-56
No files matched your search
@@ -1,53 +1,84 @@
|
||||
# Frame Dictation
|
||||
# FrameYap
|
||||
|
||||
Standalone, on-device voice typing for Steam Frame.
|
||||
Standalone, on-device voice typing POC for Steam Frame. **MIT licensed.**
|
||||
|
||||
**Status: project scaffold and design only.** The executable prints help/version;
|
||||
it does not yet render an overlay, open a microphone, load a model or type text.
|
||||
Implemented: native OpenVR overlay, remappable grip controls, bounded SDL3 capture,
|
||||
persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope,
|
||||
and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR
|
||||
server, cloud fallback or unrelated application dependency.
|
||||
|
||||
Planned path: **OpenVR overlay → local Parakeet Redux CPU inference → Gamescope
|
||||
Unicode input**. No external application checkout, library, submodule, desktop
|
||||
inference server or running scene host is required.
|
||||
**Status:** native ARM64 build, CPU inference on a public clip, overlay visibility,
|
||||
Gamescope discovery and native-only installation have been exercised on Frame.
|
||||
Live microphone → reviewed text → real target delivery is **not yet accepted**.
|
||||
|
||||
**Distribution goal:** one-command installation from GitHub releases, entirely
|
||||
user-local, with no Steam store AppID. See the [installer design](docs/install-design.md).
|
||||
There is no working installer or published release yet.
|
||||
**Runtime licensing remains unresolved.** Redux weights are CC-BY-4.0, but the
|
||||
observed Kestrel kernel runtime requires separate permission. We are checking with
|
||||
the vendor. Current native-only packages do **not** bundle or download that
|
||||
runtime; an independently authorized Python environment must be supplied.
|
||||
See [third-party notes](docs/third-party.md). No GitHub release is published yet.
|
||||
|
||||
For a read-only check of proposed release prerequisites on a Frame, run
|
||||
`sh scripts/install-preflight.sh`. It checks Linux AArch64/glibc and bootstrap
|
||||
tools, reports system Python (3.12+ for a possible source worker), Git and uv.
|
||||
Git and uv are not required for the planned bundled release. This check downloads
|
||||
and installs nothing; passing it does **not** mean an install or dictation works.
|
||||
## Controls
|
||||
|
||||
## Build the scaffold
|
||||
- **Right grip:** short tap, then hold the second squeeze to record; release to
|
||||
transcribe. Remappable through SteamVR bindings; separate hold-to-talk action too.
|
||||
- **Left grip:** double-tap to explicitly send Enter. Never inferred from speech.
|
||||
- **Overlay:** Record/Stop, Cancel, paginated preview, Insert, Enter and Quit.
|
||||
Dashboard lasers provide clickable controls without forcing global laser mode.
|
||||
- **Review-first:** focus your destination, then press Insert. No automatic insertion
|
||||
or submission. Maximum clip 20 seconds; accidental taps under 200 ms are discarded.
|
||||
- Other apps may still hear/transmit your voice. FrameYap does not mute them.
|
||||
|
||||
Requirements: CMake 3.20+ and a C++20 compiler. No third-party packages or downloads.
|
||||
Physical gesture timing, global bindings during games, mic capture, target-app
|
||||
compatibility and headset comfort still need coordinated validation. Successful
|
||||
API initialization is not delivered input or human acceptance.
|
||||
|
||||
## Build and test offline
|
||||
|
||||
CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests.
|
||||
|
||||
```sh
|
||||
cmake -S . -B build
|
||||
cmake --build build
|
||||
ctest --test-dir build --output-on-failure
|
||||
./build/frame-dictation --help
|
||||
./build/frameyap --help
|
||||
```
|
||||
|
||||
This is a host-native scaffold build, not an ARM64 deployment or headset test.
|
||||
Future OpenVR, audio and inference integrations must be explicitly configured;
|
||||
configuration/build/tests must never install packages or launch SteamVR implicitly.
|
||||
Default build has no hardware backends. It never downloads packages/models or
|
||||
initializes SteamVR, a microphone or input injection. Native dependencies and
|
||||
explicit launch/check commands are documented in [the POC guide](docs/poc.md).
|
||||
|
||||
## Installation
|
||||
|
||||
The real installer accepts a versioned, checksummed prebuilt ARM64 archive:
|
||||
|
||||
```sh
|
||||
sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
|
||||
--sha256 ARCHIVE_SHA256 --version VERSION
|
||||
```
|
||||
|
||||
This is the **local-artifact command shape**, not an available public download.
|
||||
Installation is user-local, retains rollback, refuses active-app upgrades and
|
||||
foreign files, and does not launch or register automatically. Registration uses
|
||||
OpenVR application key `local.frameyap.overlay`, not a Steam store AppID.
|
||||
Autolaunch is opt-in. See [packaging and lifecycle](docs/packaging.md).
|
||||
|
||||
A pinned GitHub one-command route is implemented in `install.sh --version TAG`,
|
||||
but **do not advertise or run it as a working public installation until a vetted
|
||||
release exists**. Native-only artifacts support the overlay/checks without an
|
||||
end-user compiler; full bundled-ASR distribution awaits runtime permission.
|
||||
|
||||
## Project map
|
||||
|
||||
- [Design](docs/design.md): UI, local inference, input backend and acceptance gates.
|
||||
- [Frame API evidence](docs/evidence/frame-dictation-apis-2026-09-24.md): successful
|
||||
mixed-script Unicode test and overlay-client initialization; known limits.
|
||||
- [Installer design](docs/install-design.md): GitHub install, standalone OpenVR
|
||||
identity, packaging, upgrades and uninstall.
|
||||
- [Provenance](docs/provenance.md): origin of the investigation, model/runtime pins.
|
||||
- `src/main.cpp`: inert CLI entry point.
|
||||
- `tests/cli.cmake`: hardware-free scaffold smoke test.
|
||||
- `scripts/install-preflight.sh`: read-only proposed packaging prerequisite check.
|
||||
- [Agent guidance](AGENTS.md): project boundaries and safe validation.
|
||||
- [POC guide](docs/poc.md): implemented boundaries, build, controls, explicit tests.
|
||||
- [Design](docs/design.md): full target design; some features remain proposed.
|
||||
- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md).
|
||||
- [Current POC observations](docs/evidence/poc-cpu-overlay-2026-09-24.md): measured
|
||||
CPU behavior and native installation checks, with acceptance limits.
|
||||
- [Earlier API evidence](docs/evidence/frameyap-apis-2026-09-24.md) and
|
||||
[provenance](docs/provenance.md): historical investigation, not live authority.
|
||||
- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md),
|
||||
[dependency/license inventory](docs/third-party.md).
|
||||
|
||||
First implementation gate: an isolated ARM64 CPU trial of the pinned Redux model.
|
||||
Then a minimal overlay and explicit insertion into a disposable target. Neither
|
||||
inference nor an overlay is implemented here yet. No model weights, recordings,
|
||||
credentials, engine assets or runtime binaries are included.
|
||||
No recordings, transcripts, private logs, model weights or runtime binaries are
|
||||
committed. The worker boundary is intentionally small for forks experimenting
|
||||
with other models/APIs; the default remains local-only Redux.
|
||||
+19
-10
@@ -1,8 +1,13 @@
|
||||
# Native Frame dictation overlay — proposal
|
||||
# FrameYap native dictation overlay — proposal
|
||||
|
||||
Standalone project design; see
|
||||
[provenance](provenance.md) and [historical Frame probes](evidence/frame-dictation-apis-2026-09-24.md).
|
||||
Only the inert build scaffold is implemented. This document describes proposed runtime behavior.
|
||||
[provenance](provenance.md) and [historical Frame probes](evidence/frameyap-apis-2026-09-24.md).
|
||||
This document is the full target design, not a blanket implementation claim.
|
||||
The native POC now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
|
||||
Quick typing/focus-generation tracking, polished status-chip UX and full hardware
|
||||
acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md)
|
||||
and [runtime licensing boundary](third-party.md).
|
||||
|
||||
## Recommendation
|
||||
|
||||
@@ -52,7 +57,10 @@ Recording… 00:04 [ Cancel ]
|
||||
|
||||
- States: disabled, warming, ready, recording, transcribing, review, inserted,
|
||||
unavailable/error. Recording uses visible icon + text, not color alone.
|
||||
- Hold to speak, release to finish. A click-to-start/stop overlay button provides
|
||||
- Default POC binding: tap right grip briefly, then hold the second squeeze to
|
||||
speak; release to finish. Double-tap left grip is a separate explicit Enter.
|
||||
Both are remappable, with a separate named hold-to-talk action available.
|
||||
A click-to-start/stop overlay button provides
|
||||
a binding-independent alternative. Bound recording to 20 seconds; discard
|
||||
accidental taps (initial threshold: 200 ms).
|
||||
- **Quick typing:** insert on completion only when the explicitly armed target
|
||||
@@ -67,7 +75,7 @@ Recording… 00:04 [ Cancel ]
|
||||
|
||||
## Host and rendering boundary
|
||||
|
||||
Implement the standalone `frame-dictation` executable, initialized
|
||||
Implement the standalone `frameyap` executable, initialized
|
||||
with `VRApplication_Overlay`. Frame accepted that application type and
|
||||
`IVROverlay_028` in the probe. Use `CreateOverlay`, tracked-device-relative
|
||||
transform, `ShowOverlay`/`HideOverlay`, and `PollNextOverlayEvent` for the panel.
|
||||
@@ -213,7 +221,7 @@ Suggested ownership, introduced only as implementation needs it:
|
||||
- `src/audio.*`: explicit SDL capture and bounded mono PCM.
|
||||
- `src/worker.*`: one local child process, bounded requests/replies, timeout/reaping.
|
||||
- `src/text_input.*`: Gamescope protocol, focus observation and delivery policy.
|
||||
- `python/frame_dictation/`: persistent CPU Redux worker, no external application imports.
|
||||
- `python/frameyap/`: persistent CPU Redux worker, no external application imports.
|
||||
|
||||
The worker reads a fixed private clip and returns an ID-correlated literal string;
|
||||
one request at a time, 64 KiB framed messages, 4096-byte transcript, and a bounded
|
||||
@@ -280,10 +288,11 @@ without moving recognition off Frame. It is not part of this initial design.
|
||||
|
||||
## Standalone dependency and test policy
|
||||
|
||||
The scaffold currently needs only CMake and a C++20 compiler. Later integrations
|
||||
will explicitly select OpenVR, SDL3, Wayland client/protocol bindings and a Python
|
||||
Redux environment. Pin revisions and review licenses when introduced. No automatic
|
||||
fetch/install in configure or normal tests; no external checkout discovery.
|
||||
The default hardware-free build needs only CMake and a C++20 compiler (Python
|
||||
runs additional offline tests). `FRAMEYAP_NATIVE=ON` explicitly selects OpenVR,
|
||||
SDL3, FreeType and Wayland client/generated protocol bindings. A separately
|
||||
authorized Python Redux environment is explicitly supplied at launch. Pin revisions
|
||||
and review licenses when introduced. No automatic fetch/install in configure or normal tests; no external checkout discovery.
|
||||
|
||||
Hardware-free tests should cover state transitions, bounded PCM/transcripts,
|
||||
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
|
||||
|
||||
File renamed without changes.
@@ -0,0 +1,103 @@
|
||||
# FrameYap POC observations — 2026-09-24
|
||||
|
||||
Dated observations on one user-authorized Steam Frame, not proof of future device
|
||||
availability, full dictation acceptance or runtime redistribution rights. No audio,
|
||||
weights, private transcript/log files, credentials or runtime binaries are included.
|
||||
The final native package below was built from source commit **`d60d8d2`**.
|
||||
|
||||
## CPU compatibility trial (before runtime-license review)
|
||||
|
||||
Isolated user-local environment: ARM64, glibc 2.39, Python 3.12.3,
|
||||
moondream 2.4.0 / kestrel 0.8.0 / kernels 0.7.0 / native 0.1.8,
|
||||
Torch **2.8.0+cpu** (`torch.version.cuda is None`). Model weights/config/tokenizer
|
||||
verified against pinned Redux revision `fad622f25f303105c20d70e201bcc477c88b620c`.
|
||||
No dense model substitution or desktop/cloud inference was used.
|
||||
|
||||
A one-second synthetic silence trial loaded in 5.645 s; three decodes took
|
||||
0.187 / 0.149 / 0.167 s and returned no text. Peak process RSS in that trial was
|
||||
989,844 KiB (~967 MiB). This is not peak RSS for arbitrary 20-second speech clips.
|
||||
|
||||
Public 11.0-second JFK speech fixture:
|
||||
<https://github.com/openai/whisper/blob/v20250625/tests/jfk.flac>.
|
||||
Source FLAC SHA-256 `63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715`.
|
||||
Converted to PCM16/mono/16 kHz WAV with ffmpeg; WAV SHA-256
|
||||
`0c397b12d7dbdd89e4fd0d9a0840c14a2c5c3707758562755cda96a931b25a91`.
|
||||
The persistent worker used bounded file/pipe IPC; no microphone was opened.
|
||||
|
||||
| CPU threads | Worker load | Five request times (seconds) | Median | RTF | Audio/compute ratio |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2 | 5.190 s | 1.550, 1.619, 1.587, 1.619, 1.659 | 1.619 s | 0.147 | 6.8× realtime |
|
||||
| 4 | 4.655 s | 1.196, 1.158, 1.143, 1.249, 1.502 | 1.196 s | 0.109 | 9.2× realtime |
|
||||
|
||||
The first result reproduced the fixture's known sentence; all five replies in
|
||||
each run were correlated and 108 UTF-8 bytes. These are medians/maxima from one
|
||||
public clip, not representative accuracy/WER, p95, streaming realtime performance,
|
||||
or compositor/thermal acceptance. Load time is excluded from RTF. Two threads
|
||||
remain the default pending scene-contention measurements.
|
||||
|
||||
An unconstrained dependency install initially selected CUDA Torch/NVIDIA wheels;
|
||||
these were replaced/removed from the owned venv before measurement. No system
|
||||
packages were changed. Later inspection found the proprietary kernel license's
|
||||
separate-agreement requirement. Permission was declared unresolved; further
|
||||
inference and proprietary-runtime packaging were paused pending vendor clarification.
|
||||
See [license boundary](../third-party.md). The acquired development environment
|
||||
remains user-local and is not included in the native-only installation.
|
||||
|
||||
## Native runtime / controller / overlay checks
|
||||
|
||||
- Native CMake build succeeded on Frame using standalone OpenVR SDK v2.15.6,
|
||||
explicitly built SDL 3.2.16, system Wayland and FreeType. No unrelated app tree
|
||||
or environment was linked/copied. No sudo or system/session restart.
|
||||
- Installed executable connected to `gamescope-0`, bound the IME and received
|
||||
ready/done state; each discovery check disconnected **without text or Enter**.
|
||||
- OpenVR accepted the raw panel. The user reported seeing the five-second window.
|
||||
This confirms visibility, not readability/comfort or successful pointer clicks.
|
||||
- Device controller properties identified `frame_controller` and its input profile.
|
||||
The profile exposes grip `click`; authored default bindings were added. A
|
||||
controls-only run reported both grip actions tracked/active. **No physical
|
||||
gesture delivery was observed in that run.** Double-tap/hold/loss behavior is
|
||||
currently verified by deterministic unit tests, not a human controller trial.
|
||||
- The ARM loader initially tried `/data/work/openvrpaths.vrpath`. Selecting the
|
||||
existing standard user registry fixed initialization; code now does so only
|
||||
when no explicit override is supplied.
|
||||
- Registration initially returned API success but did not install the app.
|
||||
Frame's manifest parser requires **`binary_path_linux_arm`**. Adding it fixed
|
||||
registration. Code now verifies `IsApplicationInstalled`, not return status alone.
|
||||
- Repeated Add, Remove and re-Add of **only** `local.frameyap.overlay` succeeded;
|
||||
autolaunch queried **off**. The final installed panel check also succeeded after
|
||||
explicit identification with that registered key. Menu-driven launch and cold
|
||||
SteamVR startup were not exercised.
|
||||
|
||||
## Installer and final state
|
||||
|
||||
Native-only artifacts `v0.1.0-poc1`/`poc2` exercised installation, same-version
|
||||
rerun, upgrade, rollback and forward selection. A producer-prefix SDL RUNPATH leak
|
||||
was found and removed; final ELF RUNPATH is only `$ORIGIN/../lib` and bundled
|
||||
SDL/OpenVR resolve inside the installed tree.
|
||||
|
||||
Explicit OpenVR unregister followed by `--uninstall --unregistered` succeeded.
|
||||
The final `v0.1.0-poc3` artifact was then installed twice and registered with
|
||||
autolaunch off. It includes native code, SDL/OpenVR, Hack font/notices and the
|
||||
worker adapter, **no ASR runtime or weights**. Installed size: ~8.1 MiB.
|
||||
|
||||
Local archive SHA-256:
|
||||
`90aab107feea2cb501bf815806bdc1d04d84bb6f87f7c795b60ad43c7fdc0634`.
|
||||
The artifact is retained only in the device's project-owned development directory;
|
||||
no GitHub release/upload was performed. Installed launcher is `~/.local/bin/frameyap`.
|
||||
Actual native CLI and installer both refused an independently held install lock.
|
||||
At the final check, no owned `frameyap` process or transient `frameyap-*` runtime
|
||||
directory remained. Installation/registration remain; nothing was autostarted.
|
||||
|
||||
Final verification on source `d60d8d2`: default local build **8/8 CTests**;
|
||||
optional native local build **9/9**; native ARM64 build **9/9**. Suites include
|
||||
13 installer cases, worker framing/hash/cancellation/deadlines, gestures, state,
|
||||
instance locking, and an isolated fake Wayland server. Those tests do not connect
|
||||
to the live compositor or record audio. `git diff --check` passed locally.
|
||||
|
||||
## Still open
|
||||
|
||||
Runtime permission and any public distribution; live microphone quality/device
|
||||
selection; physical tap/hold/buttons; actual text/Enter delivery into disposable
|
||||
and real target apps; focus/held-modifier behavior; quick typing's focus observer;
|
||||
scene/dashboard collisions; compositor cost, battery/thermal and headset comfort.
|
||||
No live microphone recording or input injection occurred in this POC pass.
|
||||
@@ -1,8 +1,11 @@
|
||||
# Installation and distribution goal
|
||||
|
||||
**User goal:** install from GitHub with a `curl … | bash`-style command, without a
|
||||
Steam store AppID. This is a requirement for the future release, not a working
|
||||
installer. No release artifacts or functional installer are published yet.
|
||||
Steam store AppID. An idempotent archive installer and native-only local artifacts
|
||||
are now implemented/tested; no public release is published. The full bundled-ASR
|
||||
experience remains blocked on runtime permission. This document retains the target
|
||||
design; see [current packaging](packaging.md), [POC evidence](evidence/poc-cpu-overlay-2026-09-24.md)
|
||||
and [third-party boundary](third-party.md).
|
||||
|
||||
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable
|
||||
for the **proposed** Linux ARM64 glibc package format and has the expected basic
|
||||
@@ -18,7 +21,7 @@ be certified until release artifacts are chosen and tested.
|
||||
OpenVR overlay applications do not require a Steam store AppID or Steamworks.
|
||||
The native executable initializes as `VRApplication_Overlay`. For discoverability
|
||||
and optional autolaunch, register an OpenVR application manifest with a stable,
|
||||
project-owned **string application key** (proposed: `local.frame-dictation.overlay`).
|
||||
project-owned **string application key** (proposed: `local.frameyap.overlay`).
|
||||
That key is not a numeric Steam AppID. No purchase/store listing or non-Steam Steam
|
||||
library shortcut should be necessary for the normal route.
|
||||
|
||||
@@ -39,7 +42,7 @@ restarting SteamVR behind the user's back.
|
||||
1. Run one documented command from the eventual GitHub repository/release.
|
||||
2. Installer identifies native Linux ARM64 Frame, resolves a pinned release and
|
||||
explains/downloads the application, compatible CPU runtime and pinned model.
|
||||
3. User-local installation provides a simple `frame-dictation` launcher, desktop
|
||||
3. User-local installation provides a simple `frameyap` launcher, desktop
|
||||
entry where supported, and an OpenVR manifest. No compiler, engine checkout,
|
||||
Python dependency troubleshooting or separate ASR server for ordinary users.
|
||||
4. The user explicitly launches/enables dictation. No installation-time microphone
|
||||
@@ -65,8 +68,8 @@ confirmation from stdin while the installer itself is arriving through that pipe
|
||||
attribution. A documented `--without-model` option can defer the large download.
|
||||
No surprise first-utterance downloads. Licensing may require obtaining particular
|
||||
runtime components from their vendor instead of redistributing them in our tarball.
|
||||
- Install under `$XDG_DATA_HOME/frame-dictation` (default `~/.local/share/...`),
|
||||
configuration under `$XDG_CONFIG_HOME/frame-dictation`, optional launcher in
|
||||
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
|
||||
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
|
||||
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
|
||||
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
|
||||
`/dev/uinput` permission changes or modifications to unrelated launchers.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# Native OpenVR POC panel
|
||||
|
||||
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
|
||||
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
|
||||
Native `--run` wiring in `src/main.cpp` is implemented behind the explicit
|
||||
`FRAMEYAP_NATIVE` build option. Neither hardware-free tests nor a successful
|
||||
compile establish Frame input, visibility, comfort or text delivery. Launching the runtime is explicit, never part of a
|
||||
normal build or test.
|
||||
|
||||
## Rendering and controls
|
||||
|
||||
Explicit development dependencies: Valve OpenVR SDK v2.15.6 and FreeType 2.
|
||||
Configure/build must not fetch them. An installed font is passed explicitly;
|
||||
Unicode coverage depends on that font. The 900×500 RGBA, 0.85 m panel uses
|
||||
`SetOverlayRaw` only when its status, detail, transcript, recording timer, or
|
||||
preview page changes. The caller may call `draw(Panel)` at 10 ms intervals.
|
||||
The complete transcript preview is paginated by glyph width and three-line
|
||||
height; Prev and Next controls navigate it. Long status/detail messages display
|
||||
a prefix and a visible truncation marker rather than disappearing silently.
|
||||
`hand=true` attaches to the validly tracked left controller and falls back to
|
||||
head-relative placement otherwise. Overlay pointer controls are Record
|
||||
(click-to-start/stop), Cancel, Insert, Enter, Quit and Prev, plus Next in the
|
||||
transcript area. Record is available to retry during startup/error; Cancel can
|
||||
stop worker startup. Enter is *always* a separate deliberate action, not
|
||||
inferred from text. Recording never automatically submits Enter.
|
||||
|
||||
`assets/actions.json` names six actions: left/right grip, PTT, cancel, insert,
|
||||
Enter. `bindings_frame_controller.json` uses the observed Frame profile's grip
|
||||
click paths; both bound actions were reported tracked/active in the device check.
|
||||
This does not prove physical gesture delivery. `bindings_knuckles.json` is an
|
||||
additional **Index/knuckles example only**. Collisions with scene actions require
|
||||
separate on-device validation. Left grip double tap
|
||||
(releases <=250 ms, second press within 350 ms) requests explicit Enter only
|
||||
when enabled. Right grip: first short squeeze and release (<=250 ms), then
|
||||
second squeeze **down** within 350 ms starts capture; hold as long as needed
|
||||
(up to runtime's clip bound), second **release** ends capture. The named PTT
|
||||
action explicitly begins on down and ends on up. On tracking-pose invalidity,
|
||||
action inactivity or overlay focus loss, a held capture emits Cancel, and
|
||||
reconnection requires a neutral observation before any new press. PTT and left
|
||||
Enter require an enabled panel; the clickable Record/Cancel controls remain
|
||||
available while disabled. The action set has normal priority: global input
|
||||
while a scene is active is not guaranteed, and experimental overlay overrides
|
||||
are not switched on automatically.
|
||||
|
||||
The UI cannot itself guarantee a capture started when a BeginRecord action
|
||||
arrives: the owning runtime checks worker readiness. `Cancel` invalidates
|
||||
capture/worker work; a failed startup can be retried with Record. The
|
||||
`UiAction::Toggle` enumerator remains for caller ABI compatibility but the
|
||||
panel no longer emits it: Prev is navigation only. A pointer Record click
|
||||
emits `Record` to toggle capture, unlike controller PTT edges.
|
||||
|
||||
## Registration
|
||||
|
||||
`assets/application.vrmanifest.in` is a **template**, not a runnable manifest.
|
||||
At install, substitute absolute executable and action JSON paths and store
|
||||
it under the user-owned install directory, keeping bindings adjacent to action
|
||||
JSON. Register `local.frameyap.overlay` using `registration(path,false,false)`;
|
||||
autolaunch is opt-in. Unregister before deleting the installed manifest.
|
||||
The app key is not a Steam store AppID. Registration uses OpenVR Utility init
|
||||
only when explicitly invoked, then verifies `IsApplicationInstalled`. The device
|
||||
required `binary_path_linux_arm` in the manifest (a generic or Linux-only path
|
||||
was silently skipped despite successful AddApplicationManifest return).
|
||||
Registration, repeated registration and removal were tested against the running
|
||||
Frame runtime, with autolaunch verified off. The overlay explicitly identifies
|
||||
its process with the registered app key before setting its action manifest.
|
||||
Cold-runtime behavior and actual SteamVR-menu launch still need validation.
|
||||
See [packaging](packaging.md); no published release is claimed here.
|
||||
|
||||
A live headset check must be opt-in and distinguish overlay API discovery from
|
||||
controller delivery, actual transcription, insertion into a disposable target,
|
||||
and human comfort/acceptance.
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
# FrameYap POC: implementation and validation
|
||||
|
||||
This is a standalone native application, not a plugin. Default builds/tests never
|
||||
initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
||||
|
||||
## Implemented
|
||||
|
||||
- Opt-in OpenVR RGBA overlay with head/left-hand placement, status, recording timer,
|
||||
paginated UTF-8 preview, Record/Cancel/Insert/Enter/Quit controls.
|
||||
- Remappable SteamVR actions. Default Steam Frame grip bindings use the observed
|
||||
`frame_controller` profile. Right: short tap, then hold the second squeeze to
|
||||
record; release to transcribe. Left: two short taps request explicit Enter.
|
||||
First squeeze <=250 ms; second squeeze begins <=350 ms after first release.
|
||||
Activity/tracking loss cancels a held recording and requires neutral rearm.
|
||||
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
|
||||
20 second maximum. Microphone is closed outside actual capture. Other apps may
|
||||
still transmit your voice: this app does **not** mute VRChat or any other app.
|
||||
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
|
||||
cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
|
||||
Model imports are lazy and loading is offline. No cloud/desktop fallback.
|
||||
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
|
||||
handling, UTF-8/control validation and explicit separate Submit action for Enter.
|
||||
- Idempotent user-local release-archive installer: SHA-256, safe extraction,
|
||||
atomic current-version selection, retained rollback, runtime/install lock,
|
||||
foreign-file refusal and explicit unregister-before-uninstall acknowledgement.
|
||||
|
||||
## Deliberately not claimed
|
||||
|
||||
**Review-first only.** No automatic insertion or inferred commands. A transcript
|
||||
must be explicitly inserted into the *current* focused destination. We do not yet
|
||||
implement the proposed Xwayland focus-generation observer or safe quick typing.
|
||||
There remains a race with focus changes after user approval. Text delivery is
|
||||
reported as **input queued**, not application consumption or message delivery.
|
||||
A request is consumed once even if transport completion is uncertain; no retries.
|
||||
Unavailable IME acquisition leaves the preview intact.
|
||||
|
||||
No VAD, always-listening mode, desktop transcription service, keyboard emulation
|
||||
fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip bindings
|
||||
are not guaranteed globally active in every scene/dashboard state, and the app
|
||||
never enables SteamVR's experimental overlay overrides on your behalf.
|
||||
|
||||
This is a compact prototype panel, not yet the proposed polished miniature status
|
||||
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
|
||||
impact and target application compatibility require further headset work.
|
||||
|
||||
## Critical runtime licensing boundary
|
||||
|
||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0.
|
||||
The installed **kestrel-kernels 0.7.0** license is different: proprietary, requiring
|
||||
an M87 Labs written agreement for use; copying/redistribution depends on that
|
||||
agreement. PyPI availability is not permission. See [third-party notes](third-party.md).
|
||||
|
||||
Do not publish a bundled Redux runtime, imply a public release is ready, or rerun
|
||||
inference while applicable permission is unresolved. Our adapter is implemented;
|
||||
that does not resolve distribution rights. An alternative runtime would be a
|
||||
separately scoped and independently licensed implementation—not a silent model swap.
|
||||
|
||||
## Developer native build
|
||||
|
||||
Requirements: Linux, CMake/C++20, SDL3 >=3.2, Wayland client + scanner, FreeType,
|
||||
and a deliberately provisioned standalone OpenVR **v2.15.6** SDK. No CMake fetches.
|
||||
|
||||
```sh
|
||||
cmake -S . -B build-native -DFRAMEYAP_NATIVE=ON \
|
||||
-DOPENVR_ROOT=/absolute/path/to/openvr
|
||||
cmake --build build-native -j2
|
||||
ctest --test-dir build-native --output-on-failure
|
||||
```
|
||||
|
||||
On ARM64, select the SDK's `lib/linuxarm64/libopenvr_api.so` explicitly with
|
||||
`-DOPENVR_LIBRARY=...` if necessary. Do not use an x86 library or another
|
||||
application's build/runtime. Installable binary has `$ORIGIN/../lib` RUNPATH;
|
||||
producer must audit and bundle its compatible dependency closure.
|
||||
|
||||
The device development trial built SDL **3.2.16** under a project-owned user
|
||||
prefix with audio enabled and video/render/GPU/joystick/haptic/sensor/camera/tests
|
||||
examples disabled (`SDL_UNIX_CONSOLE_BUILD=ON`). No OS package modifications.
|
||||
That is a producer workflow, not an end-user compiler requirement.
|
||||
|
||||
## Explicit hardware checks (not CTest)
|
||||
|
||||
```sh
|
||||
./build-native/frameyap --check-input --socket gamescope-0
|
||||
./build-native/frameyap --check-overlay --assets "$PWD/assets" --font /path/font.ttf --head
|
||||
./build-native/frameyap --check-controls --assets "$PWD/assets" --font /path/font.ttf --head
|
||||
```
|
||||
|
||||
The first acquires/releases an IME without text/actions. The second displays a
|
||||
five-second inert panel. The third displays a 30-second diagnostic panel and
|
||||
reports gestures, **without microphone or input injection**. Active action handles
|
||||
are not proof that gestures were delivered. Checks must be explicitly launched
|
||||
while the user expects the panel. Normal CLI/help/version remain inert.
|
||||
|
||||
The ARM64 OpenVR loader's default `/data/work/openvrpaths.vrpath` failed on the
|
||||
observed device. FrameYap respects `VR_PATHREG_OVERRIDE`; when absent, it selects
|
||||
an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or
|
||||
`~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files
|
||||
are edited and no runtime/session restart is performed.
|
||||
|
||||
## Deliberate launch, once runtime permission is settled
|
||||
|
||||
Explicit setup downloads only the pinned, openly licensed model:
|
||||
|
||||
```sh
|
||||
python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model"
|
||||
```
|
||||
|
||||
`fetch-model.py` is idempotent for matching hashes, rejects existing mismatched
|
||||
files, and is never invoked by build/tests or first utterance.
|
||||
|
||||
Provide your independently authorized CPU Python environment (moondream 2.4.0,
|
||||
kestrel 0.8.0) and local weights:
|
||||
|
||||
```sh
|
||||
./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \
|
||||
--python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
|
||||
--model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head
|
||||
```
|
||||
|
||||
Use a disposable text destination first. `--run` loads the model but does not
|
||||
record until an explicit recording control. Click Insert only after focusing your
|
||||
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
|
||||
and terminates only the owned worker. Installer upgrades refuse an active app.
|
||||
|
||||
The worker API is intentionally small: a fork can replace the worker implementation
|
||||
or add its own model/API integration without changing overlay and delivery code.
|
||||
The default product remains local-only Redux; extending a fork does not authorize
|
||||
sending existing users' audio to a service.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
`scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV;
|
||||
no recording, device input or automatic download. `--show-text` is a separate
|
||||
explicit disclosure of that fixture's transcript. Use only with an authorized
|
||||
runtime. See the dated [POC record](evidence/poc-cpu-overlay-2026-09-24.md) for
|
||||
measurements and their limits; they are not headset/performance acceptance.
|
||||
+8
-5
@@ -1,9 +1,10 @@
|
||||
# Technical provenance and limits
|
||||
|
||||
Frame Dictation is a standalone scaffold and proposal. No external application
|
||||
source, build tree, assets, Python environment, model weights or binaries are
|
||||
included. The dated [device probe](evidence/frame-dictation-apis-2026-09-24.md)
|
||||
records limited historical observations; its fixture code and private logs are
|
||||
FrameYap is a standalone native POC. No unrelated application's source, build
|
||||
tree, assets or Python environment is included. No model weights or runtime
|
||||
binaries are in Git. The included Gamescope XML's provenance and separately
|
||||
licensed build inputs are recorded in [third-party notes](third-party.md).
|
||||
The dated [device probe](evidence/frameyap-apis-2026-09-24.md) records limited historical observations; its fixture code and private logs are
|
||||
not part of this repository. No current device availability or release install
|
||||
can be inferred from those observations.
|
||||
|
||||
@@ -20,7 +21,9 @@ A preliminary, unpublished desktop benchmark used Python 3.12.13, moondream
|
||||
median/p95 were 61/137 ms and raw WER 11.11% on just 25 reviewed clips / 189
|
||||
words. This is not a general accuracy estimate or Frame latency prediction. The
|
||||
CPU-selected desktop process also occupied GPU memory; GPU-free operation was
|
||||
not established. No Frame inference benchmark exists yet.
|
||||
not established in that earlier desktop trial. A later isolated ARM64 CPU-only
|
||||
public-clip trial is recorded in [POC observations](evidence/poc-cpu-overlay-2026-09-24.md);
|
||||
it does not establish live-microphone/headset acceptance or distribution permission.
|
||||
|
||||
## Public platform references
|
||||
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# Dependency provenance and release boundary
|
||||
|
||||
FrameYap's original code is [MIT licensed](../LICENSE), as selected by the project
|
||||
owner. This does not relicense external models, fonts, protocols or runtimes.
|
||||
There is no dependency on another application's checkout, assets or environment.
|
||||
|
||||
## Included source
|
||||
|
||||
`protocol/gamescope-input-method.xml` is the unmodified public Gamescope
|
||||
**3.16.28** protocol, downloaded from
|
||||
<https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml>.
|
||||
SHA-256: `da35711f5d1d750bc47931132a89bf34e6b96a72bafc054d34092d3f42358ec4`.
|
||||
Its embedded permissive copyright/license notice is preserved. Generated bindings
|
||||
are build outputs, not hand-written wire encoding. The private protocol may change
|
||||
with SteamOS; compatibility must be rechecked.
|
||||
|
||||
Frame controller bindings were authored here using the observed public input
|
||||
profile names (`frame_controller`, `/input/grip`, `click`); no SteamVR driver code,
|
||||
images, protected kernels or another application's assets were extracted.
|
||||
|
||||
## Explicit native build inputs (not vendored)
|
||||
|
||||
- Valve OpenVR SDK v2.15.6: BSD-3-Clause-style license, copyright Valve 2015;
|
||||
retain its LICENSE with redistributed loader binaries.
|
||||
- SDL3: zlib license; device trial used SDL 3.2.16 built in a private user prefix.
|
||||
- Wayland client and scanner: retain upstream MIT-style notices.
|
||||
- FreeType: choose and comply with its applicable FTL/GPL licensing option.
|
||||
- Font: explicit user-supplied path; observed device check used system Hack Regular.
|
||||
A release must include the selected font's own license and assess glyph coverage.
|
||||
- Compiler runtime, libc minimum and transitive shared libraries require a release
|
||||
dependency audit. Passing a developer build is not a portable-runtime guarantee.
|
||||
|
||||
## Redux weights and proprietary runtime are different
|
||||
|
||||
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution:
|
||||
Moondream/M87 Labs, Parakeet Redux, derived from NVIDIA Parakeet TDT 0.6B v3.
|
||||
No modifications to the supplied weights are made. Exact model/config/tokenizer
|
||||
sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`.
|
||||
`fetch-model.py` fetches and retains the original model card alongside the files.
|
||||
No weights are committed to this repository.
|
||||
|
||||
The inspected `kestrel-kernels==0.7.0` wheel license identifies it as proprietary
|
||||
M87 Labs software and says use requires a separate written agreement. Copying and
|
||||
redistribution are restricted by that agreement. This includes its protected CPU
|
||||
payload, not just CUDA. The Python wrapper and model card do not override those
|
||||
terms. No attempt was made to unpack/decrypt/reverse-engineer protected kernels.
|
||||
|
||||
**Release blocker:** permission covering use and redistribution has not been
|
||||
established here. The initial on-device compatibility measurements preceded this
|
||||
license review; further inference/bundling was paused when the issue was found.
|
||||
The acquired packages remain isolated under the device's project-owned development
|
||||
directory, not in Git or a published artifact. Do not advertise the GitHub runtime
|
||||
bundle as available or automatically download/install those packages for end users.
|
||||
Resolve permission with the vendor, or separately scope an independently licensed
|
||||
runtime for the same weights. Do not silently substitute a dense/heavier model.
|
||||
|
||||
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) also
|
||||
need their own notice/license inventory before publication. The public model's
|
||||
small size is neither total runtime size nor redistribution permission.
|
||||
|
||||
## CPU trial dependency choice
|
||||
|
||||
Trial interface pins: moondream **2.4.0**, kestrel **0.8.0**, kernels **0.7.0**,
|
||||
native **0.1.8**, Python **3.12.3**, Torch **2.8.0+cpu** on ARM64. An unqualified
|
||||
moondream install initially resolved a CUDA-enabled Torch and NVIDIA wheels;
|
||||
these were replaced/removed from the owned venv before measurement. The measured
|
||||
Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
|
||||
`pip install moondream` as a CPU setup recipe.
|
||||
|
||||
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
|
||||
was downloaded but not bundled with the proprietary runtime:
|
||||
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
||||
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
|
||||
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
|
||||
Its included licenses and compatible native dependencies still require review
|
||||
before any release. No public prebuilt FrameYap release has been published.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Offline Redux worker adapter (component, not an installed product)
|
||||
|
||||
`src/worker.hpp` provides `frameyap::Worker`: call `start(python, script, model,
|
||||
threads=2)` explicitly, poll until `ready()`, then `submit(id, pcm)` and poll for
|
||||
one `WorkerReply` (text or generic per-request error). One request at a time;
|
||||
no queue, no capture and no input injection. `stop()` discards pending audio,
|
||||
terminates/reaps **only its direct child** (TERM, bounded 500 ms, then KILL),
|
||||
and is safe to repeat. Destruction stops it. `start()` returns without waiting
|
||||
for model load; `poll()` reports warmup/load/crash/timeout/protocol errors by
|
||||
throwing, then stops. Launch uses `posix_spawn`, safe with the host's OpenVR/SDL
|
||||
threads, rather than running Python setup in a forked multithreaded child.
|
||||
Caller must discard stale authorization/results after
|
||||
cancellation; this component does not implement focus or delivery policy.
|
||||
|
||||
The adapter requires an existing absolute, owner-private `$XDG_RUNTIME_DIR`
|
||||
(no symlink at the final component), creates its own 0700 `mkdtemp` directory,
|
||||
and writes only `clip.raw` with `O_EXCL|O_NOFOLLOW`, mode 0600. Clips are
|
||||
3200..320000 finite float samples, mono 16 kHz, stored as little-endian IEEE
|
||||
float32 (0.2..20 s). Files are unlinked after replies or shutdown, and the
|
||||
private directory is removed. Private clips are not encrypted against the
|
||||
account owner/root; do not use an untrusted runtime directory. The caller
|
||||
should pass a trusted interpreter and script. Neither audio nor transcripts
|
||||
are logged; child stderr is redirected to `/dev/null`, so worker diagnostics
|
||||
are deliberately generic.
|
||||
|
||||
The private pipes use unsigned LE32 payload lengths (1..65536), a one-byte
|
||||
message type and, for requests/replies, unsigned LE64 request ID. `T` + ID
|
||||
requests reading the fixed clip; `Y` means ready; `F` means load failure
|
||||
(optional `M` for missing model, `D` for runtime failure); `R` + ID + UTF-8
|
||||
text and `E` + ID + generic UTF-8 error are replies. Text is at most 4096
|
||||
bytes. An unexpected or duplicate reply, wrong ID, extra frame, closed pipe
|
||||
or oversized frame stops the worker. Warmup deadline is 120 s, transcription
|
||||
deadline 60 s; `poll()` must be called regularly to enforce deadlines. It
|
||||
never initializes a headset or starts a recording. There is no auto restart.
|
||||
|
||||
`python/frameyap/worker.py` lazily imports `moondream` only after explicit CLI
|
||||
startup, with HF/Transformers/Datasets offline variables and bounded native
|
||||
thread-pool variables set before import. It uses
|
||||
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
|
||||
device="cpu", cpu_threads=threads)` and persistent
|
||||
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
|
||||
Install an **isolated** Python runtime with the separately reviewed
|
||||
moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
|
||||
preinstalled local weights from revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
|
||||
explicitly. The code verifies exact sizes and SHA-256 of weights/config/tokenizer
|
||||
against `model_files.py` before importing model libraries. Protocol stdout is
|
||||
isolated at the file-descriptor level from third-party diagnostics. Thread limits
|
||||
cover Torch interop/native pools and CUDA is not selected. Offline environment
|
||||
flags do not prove every third-party internal is unable to access a network.
|
||||
Runtime/build/tests perform no downloads; the separate explicit setup utility
|
||||
`scripts/fetch-model.py` can provision the public pinned weights.
|
||||
|
||||
**Licensing blocker:** the observed kestrel-kernels 0.7.0 license requires a
|
||||
separate M87 Labs agreement; do not treat wheel availability as permission for use
|
||||
or bundling. See [third-party notes](third-party.md). No public runtime bundle has
|
||||
been released. Limited ARM64 measurements are in the [POC record](evidence/poc-cpu-overlay-2026-09-24.md),
|
||||
not a claim of complete headset acceptance.
|
||||
|
||||
Hardware-free tests run through CTest, including fake-child cancellation, short
|
||||
injected warmup/request deadlines, duplicate/stale replies, malformed frames,
|
||||
missing/hash-mismatched model files and symlink refusal. Default production
|
||||
deadlines remain 120/60 seconds. Python tests never import actual model libraries,
|
||||
record a microphone, download assets or initialize OpenVR.
|
||||
Reference in new issue
Block a user