diff --git a/.gitignore b/.gitignore index c77e722..dd3d006 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,6 @@ __pycache__/ .env.* !.env.example .DS_Store + +# Local archive of dated device evidence and old provenance notes (not published) +/docs/archive/ diff --git a/AGENTS.md b/AGENTS.md index 9dd6d05..4e5e59b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,9 @@ -# Frame Dictation development +# FrameYap development This is an independent project, not a plugin for another application. -Read README.md and docs/design.md before implementation. Dated evidence records -past observations; it never proves current device availability or grants a live run. +Read README.md and docs/design.md before implementation. Dated device evidence is kept +locally in the untracked `docs/archive/`; it records past observations and never proves +current device availability or grants a live run. - Keep dependencies explicit and small. Do not add a dependency/submodule/symlink to an unrelated application's build tree, assets or Python environment. diff --git a/README.md b/README.md index 974d027..20af4c9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # FrameYap -Standalone, on-device voice typing POC for Steam Frame. **MIT licensed.** +Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress). Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture, persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope, @@ -11,11 +11,11 @@ server, cloud fallback or unrelated application dependency. Gamescope discovery and native-only installation have been exercised on Frame. Live microphone → reviewed text → real target delivery is **not yet accepted**. -**Runtime licensing remains unresolved.** Redux weights are CC-BY-4.0, but the -observed Kestrel kernel runtime requires separate permission. We are checking with -the vendor. Current native-only packages do **not** bundle or download that -runtime; an independently authorized Python environment must be supplied. -See [third-party notes](docs/third-party.md). No GitHub release is published yet. +**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the +`moondream` Python package (its Kestrel runtime states that local inference is free +and needs no API key). The build and tests never download it; the installer or you +install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md). +No GitHub release is published yet. ## Controls @@ -89,7 +89,7 @@ ctest --test-dir build --output-on-failure Default build has no hardware backends. It never downloads packages/models or initializes SteamVR, a microphone or input injection. Native dependencies and -explicit launch/check commands are documented in [the POC guide](docs/poc.md). +explicit launch/check commands are documented in [the build guide](docs/build.md). `--version` uses an ISO-like UTC build timestamp (with Git hash when available), not a numbered release; use that same tag when packaging the binary. @@ -110,30 +110,28 @@ Autolaunch is opt-in. An installed desktop entry can be selected manually as a non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the panel on one Frame; registration alone did not show an entry in the first checked dashboard menu. See [packaging and lifecycle](docs/packaging.md). -For a native-only install, menu-driven inference needs a separately authorized -Python runtime and pinned model. Configure their absolute paths in +For a native-only install, menu-driven inference needs a Python runtime you +provide and the pinned model. Configure their absolute paths in `~/.config/frameyap/paths.conf` as described in the packaging guide; they are never fetched or bundled implicitly. A pinned GitHub one-command route is implemented in `install.sh --version TAG`, but **do not advertise or run it as a working public installation until a vetted release exists**. Native-only artifacts support the overlay/checks without an -end-user compiler; full bundled-ASR distribution awaits runtime permission. +end-user compiler; a bundled-ASR distribution is not yet offered. ## Project map -- [POC guide](docs/poc.md): implemented boundaries, build, controls, explicit tests. +- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests. - [Design](docs/design.md): full target design; some features remain proposed. - [Installer design](docs/install-design.md) and [packaging](docs/packaging.md). -- [Current POC observations](docs/evidence/poc-cpu-overlay-2026-09-24.md): measured - CPU behavior and native installation checks, with acceptance limits. -- [Resize / Auto Insert deployment](docs/evidence/auto-insert-deployment-2026-09-24.md): - installed ARM64 version and narrow owned-target fixture; live speech-driven - Auto Insert and physical resize acceptance remain open. -- [Earlier API evidence](docs/evidence/frameyap-apis-2026-09-24.md) and - [provenance](docs/provenance.md): historical investigation, not live authority. - [Overlay](docs/overlay.md), [worker protocol](docs/worker.md), [dependency/license inventory](docs/third-party.md). +- [TODO](TODO.md): release plan and open work items. + +Dated device-test records are kept locally (untracked) and are not authority for +current device availability. Live microphone → reviewed text → target delivery has +not been formally accepted; see Status above. No recordings, transcripts, private logs, model weights or runtime binaries are committed. The worker boundary is intentionally small for forks experimenting diff --git a/TODO.md b/TODO.md index 500c246..f1c03d3 100644 --- a/TODO.md +++ b/TODO.md @@ -1,19 +1,182 @@ # FrameYap TODO -- [ ] Resolve the Redux inference runtime license before further inference or a - bundled release. Obtain written permission covering use and redistribution of - M87 Labs Kestrel kernels (including the CPU payload), or scope an independently - licensed runtime for the same weights. Inventory the other runtime dependencies' - notices/licenses before publishing. The CC-BY-4.0 model weights do not grant - permission to use or redistribute the proprietary runtime. See - [third-party notes](docs/third-party.md). -- [x] Validate a basic menu launch on Frame: OpenVR registration did **not** show - FrameYap in the first checked dashboard menu; the user found FrameYap in Steam's - **Non-Steam** section and reported that selecting it showed a panel and Quit - worked. The server log showed a `--run` overlay connection followed by exit; - no FrameYap process remained. The installed native-only package had no bundled - runtime or model. This does not validate transcription, recording, text delivery, - cold startup, or which shortcut discovery mechanism Steam used. -- [ ] Verify the exact missing-runtime panel message and Steam's shortcut - persistence across a normal restart (without restarting sessions just for the - test). Confirm no runtime/model environment override before future live checks. +Compiled from the 2026-09-24 holistic review, with owner decisions applied (see +"Decisions" at the end). Items are sized to hand off individually. Size: S ≈ hours, +M ≈ a day, L ≈ multi-day. + +Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit +downloads, no automatic Enter, small verified commits, docs must state implemented +vs proposed behavior honestly. + +## A. First release (v0.1) blockers + +- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README + states local inference is free; blocker text removed from README/docs/scripts and + replaced with a neutral dependency note in `docs/third-party.md`. +- [x] **A2. Remove references to the unrelated app (kouseki).** (S) + Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a + final grep. Keep the Inconsolata/OFL attribution and cite the + upstream font source (googlefonts/Inconsolata) instead. + *Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains. +- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" → + "FrameYap"). (S) +- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M) + Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL + option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt + archive that includes any of them. +- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README, + `scripts/stage-native-poc.py`, `CMakeLists.txt` messages, + installer strings. v0.1 is a first small release. +- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install, + controls **table** (button → action), then a "Status / not yet validated" section. + Today it reads as a lab notebook and Controls is a wall of text. +- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved + to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`. + Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and + hedging language before v0.1. + +## B. Correctness / robustness + +- [ ] **B1. A malformed transcript must not kill the worker.** (S) + `src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control + characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and + `session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a + request-level error (like the `E` path: keep the worker, show "transcription + failed", allow retry). Add a test with a control-character reply asserting the + worker stays ready. +- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes + "moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a + worker-supplied message code. Prerequisite for C1. +- [ ] **B3. "Close mic when idle" setting, default OFF.** (M) + Default keeps the mic open while Ready (opening/closing per PTT causes an audio + spike on the physical hardware, and it avoids first-syllable clipping). The + setting closes it between clips for people who don't want a live device. Document + the tradeoff (spike/latency) next to the toggle, in Settings and in the docs. + Persist in `config.json`; add to the panel Settings tab. +- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L) + `run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no + tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`) + so PTT, cancel, quick phrases, auto-insert and error transitions are testable + without hardware. B1 is the first regression test. + +## C. Backends and model management + +- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L) + Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper, + sherpa-onnx Parakeet, …) as separate worker executables speaking the existing + `Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name, + launcher, pinned model files + hashes + attribution, license text, CPU/GPU + requirements) so nothing is hardcoded in C++ or `model_files.py`. + *Done when:* Redux is expressed as a manifest, and a second backend can be added + without touching `worker.cpp`/`runtime.cpp`. +- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4. + Settings page listing backends/models with state (not installed / installed and + verified / loading / ready / failed), the active one marked, and selection that + restarts the worker. Target user is non-technical: an **Install** button on the + panel runs the installer's machine-readable mode (D2) as a child process and shows + progress/errors in the panel, so nobody needs a terminal. The click is the consent; + there are still no implicit or background downloads, and the panel states what + will be downloaded and how large it is before it starts. Persist the choice in + `config.json`. +- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID` + (offline, hash-verifies installed files) so the UI and installer share one + implementation. + +## D. Installer + +- [ ] **D1. Installer with a binary-or-source choice.** (L) + `install.sh` offers *prebuilt archive* (checksummed) or *build from source* + (checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then + continues automatically through install after the user's choices. Retain rollback, + idempotency, no sudo, no Steam AppID, opt-in autolaunch. +- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M) + Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`, + `--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`, + `--json` output) so a model/agent can run it non-interactively; interactive + prompts only run on a TTY and print the equivalent flags they chose. Exit codes + and messages must be machine-readable. +- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the + release checklist in `docs/packaging.md`; do not advertise the one-command route + until the archive and its checksum are actually published and tested from a clean + account. + +## E. Naming and versioning + +- [x] **E1. Name: keep "FrameYap" for v0.1.** Frame (the hardware) + yap (speech) + says what it is; no rename churn before the first tag. If a hardware-neutral + project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame + front end), decide it before the cross-window work in G, not now. +- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text + + space) and **Type + Enter**; align overlay buttons, `--check-controls` output, + README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only + if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings + explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux" + vs `moondream` naming once in `docs/worker.md`. +- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S) + e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts + correctly, keeps the build date visible, URL/filename-safe. The version field is + just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted + changes) suffixes were only for telling developer builds apart, so they move out + of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev + build adds a second line like `git abc12345 (uncommitted changes)`. Release + archives are built from a clean tag, so users never see it. Update the CMake + version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`, + installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the + timestamp. Tag releases `v0.1.`. + +## F. Code structure (non-urgent) + +- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of + inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a + table-driven option parser. +- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag + state, save-failure flags, counters, pose caches): separate drag, persistence and + diagnostics. + +## G. Other windows (post-release) + +Text delivery already works into a WezTerm window on Frame (owner-tested; this is +not the recorded live acceptance in H). Deeper integration of other windows with +this app is future design and out of scope for v0.1. + +- [ ] **G1. Validate browser text fields via the Gamescope input path.** (M) + Highest-priority target. Record which fields accept Type / Type + Enter (plain + inputs, textareas, rich editors, password fields should be expected to differ) and + document the results honestly. +- [ ] **G2. Later, if needed:** a KDE desktop-mode backend behind `DeliveryLease`, + and a uinput backend as a last resort. Not scheduled; uinput needs `/dev/uinput` + access and types with no focus check, which conflicts with the project's + no-sudo/udev rule and per-window authorization. + +## H. Carried over from the earlier TODO (hardware validation) + +- [x] Basic menu launch on Frame: the user found FrameYap in Steam's **Non-Steam** + section; the panel showed and Quit worked. This does not validate transcription, + recording, text delivery, cold startup, or which shortcut discovery mechanism Steam + used. +- [ ] Verify the exact missing-runtime panel message and Steam's shortcut persistence + across a normal restart (without restarting sessions just for the test). Confirm no + runtime/model environment override before future live checks. +- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery, + Auto Insert with speech, physical resize. + +--- + +## Decisions + +- Redux runtime: treated as usable for local inference per upstream's Kestrel + README; not bundled in our archives (A1 done). +- Multiple backends + a model chooser UI: wanted (C1–C3). The panel can trigger the + install on an explicit click, for non-technical users. +- Name: keep FrameYap. v0.1 is a first small release, not a POC. +- Close-mic-when-idle: setting, default off (B3). +- Labels: Type / Type + Enter (E2). +- Installer: binary or source, continues automatically after choices, fully flag- + driven for agent use (D1–D2). +- Version: `MAJOR.MINOR.YYYYMMDDHHMM`, git hash only in dev-build `--version` output + (E3). +- Other windows: browser text fields first (G1); deeper integration is future design. + +## Open questions + +None currently blocking. diff --git a/docs/poc.md b/docs/build.md similarity index 88% rename from docs/poc.md rename to docs/build.md index 21444dd..0c13fc8 100644 --- a/docs/poc.md +++ b/docs/build.md @@ -1,4 +1,4 @@ -# FrameYap POC: implementation and validation +# Build, scope and validation This is a standalone native application, not a plugin. Default builds/tests never initialize OpenVR, open a microphone, run ASR, download files or inject input. @@ -67,17 +67,12 @@ This is a compact prototype panel, not yet the proposed polished miniature statu chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact and target application compatibility require further headset work. -## Critical runtime licensing boundary +## Inference runtime -Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. -The installed **kestrel-kernels 0.7.0** license is different: proprietary, requiring -an M87 Labs written agreement for use; copying/redistribution depends on that -agreement. PyPI availability is not permission. See [third-party notes](third-party.md). - -Do not publish a bundled Redux runtime, imply a public release is ready, or rerun -inference while applicable permission is unresolved. Our adapter is implemented; -that does not resolve distribution rights. An alternative runtime would be a -separately scoped and independently licensed implementation—not a silent model swap. +Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference +runs through the `moondream` Python package and its Kestrel runtime, which you +install in your own environment; builds/tests never fetch it. See +[third-party notes](third-party.md). ## Developer native build @@ -126,7 +121,7 @@ an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or `~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files are edited and no runtime/session restart is performed. -## Deliberate launch, once runtime permission is settled +## Deliberate launch Explicit setup downloads only the pinned, openly licensed model: @@ -137,12 +132,12 @@ python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model" `fetch-model.py` is idempotent for matching hashes, rejects existing mismatched files, and is never invoked by build/tests or first utterance. -Provide your independently authorized CPU Python environment (moondream 2.4.0, +Provide your own CPU Python environment (moondream 2.4.0, kestrel 0.8.0) and local weights: ```sh ./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \ - --python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \ + --python /path/to/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \ --model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head ``` @@ -161,5 +156,4 @@ sending existing users' audio to a service. `scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV; no recording, device input or automatic download. `--show-text` is a separate explicit disclosure of that fixture's transcript. Use only with an authorized -runtime. See the dated [POC record](evidence/poc-cpu-overlay-2026-09-24.md) for -measurements and their limits; they are not headset/performance acceptance. +runtime. Benchmark numbers from earlier trials are not headset/performance acceptance. diff --git a/docs/design.md b/docs/design.md index e5f7dde..89a3149 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,14 +1,12 @@ # FrameYap native dictation overlay — proposal -Standalone project design; see -[provenance](provenance.md) and [historical Frame probes](evidence/frameyap-apis-2026-09-24.md). -This document is the full target design, not a blanket implementation claim. -The native POC now implements overlay/actions, bounded SDL3 capture, a persistent +Standalone project design. This document is the full target design, not a blanket +implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent Redux adapter, review-first Gamescope insertion and a user-local archive installer. Opt-in conservative Xwayland focus tracking is implemented locally but not yet accepted for live automatic typing on Frame. Polished status-chip UX and full hardware -acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md) -and [runtime licensing boundary](third-party.md). +acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md). +Future work is tracked in [TODO.md](../TODO.md). ## Recommendation @@ -105,11 +103,9 @@ not an application-rendered hand-pose animation loop. Do not promise a particula GPU cost until measured. A scene renderer's canvas/MSDF resources are not an OpenVR overlay backend and -are not imported here. The Inconsolata TTF used by kouseki is independently -bundled under its retained OFL; the panel renderer is original FrameYap code. -The neon HUD frame is a visual reference, not an engine dependency. Further -source/asset reuse requires an explicit license-reviewed extraction, never a -runtime path into the engine checkout. +are not imported here. The bundled Inconsolata TTF is under its retained OFL; the panel renderer is +original FrameYap code. Further source/asset reuse requires an explicit +license-reviewed extraction, never a runtime path into another project's checkout. ### Controller bindings @@ -261,7 +257,7 @@ processing timeout. No shell commands in IPC and no input authority in the worke no CUDA device/runtime assumption. Do not copy the desktop's x86 venv. - Weights are ~178 MB; Torch, kernels, temporary conversion and activations mean install size/RSS will be larger. Measure cold load, peak RSS and package size. - Review runtime redistribution licensing separately from model attribution. + Keep runtime notices separate from model attribution. Microphone access does not mute VRChat or any other social-voice app. Shared PipeWire capture may let both hear the same utterance; the overlay must not claim @@ -312,4 +308,4 @@ Hardware-free tests should cover state transitions, bounded PCM/transcripts, worker framing/timeout/cancellation, duplicate/stale replies and focus generations with fakes. Separate opt-in Frame checks cover protocol availability, owned-window insertion, microphones, inference and overlays. No fixture establishes physical -headset acceptance. See [provenance](provenance.md) for historical source anchors. +headset acceptance. diff --git a/docs/evidence/auto-insert-deployment-2026-09-24.md b/docs/evidence/auto-insert-deployment-2026-09-24.md deleted file mode 100644 index 668514e..0000000 --- a/docs/evidence/auto-insert-deployment-2026-09-24.md +++ /dev/null @@ -1,41 +0,0 @@ -# Anchored resize / Auto Insert deployment — 2026-09-24 - -Historical observation from an explicitly coordinated Frame-only install and -owned-target fixture. It does not authorize further microphone tests or prove -actual speech-to-text Auto Insert, grip usability, arbitrary app delivery, or -human headset acceptance. - -- Source milestones: `19a3db7` adds session-only corner-anchored resize; - `c5e925c` adds default-off, fail-closed Xwayland Auto Insert. -- Local default/UI/native hardware-free suites passed: 13/13, 14/14, 18/18. - Native Xvfb focus-loss test also passed 12 repeated runs. ARM64 native build - from the `c5e925c` Git archive passed hardware-free CTest 18/18 on Frame. - No normal test initialized OpenVR, recorded audio, or injected input. -- Native-only, external-runtime archive version - `2026-09-24T201841Z-gc5e925cb` was staged with explicit licensed SDL/OpenVR - files. SHA-256 archive: - `c6bef87ec86fc0903247be714aec9c5784c3fa5ac788858dad9f50f1958d6f22`. - The pre-install checksum check succeeded; system libxcb was available and no - staged binary dependencies were unresolved. -- No FrameYap process was running when the installer checked. The idempotent - user-local installer selected the new version, retained previous - `2026-09-24T191513Z-g3c03f88f`, and backed up the existing config. Installed - `--version` matched the archive version; installed/staged binary SHA-256 both - `973fd119701093470b2ee3560df7e8049d968e1bab49f63f5242dabf055aa4ec`. - Auto Insert remained `false`; no FrameYap process was left running. -- A separate temporary ARM64 test helper used the new `FocusGuard` and existing - Gamescope IME `TextInput` against one project-owned Xterm receiver on display - `:0`. It checked the expected window ID against X keyboard focus, X active - window and Gamescope focused-window both before arming and after acquiring - the IME lease. The helper queued the literal ASCII fixture `fy-safe-probe` - without Enter; the private, disposable terminal receiver read **exactly 13 - bytes**, matching the fixture. Both temporary processes exited. No microphone, - ASR model, arbitrary destination, speech transcript, or app UI Auto Insert - workflow was exercised by that helper. - -Compositor processing and the owned terminal's exact bytes support this narrow -fixture-delivery claim. The full microphone → worker → focus guard → Auto Insert -flow remains untested on Frame, as do physical resize interactions, input during -focus changes in actual games, and native Wayland targets. X focus checks are -not atomic with delivery; review remains the fallback. No public release or -bundled ASR runtime was published. diff --git a/docs/evidence/bindings-deployment-2026-09-24.md b/docs/evidence/bindings-deployment-2026-09-24.md deleted file mode 100644 index a0aff72..0000000 --- a/docs/evidence/bindings-deployment-2026-09-24.md +++ /dev/null @@ -1,58 +0,0 @@ -# Frame bindings and explicit Enter deployment — 2026-09-24 - -Historical evidence, not current device status or permission for further live tests. - -## Implemented and checked - -User approved right X hold-to-talk, B Cancel, A Insert + space, Y pending Insert -+ Enter; existing grip gestures remain. A Bindings tab shows OpenVR origin names -and requests the runtime binding editor. Insert ensures a trailing ASCII space; -explicit Enter consumes pending review, queues its text, releases that IME lease, -then acquires a new lease for Submit. Failed text never proceeds to Submit; an -uncertain send is not retried. No transcription completion auto-submits. - -Read-only inspection of the installed Frame controller profile confirmed A/B/X/Y -on the right and a D-pad on the left. It references runtime-owned left/right SVG -diagrams for SteamVR's editor. FrameYap does not redistribute those diagrams or -claim a generic OpenVR button-glyph API. Editor display is not yet human-accepted. - -Local default CTest: 13/13. Local native hardware-free CTest: 16/16. Synthetic -Bindings canvas was inspected for layout. These do not prove actual input delivery. - -## Native install - -Committed application source: `7a3a5451`, including wrist geometry/config work -`569ba13` and `e61bf73`. A Git source archive (excluding unrelated dirty workspace -files) built on Frame with existing standalone dependencies; ARM64 native -hardware-free CTest passed 16/16. No runtime/model dependencies were downloaded. - -Installed and verified version: `2026-09-24T184918Z-g7a3a5451`. -Native-only archive SHA-256: -`9fa7991298731b27fc6dbf6d805167ee006e0b8ab93c2b859e2fad61007832a5`. -`current` selected this version and `previous` retained -`2026-09-24T183121Z-g69f5de5a`. Installer checked the supplied checksum. Installed -`--version` matched; `ldd` resolved bundled SDL/OpenVR and system dependencies. -The authorized runtime/model paths configuration hash remained unchanged. - -The installer preserved existing disabled action values. With the user's approval -of the new controls, a separate exact-byte-backed-up config update enabled the -X/B/A/Y mappings and returned experimental input priority to normal. Lasers anytime -was already off. Saved `right-wrist` mount was retained; wrist config now uses -0.30 m width, zero roll and (0, 0.18, 0.089) offset. - -A running FrameYap process appeared during configuration; only its exact verified -installed executable PID was terminated gracefully under the restart approval. -No SteamVR, SSH or user-session process was stopped. The restarted process's -`/proc/PID/exe` matched the new installed version. Startup reported normal priority, -Lasers anytime off and panel shown. This is API/process evidence, not headset -visibility or delivered-input acceptance. A normal app launch warms its configured -worker and opens/discards idle microphone samples; no recording or input-delivery -test was initiated by the assistant. - -## Open acceptance - -The wearer then reported **"wrist is backwards"**. Wrist orientation is therefore -not accepted; whether this means inverted text or a panel facing away was awaiting -clarification at this record. Do not count passing pose tests as comfort/orientation -acceptance. Actual B/A/Y delivery, text-plus-Enter ordering at a real target, and -SteamVR binding-editor behavior remain human-led checks. diff --git a/docs/evidence/debug-wrist-deployment-2026-09-24.md b/docs/evidence/debug-wrist-deployment-2026-09-24.md deleted file mode 100644 index 071474e..0000000 --- a/docs/evidence/debug-wrist-deployment-2026-09-24.md +++ /dev/null @@ -1,50 +0,0 @@ -# Debug diagnostics and right-wrist correction — 2026-09-24 - -Historical observation, not current availability or permission to run tests. - -The wearer clarified that the right-wrist panel faced away (its back was visible). -Commit `29f3729d` reverses right-wrist panel-right and panel-front, preserving -panel-up, center and size. Offline pose tests check both hands, orthonormality -and positive determinant over several rolls; this is not physical acceptance. - -The wearer also reported repeated generic transcription failures. The retained -app log contained startup status only. Inspection found that the Python worker -replaced every request exception with `transcription failed`, and both the native -parent and Python suppressed stderr. No underlying exception had been retained; -the root cause therefore remained unknown. The user declined a public-clip -inference trial and chose to retry speech themselves after diagnostics were added. - -Commit `6592e1af` adds default-safe stage/category errors and explicitly opt-in -advanced debugging through config/Settings. Full worker stdout/stderr, tracebacks -and transcripts may appear in private bounded logs when enabled; no raw clip -archive is created. The native receiver allowlists error labels before showing -or logging the non-debug error. Settings changes restart the owned worker and -discard current work; defaults remain off. See [diagnostic policy](../worker.md#advanced-debugging). - -## Checks and deployment - -- Local default CTest: 13/13; local native hardware-free CTest: 16/16. -- Inspected a synthetic Settings canvas containing the toggle and privacy warning. -- ARM64 native build from Git archive `6592e1af`: hardware-free CTest 16/16. - Includes fake-child stderr/protocol isolation, log bounds/rotation/permissions, - unsafe path refusal, opt-in/off behavior and safe-error privacy tests. -- Installed version: `2026-09-24T190806Z-g6592e1af`. -- Native-only archive SHA-256: - `eac25ae6e36970301e5cb67614eaa4053d20b79711397b462e2caf1359c09709`. -- Checksum-verified installer selected the version above; installed `--version` - and the restarted process's executable path matched. Previous version retained: - `2026-09-24T184918Z-g7a3a5451`. The intermediate wrist-only archive was not installed. -- Runtime/model paths configuration hash unchanged. Installer backed up the - previous config, added `advanced_debug: false`, and retained normal priority, - approved X/B/A/Y mappings and the wrist size/offset settings. -- The saved mount was now `left-wrist` (changed since the earlier right-wrist - observation); installation preserved it rather than choosing for the wearer. -- No FrameYap process was present immediately before this install. Relaunched - only FrameYap. No SteamVR/SSH/user-session process was stopped. Startup reported - normal priority and Lasers anytime off; this does not establish panel visibility. - -No public-fixture inference, assistant-triggered microphone recording or input -injection was performed. Normal authorized app startup warms the configured worker -and opens/discards idle microphone samples. Advanced logging was left **off** for -the wearer to enable. Actual failure diagnosis, Settings-toggle behavior on Frame, -and physical wrist acceptance still require the user's next trial. diff --git a/docs/evidence/experimental-input-priority-2026-09-24.md b/docs/evidence/experimental-input-priority-2026-09-24.md deleted file mode 100644 index 297c7e2..0000000 --- a/docs/evidence/experimental-input-priority-2026-09-24.md +++ /dev/null @@ -1,67 +0,0 @@ -# Experimental overlay input priority — 2026-09-24 - -Source commit: `69f5de5a`. -Installed native ARM64 version: `2026-09-24T183121Z-g69f5de5a`. - -The user confirmed the Vulkan flicker fix, reported controller actions becoming -unavailable in system laser/dashboard interaction states, and agreed to try -OpenVR's experimental priority mechanism. The investigation concerns action -delivery across modes, independent of any specific physical button. - -## Implementation and preparation - -FrameYap config `input_priority` accepts `normal` (default) or `experimental`. -Experimental requests `k_nActionSetOverlayGlobalPriorityMin` (`0x01000000`) for -the existing action set through `UpdateActionState`, including while system -laser mode or the dashboard is active. The request covers the sources bound to -FrameYap actions; bindings and action paths are unchanged. The installer -preserves the selection and backs up invalid config before repair. - -SteamVR separately permits global input priority through its Developer setting. -The current Frame's saved `steamvr.globalActionSetPriority` was already true; -its runtime default was false. This was a targeted file read, not an effective -runtime API query. FrameYap now reads this permission through `IVRSettings` and -reports it separately from its priority request at startup. It does not write -the SteamVR setting. - -The controls-only diagnostic reports all six actions' activity, press state and -pose/role acceptance, alongside dashboard visibility, our Lasers anytime flag, -`IsInputAvailable`, panel visibility and the application focus gate. This can -distinguish runtime action inactivity from application rejection. The laser flag -records our request; it is not a detector for all system laser activation. - -## Verification and deployment - -- Local default build and CTest: 13/13 passed. -- Local native build and hardware-free CTest: 16/16 passed. The fake Wayland - test used sandbox escalation to bind its local Unix socket. -- Frame ARM64 Release build from a checksum-verified archive of the source - commit above: 16/16 hardware-free CTest checks passed. -- The package was installed through the existing installer. Installed - `--version` matched and its executable hash matched the staged executable. -- FrameYap's config was set to `experimental` under the application lock, with - an exact backup; all other config values were verified unchanged by that edit. -- No OpenVR probe, microphone capture, inference or text delivery was started. - No application, SSH or session process was stopped. - -Native archive SHA-256: -`40a9445427469286e8997563bc5598ace7769ba4c13b1a9290911c821d0e419c`. -Installed executable SHA-256: -`da842efcad926660efc66c03d843592573380b30e563e797df2944a0d6d77da7`. - -The prior Vulkan version `2026-09-24T181314Z-g4c043c99` is retained. To end the -priority experiment on the new build, set `input_priority` to `normal` and -relaunch. To roll back to the older binary, first restore the pre-upgrade config -backup `config.json.backup-sih_efaa`: that binary predates the new config key. -The separate `config.json.backup-priority-2026-09-24T183121Z-g69f5de5a` preserves -the upgraded config before selecting experimental priority. - -## Acceptance boundary - -The user was told the new build is ready for headset testing. Compare the same -bindings with the dashboard open/closed and Lasers anytime on/off; check both -pointer interaction and controller press/release, including mode transitions. -Higher priority may consume input used by a scene or the dashboard. This record -does not claim that actions now arrive in every state or that simultaneous -dashboard interaction is accepted. Dated deployment evidence does not establish -future device availability or authorize future live runs. diff --git a/docs/evidence/focus-probe-2026-09-24.md b/docs/evidence/focus-probe-2026-09-24.md deleted file mode 100644 index 00c6ed6..0000000 --- a/docs/evidence/focus-probe-2026-09-24.md +++ /dev/null @@ -1,29 +0,0 @@ -# Guided Frame focus probe — 2026-09-24 - -Historical observation, not authorization for later live input or proof of target safety. -The wearer consented to an opt-in, disposable-target focus trial. Two project-owned -`xmessage` windows were created on Xwayland `:0` for 65 seconds, then closed by -the owning finite command. No microphone, transcript, input injection or SteamVR -session cleanup was used. Other SSH/user processes were left untouched. - -An earlier read-only snapshot showed `/tmp/.X11-unix/X0` and `X1`, Gamescope -socket `gamescope-0`, and `_NET_ACTIVE_WINDOW` matching -`GAMESCOPE_FOCUSED_WINDOW` on `:0`. Brief snapshots of an owned test window -also showed disagreement between these root properties, so either property -alone is insufficient to authorize automatic typing. - -The wearer selected A/B and reported doing several focus changes. Window A was -`0x3e00022`, B was `0x4200022` in that run. A 120 ms sampled observer saw -X keyboard focus, `_NET_ACTIVE_WINDOW` and `GAMESCOPE_FOCUSED_WINDOW` agree at -A, change to B, then return to A several times. A transition to a third window -`0x3c00003` was also observed. At other moments the X keyboard-focus/active -window IDs changed while Gamescope's focused-window property still named A. -The root property is a window ID encoded as CARDINAL, not a generation token. - -These samples establish *observable correlation for these two owned Xwayland -windows*, not continuous seat identity across all targets, a focus-loss event -stream, transcript quality, delivered input, native Wayland coverage or headset -acceptance. The offline fail-closed observer subscribes to X property changes -and focus-out on the exact armed window; its own live behavior and actual IME -delivery still require a separate disposable-target validation. There is still -a non-atomic gap between final focus check and compositor input processing. diff --git a/docs/evidence/frameyap-apis-2026-09-24.md b/docs/evidence/frameyap-apis-2026-09-24.md deleted file mode 100644 index 63eaa44..0000000 --- a/docs/evidence/frameyap-apis-2026-09-24.md +++ /dev/null @@ -1,44 +0,0 @@ -# Historical Steam Frame API probes — 2026-09-24 - -These observations were made on one device and SteamOS build, not rerun as part -of this repository's scaffold. They do not prove current availability or headset -acceptance. No probe code, user audio, credentials or runtime binaries are shipped. -See the [design](../design.md) for proposed implementation and safety gates. - -## Environment - -SteamOS 0.4.0 (VR variant), AArch64, glibc 2.39, Python 3.12.3; -Gamescope 3.16.28-2. Linux ARM64 package availability is not evidence of -inference speed or compatibility with this device. - -## Observations and limits - -| Surface | Historical observation | Limit | -| --- | --- | --- | -| OpenVR | Native `VRApplication_Overlay` initialization succeeded; `IVROverlay_028`, `IVRInput_011`, `IVRSystem_026` and `IVRApplications_008` were accepted. | No rendered overlay, binding or autolaunch test. | -| Gamescope IME | `gamescope_input_method_manager` v3 advertised; v2 binding accepted and returned `done(serial=1)`. | Discovery is not delivered input. | -| Unicode delivery | A separate disposable X11/XIM receiver on the device accepted an exact mixed-script Unicode fixture via `set_string` and `commit` after checking focus on its owned window. | One Xwayland receiver, not general games, native Wayland or PC-streamed targets. No Enter was sent. | -| libei | Sender connected; seat advertised KEYBOARD, not TEXT. | No bound device or delivered key test. | -| XTEST / uinput | XTEST advertised; `/dev/uinput` was openable by the test account. | No injected XTEST key or virtual device. | -| Audio | PipeWire input device enumerated. | No microphone recording or model inference in this probe. | -| Portal | Existing portal introspection exposed no RemoteDesktop, InputCapture, Clipboard, ScreenCast or GlobalShortcuts interface. | Not a guarantee for future OS releases. | - -The Xwayland receiver was destroyed after the test. No packages, services or -Steam settings were changed. No microphone audio was recorded. This is a -historical API probe, not a live availability check or install smoke test. - -## Implementation cautions - -Gamescope's [input-method protocol](https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml) -is private and version-sensitive; see its [implementation](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/ime.cpp). -The serial is not a verified focus-generation guard; a roundtrip is not a text -consumption acknowledgement. The inspected [`gamescope-type` example](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/Apps/gamescope_type.c) -interprets newline as Submit: do not use it as a transcript pipe. -Generic virtual-keyboard and data-control Wayland globals were not advertised -in this probe. Focus tracking and Steam keyboard coexistence need separate tests. - -The proposed Redux model is pinned to revision -`fad622f25f303105c20d70e201bcc477c88b620c`; the weight file was -177,774,490 bytes in an earlier local inspection. ARM64 Python 3.12 wheels -appeared available, but no native model load or inference was performed on the -Frame. Runtime redistribution terms require separate review. diff --git a/docs/evidence/free-grab-lock-deployment-2026-09-24.md b/docs/evidence/free-grab-lock-deployment-2026-09-24.md deleted file mode 100644 index 7c171a0..0000000 --- a/docs/evidence/free-grab-lock-deployment-2026-09-24.md +++ /dev/null @@ -1,31 +0,0 @@ -# Free controller grab / layout lock deployment — 2026-09-24 - -Historical deployment evidence; physical interaction remains user-led acceptance. -The user authorized continuing after the [raw-overlay incident analysis](raw-overlay-teardown-crash-2026-09-24.md), using controller/visual testing instead of another raw-overlay probe. - -- Source: `0d4cfbfdb74aaaa8e56be844122ac51eeb8fe365`. Includes full controller-relative - position/rotation, saved `lock_layout`, corrected top-left mask coordinates, and - earlier gradient, clock/date and wrist-fade commits. -- Local default suite: 15/15 passed. Local native suite: 20/20 passed. - Native ARM64 Release build on Frame: 20/20 hardware-free tests passed. -- Exact source archive SHA-256 (verified before extraction): - `57aa17045ddfee0fc8f6f1bb2bd4cbf19b082cf62321edaded2fb40ebd19e9d2`. -- Installed version: `2026-09-24T210207Z-g0d4cfbfd`. - Native-only external-runtime package SHA-256: - `85574b1645b68e57a609c3e46b8068613ac5bc64a25fbc8b7ed7c1e4bcb17f9a`. - Installed/staged binary SHA-256 both matched: - `4d256feb1b23b649e4720fb3ca11eab786229a445c22d4fc31b209ffe7e6b187`. -- No existing FrameYap process was running at install. The managed installer - retained rollback and backed up the existing config before filling missing - preferences. No ASR runtime/model was bundled or downloaded. -- Installed `--version` and the running executable path were verified after - launching the normal Vulkan runtime (PID 78418 at that check). Startup reported - dashboard visible, normal input priority, Lasers anytime off and panel-shown=N. - The latter is not visual acceptance; the selected wrist's angle/tracking gate - can keep it hidden. The user was asked to face the wrist toward them if needed. -- No raw-overlay probe, deliberate recording or synthesized input was run during - this deployment. No SteamVR, Gamescope, SSH or terminal session was restarted. - -Requested human checks: corner-bracket hit/resize; free depth/rotation and stable -release; Settings lock hides/disables both handles and unlock restores them. -At this record's creation those checks were requested, not yet reported passed. diff --git a/docs/evidence/grab-scale-deployment-2026-09-24.md b/docs/evidence/grab-scale-deployment-2026-09-24.md deleted file mode 100644 index 82fdef8..0000000 --- a/docs/evidence/grab-scale-deployment-2026-09-24.md +++ /dev/null @@ -1,44 +0,0 @@ -# External grab/scale handles — 2026-09-24 - -Historical observation from an explicitly authorized Frame install/relaunch. -This record is not permission for later hardware runs and does not establish -physical drag/release behavior or human headset acceptance. - -- Source commit: `33533ba8a2c85ccef3a388da4c2155e7dbbc14e0`. -- The user's Frame screenshot showed a thin grab underline below the terminal - and an external lower-right corner bracket. FrameYap independently draws that - layout in transparent RGBA margins; no Steam private UI code/assets are bundled. - The screenshot remains outside Git. -- Local default, FreeType UI and native hardware-free suites passed 14/14, - 15/15 and 19/19 respectively. A synthetic panel preview was visually inspected; - renderer tests check transparent gaps, opaque handle centers, antialiased alpha, - cursor ownership and suppression of stale control approvals. -- Controller-ray math tests cover stationary stability, scaling on independent - axes, rotated/relative geometry, out-of-bounds intersections and invalid rays. - The original per-event resize feedback path was removed. Grab currently - translates in the panel plane, not depth or orientation. -- Exact source archive SHA-256: - `a8b8d4387de1fdd5cb1031e8905a6d6766e89616ceff3b62f902ea192a70e07b`. - The checksum was verified before extraction on Frame. Native ARM64 Release - build and hardware-free CTest passed 19/19; no test initialized OpenVR, - recorded audio or injected input. -- Installed version: `2026-09-24T203812Z-g33533ba8`. - Native-only external-runtime package SHA-256: - `922c49de4188cba2e58bae829c02d5e9b8af3a33dba5bc6619b469389c8e73da`. - No model or ASR runtime was bundled/downloaded. Staged `ldd` had no unresolved - dependencies. Installed and staged binary SHA-256 both matched: - `d77e4bec5b585a12a06885086c90c7d0e3b5b447d857b39a7f8586dfa7ce8518`. -- No old FrameYap binary was running at installation time. The managed installer - selected the new version and retained `2026-09-24T201841Z-gc5e925cb` as previous. - Installed `--version` and the relaunched process's executable path were verified. - Only FrameYap was launched; no SSH, SteamVR or terminal session was stopped. -- Startup reported normal input priority, Lasers anytime off, and - `panel-shown=Y`. This establishes successful initialization/show request, - including the new intersection-mask call, not visual/physical acceptance. - The normal runtime was left running for user testing; no recording or text - delivery was deliberately triggered by this check. - -Remaining acceptance: actual source-device reporting, grab/scale tracking, -release outside the mask, wrist/head behavior, transparency and comfortable -hit-target sizes. When legacy trigger release is not observable, dragging cancels -on loss of hover rather than relying on an outside MouseButtonUp event. diff --git a/docs/evidence/overlay-flicker-research-2026-09-24.md b/docs/evidence/overlay-flicker-research-2026-09-24.md deleted file mode 100644 index 9a7c2db..0000000 --- a/docs/evidence/overlay-flicker-research-2026-09-24.md +++ /dev/null @@ -1,105 +0,0 @@ -# Overlay flicker: source review and upstream reports — 2026-09-24 - -Local source review and public web research only. No Frame connection, OpenVR -initialization, microphone capture or input delivery was performed for this -investigation. No runtime fix or headset update is claimed. The earlier -[controls-only observations](ui-click-flicker-2026-09-24.md) remain separate -evidence; their device availability and permission do not carry forward. - -## Does FrameYap recreate the panel on interaction? - -The inspected `src/overlay.cpp` creates one OpenVR overlay in `Overlay::Impl`'s -constructor. Its only `DestroyOverlay` call is in cleanup, including startup -failure cleanup. Pointer handling, tab changes and `draw()` do not recreate that -handle. `PanelSurface` retains one fixed-size RGBA vector; normal redraws replace -its pixels, not its dimensions. This native panel is an OpenVR overlay, not an -SDL/X11/Wayland desktop window. - -`draw()` calls `SetOverlayRaw` when `PanelSurface::render()` reports changed -content. Hover and button down do not invalidate the canvas; action clicks in -`--check-controls` only log diagnostics. Tabs and placement notes can still -repaint. In normal dictation, action-induced status changes repaint too. - -The earlier event-counter trial recorded one show, zero hides and no hidden -events during interaction. The earlier static-canvas trial nevertheless had a -wearer report of whole-panel disappearance on action clicks. Thus application -handle recreation is unsupported by the code, and uploads cannot yet explain -all reported flicker. A compositor texture replacement or composition problem -could look like window recreation while the API overlay remains alive; this is -a hypothesis, not an observation of SteamVR internals. - -## Relevant primary sources - -- **OpenVR #772, April 2018:** a Linux C++ overlay author reported the entire - overlay disappearing between `SetOverlayRaw` updates, with different behavior - in the two eyes. Contributor Joe Ludwig advised using an OpenGL or Vulkan - texture with `SetOverlayTexture` for frequent updates, describing substantial - raw-upload latency and CPU/memory cost. This is a close match for redraw - flashes, but is historical guidance, not a Frame measurement or confirmation - that an overlay handle is destroyed. - [Report and recommendation](https://github.com/ValveSoftware/openvr/issues/772#issuecomment-380539744). -- **OpenVR #941, November 2018:** Ludwig reiterated that raw uploads are a poor - video path and recommended a graphics texture. This corroborates the API - recommendation; it is not an independent reproduction of our click-only case. - [Maintainer response](https://github.com/ValveSoftware/openvr/issues/941#issuecomment-440004776). -- **SteamVR 2.17.1 beta discussion, June 5, 2026:** Desktop+ developer - `elvissteinjr` reported severe flickering with cursor override and the default - cursor blob, plus problems with transparency and overlay ordering. FrameYap's - inspected path does not use cursor override, so this is evidence of related - compositor trouble, not an exact reproduction or a confirmed Frame bug. - [Firsthand reports, comments 7 and 10](https://steamcommunity.com/app/250820/eventcomments/572665855469650962/). -- **Valve's SteamVR 2.17 release notes, September 10, 2026:** include fixes for - dashboard/overlay cursor visibility and `MinimalControlBar` handling. These - establish intervening changes after the June report; they do not identify a - fix for FrameYap. Record the actual Frame runtime build before comparing it - with these reports. OpenVR SDK v2.15.6 does not identify the running SteamVR - version. - [Official announcement feed](https://steamcommunity.com/app/250820/announcements/?l=english). - -The GitHub web viewer omitted issue comments during this review; the linked -responses were checked through GitHub's public issues/comments API as well. - -## What existing counters can and cannot establish - -The pinned SDK defines `ImageLoaded` as completion of a raw/file image load, -not overlay creation. It separately defines `OverlayCreated` and -`OverlayDestroyed`. Shown/hidden events reflect API visibility, not proof that -every headset frame contains the panel. -[OpenVR v2.15.6 event definitions](https://github.com/ValveSoftware/openvr/blob/v2.15.6/headers/openvr.h#L853-L895). - -Our existing totals do not timestamp each click, identify every hit target, -record lifecycle events, or resolve the named overlay again. Consequently they -cannot distinguish an internal compositor resource change from a render-order -problem. A constant application handle alone would not distinguish these either. - -## Next controlled comparison, proposed only - -First extend the finite controls-only probe with monotonic timestamps, hit -targets, raw-upload/image-load sequence numbers, and the current handle plus -read-only `FindOverlay` results. Record created/destroyed events with their -target handles: cursor or dashboard overlays must not be counted as FrameYap -recreation. Keep diagnostics off the canvas. Record the runtime version and -whether the flash affects one eye, both eyes, just the cursor or the whole panel. - -Then compare one variable at a time with a contemporaneous wearer report: - -| Trial | Purpose | -| --- | --- | -| Static canvas, default laser, diagnostic action clicks | Reproduce interaction without new raw uploads after startup settles. | -| Same static canvas, only `HideLaserIntersection` enabled | Test whether the compositor's cursor blob participates; this deliberately removes cursor feedback. | -| Same panel placed clear of dashboard surfaces | Test overlap/ordering without a renderer change. | -| Scheduled content updates with no pointing or clicking | Test raw-image replacement independently of interaction. | -| Same content updates through a persistent GPU texture | Compare the raw path with `SetOverlayTexture`, retaining and synchronizing the texture correctly. | - -`HideLaserIntersection` suppresses the cursor blob; it does not disable mouse -input. `VisibleInDashboard` permits visibility there, whereas -`MakeOverlaysInteractiveIfVisible` activates global laser mode. These have -different effects and should not be changed together to diagnose flicker. -[Pinned flag definitions](https://github.com/ValveSoftware/openvr/blob/v2.15.6/headers/openvr.h#L3740-L3757). - -A persistent GPU texture is a justified rendering experiment for content -updates, but cannot be promised to fix a static overlay blinking on clicks. -If static-click flicker survives the cursor/placement comparisons, the resulting -minimal reproduction is useful for an upstream compositor report. No report has -been submitted. Any native probe change still needs offline checks, authorized -deployment/version verification and an opt-in human headset check. diff --git a/docs/evidence/pcm-range-fix-2026-09-24.md b/docs/evidence/pcm-range-fix-2026-09-24.md deleted file mode 100644 index 8e81667..0000000 --- a/docs/evidence/pcm-range-fix-2026-09-24.md +++ /dev/null @@ -1,38 +0,0 @@ -# Captured PCM range rejection — 2026-09-24 - -Historical observations; not permission for additional recording or inference. - -After the wearer enabled advanced debugging and retried speech, the private -worker log showed successful requests followed by an inference-stage exception: -`ValueError: PCM must contain finite samples in [-1, 1]`, raised by the runtime's -`_float_pcm` validation. No captured speech or full private log is reproduced here. -Both native submission and Python clip reading already rejected nonfinite values; -finite amplitude overshoot was not bounded. The precise source of that overshoot -(microphone gain, capture processing or resampling) was not measured. - -Commit `019b813` saturates finite microphone samples to `[-1, 1]` after SDL -conversion, preserving in-range samples and clip length. It does not rescale -whole clips or accept NaN/infinity. `Worker::submit` independently validates the -range before clip-file/IPC mutation. Fake-device and IPC regression tests cover -both signs of overshoot, extreme finite values, exact boundaries, unchanged -ordinary samples, nonfinite rejection and subsequent capture recovery. - -Local default CTest passed 13/13; local native hardware-free CTest passed 16/16. -The coordinated native Git snapshot also includes `3c03f88f`, which changes the -Bindings button to open SteamVR's editor directly. Its source diff was reviewed -before the combined build; no competing installer was run. - -ARM64 native hardware-free CTest: 16/16. Installed/verified build: -`2026-09-24T191513Z-g3c03f88f`. -Native-only package SHA-256: -`d8433bc6e9e84ee9b54728d99cf1f7631dd999d6aee562a54cc0d04cad484716`. -Installed `--version`, `current` selection and the relaunched executable path -matched. `previous` retains `2026-09-24T190806Z-g6592e1af`. - -User config and authorized runtime/model paths hashes were unchanged across -installation; advanced debugging remained enabled by the user. No FrameYap -process was running at the pre-install check. Only FrameYap was launched; -no SteamVR, SSH or user-session process was stopped. The new diagnostic log -was mode 0600. No assistant-triggered recording, private-audio replay, public -fixture inference or input injection was performed. Actual speech retry after -this fix, recognition quality and headset interaction remain wearer-led checks. diff --git a/docs/evidence/poc-cpu-overlay-2026-09-24.md b/docs/evidence/poc-cpu-overlay-2026-09-24.md deleted file mode 100644 index 1290c94..0000000 --- a/docs/evidence/poc-cpu-overlay-2026-09-24.md +++ /dev/null @@ -1,103 +0,0 @@ -# FrameYap POC observations — 2026-09-24 - -Dated observations on one user-authorized Steam Frame, not proof of future device -availability, full dictation acceptance or runtime redistribution rights. No audio, -weights, private transcript/log files, credentials or runtime binaries are included. -The final native package below was built from source commit **`d60d8d2`**. - -## CPU compatibility trial (before runtime-license review) - -Isolated user-local environment: ARM64, glibc 2.39, Python 3.12.3, -moondream 2.4.0 / kestrel 0.8.0 / kernels 0.7.0 / native 0.1.8, -Torch **2.8.0+cpu** (`torch.version.cuda is None`). Model weights/config/tokenizer -verified against pinned Redux revision `fad622f25f303105c20d70e201bcc477c88b620c`. -No dense model substitution or desktop/cloud inference was used. - -A one-second synthetic silence trial loaded in 5.645 s; three decodes took -0.187 / 0.149 / 0.167 s and returned no text. Peak process RSS in that trial was -989,844 KiB (~967 MiB). This is not peak RSS for arbitrary 20-second speech clips. - -Public 11.0-second JFK speech fixture: -. -Source FLAC SHA-256 `63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715`. -Converted to PCM16/mono/16 kHz WAV with ffmpeg; WAV SHA-256 -`0c397b12d7dbdd89e4fd0d9a0840c14a2c5c3707758562755cda96a931b25a91`. -The persistent worker used bounded file/pipe IPC; no microphone was opened. - -| CPU threads | Worker load | Five request times (seconds) | Median | RTF | Audio/compute ratio | -| --- | --- | --- | --- | --- | --- | -| 2 | 5.190 s | 1.550, 1.619, 1.587, 1.619, 1.659 | 1.619 s | 0.147 | 6.8× realtime | -| 4 | 4.655 s | 1.196, 1.158, 1.143, 1.249, 1.502 | 1.196 s | 0.109 | 9.2× realtime | - -The first result reproduced the fixture's known sentence; all five replies in -each run were correlated and 108 UTF-8 bytes. These are medians/maxima from one -public clip, not representative accuracy/WER, p95, streaming realtime performance, -or compositor/thermal acceptance. Load time is excluded from RTF. Two threads -remain the default pending scene-contention measurements. - -An unconstrained dependency install initially selected CUDA Torch/NVIDIA wheels; -these were replaced/removed from the owned venv before measurement. No system -packages were changed. Later inspection found the proprietary kernel license's -separate-agreement requirement. Permission was declared unresolved; further -inference and proprietary-runtime packaging were paused pending vendor clarification. -See [license boundary](../third-party.md). The acquired development environment -remains user-local and is not included in the native-only installation. - -## Native runtime / controller / overlay checks - -- Native CMake build succeeded on Frame using standalone OpenVR SDK v2.15.6, - explicitly built SDL 3.2.16, system Wayland and FreeType. No unrelated app tree - or environment was linked/copied. No sudo or system/session restart. -- Installed executable connected to `gamescope-0`, bound the IME and received - ready/done state; each discovery check disconnected **without text or Enter**. -- OpenVR accepted the raw panel. The user reported seeing the five-second window. - This confirms visibility, not readability/comfort or successful pointer clicks. -- Device controller properties identified `frame_controller` and its input profile. - The profile exposes grip `click`; authored default bindings were added. A - controls-only run reported both grip actions tracked/active. **No physical - gesture delivery was observed in that run.** Double-tap/hold/loss behavior is - currently verified by deterministic unit tests, not a human controller trial. -- The ARM loader initially tried `/data/work/openvrpaths.vrpath`. Selecting the - existing standard user registry fixed initialization; code now does so only - when no explicit override is supplied. -- Registration initially returned API success but did not install the app. - Frame's manifest parser requires **`binary_path_linux_arm`**. Adding it fixed - registration. Code now verifies `IsApplicationInstalled`, not return status alone. -- Repeated Add, Remove and re-Add of **only** `local.frameyap.overlay` succeeded; - autolaunch queried **off**. The final installed panel check also succeeded after - explicit identification with that registered key. Menu-driven launch and cold - SteamVR startup were not exercised. - -## Installer and final state - -Native-only artifacts `v0.1.0-poc1`/`poc2` exercised installation, same-version -rerun, upgrade, rollback and forward selection. A producer-prefix SDL RUNPATH leak -was found and removed; final ELF RUNPATH is only `$ORIGIN/../lib` and bundled -SDL/OpenVR resolve inside the installed tree. - -Explicit OpenVR unregister followed by `--uninstall --unregistered` succeeded. -The final `v0.1.0-poc3` artifact was then installed twice and registered with -autolaunch off. It includes native code, SDL/OpenVR, Hack font/notices and the -worker adapter, **no ASR runtime or weights**. Installed size: ~8.1 MiB. - -Local archive SHA-256: -`90aab107feea2cb501bf815806bdc1d04d84bb6f87f7c795b60ad43c7fdc0634`. -The artifact is retained only in the device's project-owned development directory; -no GitHub release/upload was performed. Installed launcher is `~/.local/bin/frameyap`. -Actual native CLI and installer both refused an independently held install lock. -At the final check, no owned `frameyap` process or transient `frameyap-*` runtime -directory remained. Installation/registration remain; nothing was autostarted. - -Final verification on source `d60d8d2`: default local build **8/8 CTests**; -optional native local build **9/9**; native ARM64 build **9/9**. Suites include -13 installer cases, worker framing/hash/cancellation/deadlines, gestures, state, -instance locking, and an isolated fake Wayland server. Those tests do not connect -to the live compositor or record audio. `git diff --check` passed locally. - -## Still open - -Runtime permission and any public distribution; live microphone quality/device -selection; physical tap/hold/buttons; actual text/Enter delivery into disposable -and real target apps; focus/held-modifier behavior; quick typing's focus observer; -scene/dashboard collisions; compositor cost, battery/thermal and headset comfort. -No live microphone recording or input injection occurred in this POC pass. diff --git a/docs/evidence/raw-overlay-teardown-crash-2026-09-24.md b/docs/evidence/raw-overlay-teardown-crash-2026-09-24.md deleted file mode 100644 index e2fd594..0000000 --- a/docs/evidence/raw-overlay-teardown-crash-2026-09-24.md +++ /dev/null @@ -1,100 +0,0 @@ -# Raw-overlay probe → compositor/session crash — 2026-09-24 - -Incident analysis from read-only boot, journal, coredump metadata and SteamVR log -inspection after the user reported a headset reboot and loss of their terminal. -No repeat crash experiment was performed during analysis. This record does not -authorize a probe, restart or deployment; live reproduction can destroy the desktop -session even when the test overlay is hidden. - -## Established timeline (UTC) - -- Linux stayed on the same boot throughout; uptime was about seven hours when - checked at 20:55. This was **not an OS reboot**. -- 20:50:24.955: the normal FrameYap process called `VR_Shutdown`, then disconnected. - Its later absence explains the read-only probe's `FindOverlay` failure; that - failure is not evidence that SteamVR had already crashed. -- 20:51:42.264: the final `mask-probe` process started as `VRApplication_Overlay`. -- 20:51:42.279: it connected to the existing compositor (PID 2316). -- 20:51:42.284: it called `VR_Shutdown`; compositor logs record its pipe disconnect. -- 20:51:42.357: SteamVR's crash reporter was already processing a compositor dump. - Coredump metadata timestamps the crash at 20:51:42, signal **SIGBUS (7)**. -- 20:51:42.563: systemd-coredump began processing it; completion was logged at - 20:51:44.195. The completion time is not the initial fault time. -- 20:51:47: `steamvr.service` failed and scheduled a restart. -- 20:51:57: Gamescope failed to stop within ten seconds; systemd killed its - process group, including Xwayland, and restarted the graphical session. - This accounts for the headset appearing to reboot and terminal-session loss. - -Read-only `systemctl --user show` established the coupling: -`steamvr.service` has `Restart=always` and requires `gamescope-session.service`; -`gamescope-session.service` has `PartOf=steamvr.service graphical-session.target` -and a ten-second stop timeout. The agent did not issue a system reboot or kill -SSH/terminal processes; the service recovery cascade performed the session kills. - -## Trigger sequence and result - -The temporary native ARM64 probe used OpenVR SDK 2.15.6; client startup reported -runtime 2.17.10. It created its own hidden overlay, set an absolute pose, 1 m width, -mouse input and 1080×780 mouse scale, then: - -1. Allocated a zero-filled 1080×780×4-byte RGBA buffer (3,369,600 bytes). -2. Called `SetOverlayRaw`; returned `VROverlayError_None`. -3. Applied one rectangular intersection mask, first at (996,32,68,68), then at - (996,680,68,68). Both calls returned success. -4. Called `ComputeOverlayIntersection` for three synthetic rays per mask. All - six returned false, including the intended positive cases. Therefore this - probe **did not validate either mask coordinate convention**. -5. Immediately called `DestroyOverlay` and `VR_Shutdown`, without waiting for an - image-loaded event or compositor upload completion. It never showed the overlay. - -The normal FrameYap renderer uses a persistent Vulkan texture, not this raw-upload -path. Free-grab/lock changes were still local and were not running on the device. -The final probe source was retained at `/tmp/frameyap-mask-probe.cpp` locally and -`~/frameyap-poc/mask-probe.cpp` on Frame at analysis time; do not rerun casually. - -## Crash evidence and interpretation - -The faulting compositor thread's available stack began: - -```text -__memcpy_sve libc.so.6 + 0xa0008 -vrcompositor + 0x298cb8 -vrcompositor + 0x6eaf4 -vrcompositor + 0x11bcf0 -vrcompositor + 0x11c0c4 -vrcompositor + 0x14b9b4 -vrcompositor + 0x14f9e0 -vrcompositor + 0x183bf0 -start_thread -``` - -The compositor core exists but was inaccessible to the unprivileged account; -no privilege escalation or core extraction was attempted. No GPU reset, kernel -panic or OOM event appeared in the inspected kernel interval 20:50–20:52. -SteamVR's own crash reporter automatically uploaded its minidump, reporting -CrashID `bp-f1d750cf-8943-4834-8873-9eec02260924`; the agent did not initiate that -upload or send a dump separately. - -**Strongly supported trigger:** the hidden raw-overlay probe and its immediate -teardown. The compositor fault followed its disconnect within the same second. - -**Leading unproven mechanism:** an asynchronous raw-image copy outliving its -shared-memory backing during overlay destruction/client shutdown. SIGBUS in -`memcpy` is consistent with an invalid/truncated mapped backing object. It does -not prove that mechanism: raw-upload buffer sizing/limits or another compositor -memory-handling fault remain alternatives. Symbols, fault-address/mapping data, -or a controlled comparison are needed to distinguish them. - -## Consequences for further work - -- Hidden overlays are not isolated from compositor upload/lifetime machinery. -- API success and a clean probe exit do not establish compositor completion. -- Keep normal rendering on the existing Vulkan path; do not reintroduce - `SetOverlayRaw` as a convenient live diagnostic shortcut. -- A future explicitly authorized experiment should separate raw upload from - immediate teardown, vary only one factor at a time, and capture process/journal - evidence from a session outside the headset graphical service. A longer-lived - probe is an experiment, not a proven workaround; adding a sleep is not a fix. -- Tmux protects a remote coding process from a terminal/SSH disconnect. It does - not itself isolate the headset compositor, and a tmux server inside a killed - service group could still die. Confirm where the server lives before replaying. diff --git a/docs/evidence/ui-click-flicker-2026-09-24.md b/docs/evidence/ui-click-flicker-2026-09-24.md deleted file mode 100644 index 1b2dc9a..0000000 --- a/docs/evidence/ui-click-flicker-2026-09-24.md +++ /dev/null @@ -1,59 +0,0 @@ -# FrameYap click flicker investigation — 2026-09-24 - -Dated controls-only observations on one user-authorized Frame. This is not -headset acceptance or proof that a compositor update will behave identically. -No microphone was opened and no text/Enter was delivered. Both checks used the -finite `--check-controls --mount world` mode; clicks were diagnostic only. - -## What changed - -The earlier diagnostic painted pointer counts, action text and controller state -onto the panel, causing `SetOverlayRaw` uploads even for clicks that would not -otherwise alter its pixels. Commit `b5499ec` logs those values to the terminal -instead; it keeps the panel static for Record/Cancel/Insert/Enter clicks, while -navigation and mount changes still repaint. A subsequent commit, `c59c8b1`, -counts application raw uploads, show/hide calls and OpenVR visibility/focus/image -events without adding repaints. - -The rendering loop in `src/runtime.cpp` and `src/overlay.cpp` is single-threaded: -it draws before polling input and processes click actions before the next draw. -No separate logic/render thread race was found in this path. That alone does not -identify SteamVR's compositor behavior. - -## Controls-only trials - -- Static-canvas build `2026-09-24T173900Z-gb5499ec` was built on Frame from a - clean snapshot; 11/11 hardware-free ARM64 tests passed. Its native-only archive - was checksum-verified and installed; the previous version was retained. The - 30-second probe exited normally with 22 pointer downs, 22 ups and 8 action/ - mount/recenter hits. The wearer reported that the **whole overlay still - disappeared on every action click**, including clicks that leave the canvas - static. This contradicts full raw uploads being the *sole* cause of the flash. -- Event-counter build `2026-09-24T174700Z-gc59c8b1` was built on Frame from a - clean snapshot; 12/12 hardware-free ARM64 tests passed. It was checksum-verified, - installed and version-checked. The 30-second probe exited normally with 7 - pointer downs, 7 ups and 2 action/mount/recenter hits. From start to finish, - application `ShowOverlay` calls stayed at **1**, `HideOverlay` calls at **0**, - `VREvent_OverlayShown` at **1**, and `VREvent_OverlayHidden` at **0**. - `SetOverlayRaw` calls rose from 2 initial uploads to 7 during the trial; - image-loaded events reached 7 with no image-failed events. Four overlay and - four global focus-change notifications were observed, but not on every click; - input-focus-captured and gamepad-focus-lost counters stayed at zero. Pointer - events occurred without intervening raw uploads or hide/show events. This - second probe had no separate contemporaneous headset visibility report. - -The second probe did not log hit targets; its five later raw uploads may include -tab, mount or placement-note changes and are not a per-action-click count. The -log does **not** establish that SteamVR -never briefly occluded the panel: it only shows that the app did not call hide -and received no hidden event in that finite trial. The first trial's visual -report plus static-click rendering behavior point toward dashboard laser/input -compositing rather than a FrameYap repaint race, but the exact compositor cause -remains unverified. A persistent GPU texture alone cannot explain or guarantee a -fix for flicker observed when no texture is uploaded. - -The installed diagnostic version at the end of this investigation was -`2026-09-24T174700Z-gc59c8b1` (native-only, external ASR runtime unchanged); an -independent controls-only right-X binding trial occurred between these two -builds. No session or SSH process was terminated. No further hardware remedy is -claimed here. diff --git a/docs/evidence/vulkan-overlay-2026-09-24.md b/docs/evidence/vulkan-overlay-2026-09-24.md deleted file mode 100644 index 29a86d3..0000000 --- a/docs/evidence/vulkan-overlay-2026-09-24.md +++ /dev/null @@ -1,61 +0,0 @@ -# Persistent Vulkan overlay deployment — 2026-09-24 - -Source commit: `4c043c99`. -Installed native ARM64 version: `2026-09-24T181314Z-g4c043c99`. - -The user requested replacing raw uploads with a GPU texture and supplied the -current Frame SSH target and key for native build/install verification. These -are dated observations, not future device availability or permission to launch -hardware checks. - -## Implementation - -The overlay now uses `SetOverlayTexture` with a persistent Vulkan RGBA8 image. -The CPU panel rasterizer still produces the pixels. One staging allocation, -image and command buffer are reused; redraws do not recreate them. SteamVR -selects the physical device and required instance/device extensions. Transfers -share one FrameYap-owned graphics queue with SteamVR, and GPU resources survive until -`VR_Shutdown` completes. No desktop surface, swapchain or raw-upload fallback -was introduced. See [rendering details](../overlay.md). - -## Verification - -- Default local offline build: 13/13 CTest checks passed. -- Local native build: all 16 checks passed; the fake Wayland protocol check - required permission to bind its local test socket outside the sandbox. -- Native ARM64 Release build on Frame from a checksum-verified Git archive: - 16/16 hardware-free checks passed, including the Vulkan fake-driver test. -- Explicit offscreen Vulkan check on the workstation's RTX 4090, with - `VK_LAYER_KHRONOS_validation` enabled: eight exact 1000×680 RGBA readbacks, - one stable image, no validation messages. -- The same explicit offscreen check on Frame's `Turnip Adreno (TM) 750`: - eight exact 1000×680 RGBA readbacks and one stable image. This exercised the - actual image upload/layout/readback path, without OpenVR initialization. -- Staged and installed native dependency resolution succeeded. Vulkan resolves - to the system loader; SDL/OpenVR resolve inside FrameYap's own `lib/`. -- The native archive was checksum-verified and installed. `current` selects - the version above; `previous` retains `2026-09-24T174700Z-gc59c8b1`. - Installed `--version` matched and its executable hash matched the staged - binary. No app, SSH or user-session process was terminated. - -Native archive SHA-256: -`d6f0173b25a879c02f0ee67063c881a23dff874a71674826ddd7666d0980759d`. -Installed executable SHA-256: -`761ea0e4d2dc235c5f056d8d944cb68392713c246aaec008d0b6c235954ef645`. - -## Acceptance boundary - -The updated headset installation is verified, but this session did not launch -an OpenVR visual/controls probe or collect a wearer report. GPU readback does -not establish SteamVR texture acceptance, orientation, click behavior or a -flicker fix. No microphone, inference or text/Enter delivery was exercised. -The next human headset check should compare both static action clicks and -content-changing tabs using the installed Vulkan build's `--check-controls`. - -## Subsequent wearer report — 2026-09-24 - -After testing the installed Vulkan build, the user confirmed that the flicker -is fixed. They separately reported controller shortcuts becoming unavailable -in system laser/dashboard interaction states while pointer clicks work. This -is wearer confirmation of the visual fix and a distinct input-routing issue; -it does not establish microphone, transcription or text-delivery acceptance. diff --git a/docs/install-design.md b/docs/install-design.md index dadba07..b7a9824 100644 --- a/docs/install-design.md +++ b/docs/install-design.md @@ -2,10 +2,11 @@ **User goal:** install from GitHub with a `curl … | bash`-style command, without a Steam store AppID. An idempotent archive installer and native-only local artifacts -are now implemented/tested; no public release is published. The full bundled-ASR -experience remains blocked on runtime permission. This document retains the target -design; see [current packaging](packaging.md), [POC evidence](evidence/poc-cpu-overlay-2026-09-24.md) -and [third-party boundary](third-party.md). +are now implemented/tested; no public release is published. A bundled-ASR +experience is not yet offered. This document retains the target +design; see [current packaging](packaging.md) and [third-party notes](third-party.md). +Planned installer work (binary-or-source choice, flag-driven operation) is in +[TODO.md](../TODO.md). The read-only `scripts/install-preflight.sh` checks whether a host appears suitable for the **proposed** Linux ARM64 glibc package format and has the expected basic @@ -66,16 +67,15 @@ confirmation from stdin while the installer itself is arriving through that pipe no external application libraries/assets and no system Python modification. - Model fetched during explicit installation/setup, with pinned revision/hash and attribution. A documented `--without-model` option can defer the large download. - No surprise first-utterance downloads. Licensing may require obtaining particular - runtime components from their vendor instead of redistributing them in our tarball. + No surprise first-utterance downloads. Runtime components are installed from + their own package index rather than redistributed in our tarball. - Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`), configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in `~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory. - No sudo, OS read-only-root changes, package-manager installs, udev changes, `/dev/uinput` permission changes or modifications to unrelated launchers. - No Steam store AppID, Steamworks SDK, root service or network ASR dependency. -- Distribution license/third-party notices must be resolved before publication; - the small model file alone is not the full runtime or redistribution permission. +- Third-party notices for everything we redistribute must be included with a release. ## Installer lifecycle and safety diff --git a/docs/overlay.md b/docs/overlay.md index 00f3243..0092cdf 100644 --- a/docs/overlay.md +++ b/docs/overlay.md @@ -1,4 +1,4 @@ -# Native OpenVR POC panel +# Native OpenVR panel `src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion. @@ -12,7 +12,7 @@ normal build or test. Explicit development dependencies: Valve OpenVR SDK v2.15.6, Vulkan headers/loader and FreeType 2. The native runtime needs a compatible system Vulkan driver. Configure/build must not fetch them. The default font is the bundled Inconsolata -Regular, also used by kouseki; its OFL and extraction provenance are included in +Regular; its OFL and notices are included in [third-party notes](third-party.md). `--font FILE` overrides the JSON selection. A missing selected font falls back to bundled Inconsolata, then a system DejaVu Sans face if present. Glyph coverage depends on the selected face; full CJK @@ -24,7 +24,7 @@ margins for a thin grab underline and an external L-shaped scale handle. `src/ov persistent Vulkan RGBA8 image and submits it with `SetOverlayTexture`. The image, staging allocation and command buffer are reused; tabs do not create extra overlays or render targets. The rounded mint-to-blue perimeter, -shallow curved accent, and dark cards borrow kouseki's VR visual language. Rounded +shallow curved accent, and dark cards form the panel's visual language. Rounded preview, status and control surfaces use independently rasterized antialiased edges and restrained baked neon halos rather than GPU bloom. The recording indicator and selected controls remain distinguishable by their labels, not color alone. Rounded @@ -43,13 +43,10 @@ transfer before reusing staging memory. Image barriers finish in [OpenVR's Vulkan contract](https://github.com/ValveSoftware/openvr/wiki/Vulkan). The queue is used on the overlay thread; GPU resources outlive `VR_Shutdown`. The device selection, texture description and persistent panel-upload patterns -were compared with kouseki's `openvr_session.cpp` and `vulkan_renderer.cpp` at -`738569f4c41ff4c8fc9edd5bfff9c861957ea39e`; FrameYap owns this implementation. +are FrameYap's own implementation. GPU setup/submission errors stop startup or the run with an explicit error. This replaces the raw-upload rendering path; headset flicker acceptance still -requires an on-device comparison. The -[Vulkan deployment record](evidence/vulkan-overlay-2026-09-24.md) documents the -native installation and offscreen GPU checks separately from headset acceptance. +requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance. The header shows local time and date instead of the former on-device/review and current-mount labels. It updates when the displayed minute or date changes, @@ -280,16 +277,13 @@ and panel-front (controller +Z and -X), keeping panel-up unchanged so it faces inward with upright, unmirrored text. The controller-relative center is (0, 0.18, 0.089) m, approximating the compact HUD's surface center: its 0.12 m wrist lift, 0.09 m bottom anchor and ~0.03 m panel-center correction; -Z combines the fallback 0.054 m wrist calibration and 0.035 m finger-back offset. This copies placement geometry, not -VR Workspace's avatar-dependent wrist calibration. Wrist-mounted panels now use -the same *behavior* as kouseki's watch HUD: fully visible while their entire +Z combines the fallback 0.054 m wrist calibration and 0.035 m finger-back offset. Wrist-mounted panels are fully visible while their entire orientation is within 60° of an upright, viewer-facing panel; linear opacity fade from 60° to 75°, then hidden (including laser interaction). Pitch, yaw and roll contribute together; turning the wrist away or moving the head around it changes the angle. OpenVR's overlay alpha changes without rerendering the panel. World and head mounts do not fade. Missing headset tracking hides a wrist panel; -a lost wrist still uses the existing world-space fallback. This was implemented -independently with no kouseki library or runtime dependency. Headset readability, +a lost wrist still uses the existing world-space fallback. Headset readability, fade feel and interaction at the threshold still need live acceptance. To tune the selected wrist, set `wrist` in `config.json` as in the example above: diff --git a/docs/packaging.md b/docs/packaging.md index 23a2df1..896c658 100644 --- a/docs/packaging.md +++ b/docs/packaging.md @@ -1,9 +1,8 @@ # Release packaging and idempotent user-local installer **No GitHub release is published.** Native-only local artifacts have been installed -and reinstalled on Frame. The end-to-end bundled-ASR release remains blocked on -runtime permission; see [third-party notes](third-party.md). The installer never -pretends the proprietary runtime is included when it is not. +and reinstalled on Frame. The installer never pretends an ASR runtime is included when it is not; +see [third-party notes](third-party.md). ## Producer @@ -21,11 +20,11 @@ model/* # optional pinned public weights + attribution runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact ``` -For the current **external-runtime** POC, `scripts/stage-native-poc.py --help` +For the current **external-runtime** package, `scripts/stage-native-poc.py --help` documents explicit inputs. It invokes `cmake --install` on an existing native build, copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does -not build, download, run the app, or copy a proprietary ASR runtime. The native -POC relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType, +not build, download, run the app, or copy an ASR runtime. The native +app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType, libstdc++ and glibc; audit `ldd` on the installed binary. SDL/OpenVR resolve inside its own `lib/`, not a producer prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux. @@ -46,10 +45,8 @@ the installer still refuses a reused tag whose contents have changed. Historic `--external-runtime` refuses a runtime directory and records `runtime: external-authorized-python` in `release.json`. The installer explicitly -reports that ASR is not supplied. Without that flag, a complete independently -licensed, compatible isolated CPU Python runtime is required. **Do not use that -bundled route for Kestrel without permission covering redistribution.** Staging -validation is not a license grant or an inference test. +reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required +in the archive. Staging validation is not an inference test. The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz` plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal, @@ -129,7 +126,7 @@ Runtime/check/registration modes and installer share an exclusive nonblocking Selection of a completed `current` is atomic; `previous` is retained. `sh install.sh --rollback` switches to the prior validated version. Foreign/modified wrappers, untracked install files and inconsistent ownership metadata are refused. -Same-user malicious concurrent filesystem mutation is outside the POC threat model. +Same-user malicious concurrent filesystem mutation is outside the current threat model. The generated `frameyap.vrmanifest` uses `local.frameyap.overlay`, **not a store AppID**. Linux ARM requires `binary_path_linux_arm`; both Linux fields are written. @@ -178,8 +175,7 @@ not hand-edit Steam's shortcut database. This check does not validate microphone capture, transcription, controller input, text delivery or cold SteamVR startup. With a configured inference runtime, the -menu launch attempts to load the model; defer that test until runtime licensing is -resolved or independent authorization is established. If an +menu launch attempts to load the model; defer that test unless you intend to load the model. If an environment/configuration unexpectedly supplies a runtime/model, do not perform this inert launcher check. @@ -189,6 +185,5 @@ an acknowledgement, not a hidden SteamVR edit. Only owned files are removed; config stays, models move to `saved-models/VERSION`, conflicts/untracked files abort. Offline tests: `python3 -m unittest discover -s tests -p test_installer.py`. -They use temporary homes/local fixtures. Dated live-device observations are in the -[POC record](evidence/poc-cpu-overlay-2026-09-24.md). When editing the Python helper, +They use temporary homes/local fixtures. When editing the Python helper, run `python3 scripts/sync-installer.py`; tests enforce embedded installer parity. diff --git a/docs/provenance.md b/docs/provenance.md deleted file mode 100644 index d4d28a9..0000000 --- a/docs/provenance.md +++ /dev/null @@ -1,32 +0,0 @@ -# Technical provenance and limits - -FrameYap is a standalone native POC. No unrelated application's source, build -tree, assets or Python environment is included. No model weights or runtime -binaries are in Git. The included Gamescope XML's provenance and separately -licensed build inputs are recorded in [third-party notes](third-party.md). -The dated [device probe](evidence/frameyap-apis-2026-09-24.md) records limited historical observations; its fixture code and private logs are -not part of this repository. No current device availability or release install -can be inferred from those observations. - -## Model and preliminary measurements - -Proposed model: [moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux/tree/fad622f25f303105c20d70e201bcc477c88b620c), -revision `fad622f25f303105c20d70e201bcc477c88b620c`. An earlier local -inspection measured a 177,774,490-byte weight file. Its model card identifies -CC-BY-4.0; separate runtime/kernel redistribution terms must be reviewed before -packaging. No license for those artifacts is granted by this repository. - -A preliminary, unpublished desktop benchmark used Python 3.12.13, moondream -2.4.0, kestrel 0.8.0 and four CPU threads. On an i7-13700K, warm decode -median/p95 were 61/137 ms and raw WER 11.11% on just 25 reviewed clips / 189 -words. This is not a general accuracy estimate or Frame latency prediction. The -CPU-selected desktop process also occupied GPU memory; GPU-free operation was -not established in that earlier desktop trial. A later isolated ARM64 CPU-only -public-clip trial is recorded in [POC observations](evidence/poc-cpu-overlay-2026-09-24.md); -it does not establish live-microphone/headset acceptance or distribution permission. - -## Public platform references - -- [OpenVR 2.15.6](https://github.com/ValveSoftware/openvr/releases/tag/v2.15.6) -- [Gamescope 3.16.28 input-method protocol](https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml) -- [Gamescope IME implementation](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/ime.cpp) diff --git a/docs/third-party.md b/docs/third-party.md index d20a8ca..db95b37 100644 --- a/docs/third-party.md +++ b/docs/third-party.md @@ -20,9 +20,7 @@ images, protected kernels or another application's assets were extracted for the ## Bundled font and UI reference -`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of the typeface used -by kouseki's editor and VR canvas, extracted from its `assets/fonts` directory at -checkout revision `738569f4c41ff4c8fc9edd5bfff9c861957ea39e`. +`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular. SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`. Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**, retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace @@ -32,10 +30,8 @@ OFL, not MIT, and is not sold by itself. Upstream: , exact revision `fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution: @@ -66,24 +62,16 @@ sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`. `fetch-model.py` fetches and retains the original model card alongside the files. No weights are committed to this repository. -The inspected `kestrel-kernels==0.7.0` wheel license identifies it as proprietary -M87 Labs software and says use requires a separate written agreement. Copying and -redistribution are restricted by that agreement. This includes its protected CPU -payload, not just CUDA. The Python wrapper and model card do not override those -terms. No attempt was made to unpack/decrypt/reverse-engineer protected kernels. +Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels` +dependencies, installed by the user from PyPI into their own environment. Upstream's +Kestrel README states: "Local inference is free and requires no API key" +(); finetuned-model inference needs a Moondream +API key and is not used here. FrameYap's release archives do not vendor or bundle +these packages; they are installed from PyPI onto the user's machine, either by the +user or by the installer at the user's request. -**Release blocker:** permission covering use and redistribution has not been -established here. The initial on-device compatibility measurements preceded this -license review; further inference/bundling was paused when the issue was found. -The acquired packages remain isolated under the device's project-owned development -directory, not in Git or a published artifact. Do not advertise the GitHub runtime -bundle as available or automatically download/install those packages for end users. -Resolve permission with the vendor, or separately scope an independently licensed -runtime for the same weights. Do not silently substitute a dense/heavier model. - -Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) also -need their own notice/license inventory before publication. The public model's -small size is neither total runtime size nor redistribution permission. +Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need +their own notice/license inventory before a bundled release. ## CPU trial dependency choice @@ -95,7 +83,7 @@ Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained `pip install moondream` as a CPU setup recipe. For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution -was downloaded but not bundled with the proprietary runtime: +was downloaded but not bundled with any ASR runtime: , `cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`, SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`. diff --git a/docs/worker.md b/docs/worker.md index 48736a6..f3898d0 100644 --- a/docs/worker.md +++ b/docs/worker.md @@ -46,7 +46,7 @@ thread-pool variables set before import. It uses `md.photon("moondream/parakeet-redux", model_path=, device="cpu", cpu_threads=threads)` and persistent `transcribe(audio=, sample_rate=16000)["text"]`. -Install an **isolated** Python runtime with the separately reviewed +Install an **isolated** Python runtime with the moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide preinstalled local weights from revision `fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory @@ -58,11 +58,9 @@ flags do not prove every third-party internal is unable to access a network. Runtime/build/tests perform no downloads; the separate explicit setup utility `scripts/fetch-model.py` can provision the public pinned weights. -**Licensing blocker:** the observed kestrel-kernels 0.7.0 license requires a -separate M87 Labs agreement; do not treat wheel availability as permission for use -or bundling. See [third-party notes](third-party.md). No public runtime bundle has -been released. Limited ARM64 measurements are in the [POC record](evidence/poc-cpu-overlay-2026-09-24.md), -not a claim of complete headset acceptance. +Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public +runtime bundle has been released. Limited ARM64 measurements were taken during development; +they are not a claim of complete headset acceptance. ## Advanced debugging diff --git a/scripts/stage-native-poc.py b/scripts/stage-native-poc.py index 2864abb..3a8c801 100644 --- a/scripts/stage-native-poc.py +++ b/scripts/stage-native-poc.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Stage a native-only POC from an explicit native build and licensed files. -No downloads, compiler invocation, proprietary ASR runtime, registration or launch. +No downloads, compiler invocation, ASR runtime, registration or launch. System Vulkan loader/driver, Wayland/FreeType/libstdc++/glibc remain platform prerequisites. """ import argparse @@ -43,7 +43,7 @@ def main(): notices = ["FrameYap native-only POC. No ASR runtime or model is included.\n", "Original FrameYap code: MIT. System Vulkan/Wayland/FreeType/libstdc++/glibc are not bundled.\n", "Bundled libraries: Valve OpenVR and unmodified SDL3; font license included below.\n", - "This package does not grant any rights to kestrel-kernels or provide a functioning ASR environment.\n"] + "This package does not include Kestrel or provide a functioning ASR environment.\n"] for label, file in (("FrameYap", root / "LICENSE"), ("OpenVR", args.openvr_license), ("SDL3", args.sdl_license), ("Font", args.font_license)): notices.extend([f"\n--- {label} ---\n", file.read_text()]) diff --git a/src/panel_surface.cpp b/src/panel_surface.cpp index 57ba4df..7719def 100644 --- a/src/panel_surface.cpp +++ b/src/panel_surface.cpp @@ -171,7 +171,7 @@ struct PanelSurface::Impl { } } void frame() { - // Independently rasterized version of kouseki's HUD visual language: + // Independently rasterized neon-frame HUD style: // rounded mint-to-blue perimeter and a second shallow curved accent. for (int y = 0; y < CH; ++y) for (int x = 0; x < CW; ++x) { if (x > 34 && x < CW - 34 && y > 34 && y < CH - 34) continue;