mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-04 22:00:03 +02:00
docs: prepare FrameYap release plan and archive device notes
This commit is contained in:
baketnk committed
1 parent
1bde401aa8
commit
07c03ea27e
28 files changed
+268
-1000
No files matched your search
@@ -11,3 +11,6 @@ __pycache__/
|
||||
.env.*
|
||||
!.env.example
|
||||
.DS_Store
|
||||
|
||||
# Local archive of dated device evidence and old provenance notes (not published)
|
||||
/docs/archive/
|
||||
@@ -1,8 +1,9 @@
|
||||
# Frame Dictation development
|
||||
# FrameYap development
|
||||
|
||||
This is an independent project, not a plugin for another application.
|
||||
Read README.md and docs/design.md before implementation. Dated evidence records
|
||||
past observations; it never proves current device availability or grants a live run.
|
||||
Read README.md and docs/design.md before implementation. Dated device evidence is kept
|
||||
locally in the untracked `docs/archive/`; it records past observations and never proves
|
||||
current device availability or grants a live run.
|
||||
|
||||
- Keep dependencies explicit and small. Do not add a dependency/submodule/symlink
|
||||
to an unrelated application's build tree, assets or Python environment.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# FrameYap
|
||||
|
||||
Standalone, on-device voice typing POC for Steam Frame. **MIT licensed.**
|
||||
Standalone, on-device voice typing for Steam Frame. **MIT licensed.** Early release (v0.1 in progress).
|
||||
|
||||
Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture,
|
||||
persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope,
|
||||
@@ -11,11 +11,11 @@ server, cloud fallback or unrelated application dependency.
|
||||
Gamescope discovery and native-only installation have been exercised on Frame.
|
||||
Live microphone → reviewed text → real target delivery is **not yet accepted**.
|
||||
|
||||
**Runtime licensing remains unresolved.** Redux weights are CC-BY-4.0, but the
|
||||
observed Kestrel kernel runtime requires separate permission. We are checking with
|
||||
the vendor. Current native-only packages do **not** bundle or download that
|
||||
runtime; an independently authorized Python environment must be supplied.
|
||||
See [third-party notes](docs/third-party.md). No GitHub release is published yet.
|
||||
**Inference runtime:** Redux weights are CC-BY-4.0 and run locally through the
|
||||
`moondream` Python package (its Kestrel runtime states that local inference is free
|
||||
and needs no API key). The build and tests never download it; the installer or you
|
||||
install it from PyPI into a Python environment. See [third-party notes](docs/third-party.md).
|
||||
No GitHub release is published yet.
|
||||
|
||||
## Controls
|
||||
|
||||
@@ -89,7 +89,7 @@ ctest --test-dir build --output-on-failure
|
||||
|
||||
Default build has no hardware backends. It never downloads packages/models or
|
||||
initializes SteamVR, a microphone or input injection. Native dependencies and
|
||||
explicit launch/check commands are documented in [the POC guide](docs/poc.md).
|
||||
explicit launch/check commands are documented in [the build guide](docs/build.md).
|
||||
`--version` uses an ISO-like UTC build timestamp (with Git hash when available),
|
||||
not a numbered release; use that same tag when packaging the binary.
|
||||
|
||||
@@ -110,30 +110,28 @@ Autolaunch is opt-in. An installed desktop entry can be selected manually as a
|
||||
non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the
|
||||
panel on one Frame; registration alone did not show an entry in the first checked
|
||||
dashboard menu. See [packaging and lifecycle](docs/packaging.md).
|
||||
For a native-only install, menu-driven inference needs a separately authorized
|
||||
Python runtime and pinned model. Configure their absolute paths in
|
||||
For a native-only install, menu-driven inference needs a Python runtime you
|
||||
provide and the pinned model. Configure their absolute paths in
|
||||
`~/.config/frameyap/paths.conf` as described in the packaging guide; they are
|
||||
never fetched or bundled implicitly.
|
||||
|
||||
A pinned GitHub one-command route is implemented in `install.sh --version TAG`,
|
||||
but **do not advertise or run it as a working public installation until a vetted
|
||||
release exists**. Native-only artifacts support the overlay/checks without an
|
||||
end-user compiler; full bundled-ASR distribution awaits runtime permission.
|
||||
end-user compiler; a bundled-ASR distribution is not yet offered.
|
||||
|
||||
## Project map
|
||||
|
||||
- [POC guide](docs/poc.md): implemented boundaries, build, controls, explicit tests.
|
||||
- [Build guide](docs/build.md): implemented boundaries, build, controls, explicit tests.
|
||||
- [Design](docs/design.md): full target design; some features remain proposed.
|
||||
- [Installer design](docs/install-design.md) and [packaging](docs/packaging.md).
|
||||
- [Current POC observations](docs/evidence/poc-cpu-overlay-2026-09-24.md): measured
|
||||
CPU behavior and native installation checks, with acceptance limits.
|
||||
- [Resize / Auto Insert deployment](docs/evidence/auto-insert-deployment-2026-09-24.md):
|
||||
installed ARM64 version and narrow owned-target fixture; live speech-driven
|
||||
Auto Insert and physical resize acceptance remain open.
|
||||
- [Earlier API evidence](docs/evidence/frameyap-apis-2026-09-24.md) and
|
||||
[provenance](docs/provenance.md): historical investigation, not live authority.
|
||||
- [Overlay](docs/overlay.md), [worker protocol](docs/worker.md),
|
||||
[dependency/license inventory](docs/third-party.md).
|
||||
- [TODO](TODO.md): release plan and open work items.
|
||||
|
||||
Dated device-test records are kept locally (untracked) and are not authority for
|
||||
current device availability. Live microphone → reviewed text → target delivery has
|
||||
not been formally accepted; see Status above.
|
||||
|
||||
No recordings, transcripts, private logs, model weights or runtime binaries are
|
||||
committed. The worker boundary is intentionally small for forks experimenting
|
||||
|
||||
@@ -1,19 +1,182 @@
|
||||
# FrameYap TODO
|
||||
|
||||
- [ ] Resolve the Redux inference runtime license before further inference or a
|
||||
bundled release. Obtain written permission covering use and redistribution of
|
||||
M87 Labs Kestrel kernels (including the CPU payload), or scope an independently
|
||||
licensed runtime for the same weights. Inventory the other runtime dependencies'
|
||||
notices/licenses before publishing. The CC-BY-4.0 model weights do not grant
|
||||
permission to use or redistribute the proprietary runtime. See
|
||||
[third-party notes](docs/third-party.md).
|
||||
- [x] Validate a basic menu launch on Frame: OpenVR registration did **not** show
|
||||
FrameYap in the first checked dashboard menu; the user found FrameYap in Steam's
|
||||
**Non-Steam** section and reported that selecting it showed a panel and Quit
|
||||
worked. The server log showed a `--run` overlay connection followed by exit;
|
||||
no FrameYap process remained. The installed native-only package had no bundled
|
||||
runtime or model. This does not validate transcription, recording, text delivery,
|
||||
cold startup, or which shortcut discovery mechanism Steam used.
|
||||
- [ ] Verify the exact missing-runtime panel message and Steam's shortcut
|
||||
persistence across a normal restart (without restarting sessions just for the
|
||||
test). Confirm no runtime/model environment override before future live checks.
|
||||
Compiled from the 2026-09-24 holistic review, with owner decisions applied (see
|
||||
"Decisions" at the end). Items are sized to hand off individually. Size: S ≈ hours,
|
||||
M ≈ a day, L ≈ multi-day.
|
||||
|
||||
Guardrails from `AGENTS.md` apply to every item: offline default build, no implicit
|
||||
downloads, no automatic Enter, small verified commits, docs must state implemented
|
||||
vs proposed behavior honestly.
|
||||
|
||||
## A. First release (v0.1) blockers
|
||||
|
||||
- [x] **A1. Remove the runtime-license blocker notes.** Upstream's Kestrel README
|
||||
states local inference is free; blocker text removed from README/docs/scripts and
|
||||
replaced with a neutral dependency note in `docs/third-party.md`.
|
||||
- [x] **A2. Remove references to the unrelated app (kouseki).** (S)
|
||||
Docs and one `src/panel_surface.cpp` comment were cleaned; remaining check is a
|
||||
final grep. Keep the Inconsolata/OFL attribution and cite the
|
||||
upstream font source (googlefonts/Inconsolata) instead.
|
||||
*Done when:* `grep -ri kouseki` is empty and the font SHA/attribution remains.
|
||||
- [ ] **A3. Commit the pending `AGENTS.md` rename** ("Frame Dictation" →
|
||||
"FrameYap"). (S)
|
||||
- [ ] **A4. Inventory the remaining runtime dependencies' licenses.** (M)
|
||||
Torch CPU, numpy, tokenizers, SDL3, wayland, libxcb, FreeType (pick FTL or GPL
|
||||
option), compiler runtime / libc floor. Prerequisite for shipping a prebuilt
|
||||
archive that includes any of them.
|
||||
- [ ] **A5. Drop "POC" from the shipped surface.** (S) `--help` text, README,
|
||||
`scripts/stage-native-poc.py`, `CMakeLists.txt` messages,
|
||||
installer strings. v0.1 is a first small release.
|
||||
- [ ] **A6. Rewrite the README front.** (M) 3-line pitch, requirements, install,
|
||||
controls **table** (button → action), then a "Status / not yet validated" section.
|
||||
Today it reads as a lab notebook and Controls is a wall of text.
|
||||
- [x] **A7. Archive docs.** Dated evidence (`docs/evidence/`) and `provenance.md` moved
|
||||
to the untracked, gitignored `docs/archive/`; `poc.md` renamed `docs/build.md`.
|
||||
Remaining: skim `design.md`/`overlay.md`/`packaging.md` for stale "proposal" and
|
||||
hedging language before v0.1.
|
||||
|
||||
## B. Correctness / robustness
|
||||
|
||||
- [ ] **B1. A malformed transcript must not kill the worker.** (S)
|
||||
`src/runtime.cpp:141` → `session.reply()` → `literal_text()` throws on control
|
||||
characters or bad UTF-8; the catch at ~line 167 calls `worker.stop()` and
|
||||
`session.fail()`, unloading the model (reload can take up to 120 s). Treat it as a
|
||||
request-level error (like the `E` path: keep the worker, show "transcription
|
||||
failed", allow retry). Add a test with a control-character reply asserting the
|
||||
worker stays ready.
|
||||
- [ ] **B2. Make the C++ side engine-agnostic.** (S) `src/worker.cpp` hardcodes
|
||||
"moondream/torch" in the user-facing `F`/`I` errors. Use neutral wording or a
|
||||
worker-supplied message code. Prerequisite for C1.
|
||||
- [ ] **B3. "Close mic when idle" setting, default OFF.** (M)
|
||||
Default keeps the mic open while Ready (opening/closing per PTT causes an audio
|
||||
spike on the physical hardware, and it avoids first-syllable clipping). The
|
||||
setting closes it between clips for people who don't want a live device. Document
|
||||
the tradeoff (spike/latency) next to the toggle, in Settings and in the docs.
|
||||
Persist in `config.json`; add to the panel Settings tab.
|
||||
- [ ] **B4. Extract the interaction logic from `run()` and test it.** (L)
|
||||
`run()` in `src/runtime.cpp` is one ~220-line function of captured lambdas with no
|
||||
tests. Pull out a `Controller` (events + worker/audio/input interfaces → `Panel`)
|
||||
so PTT, cancel, quick phrases, auto-insert and error transitions are testable
|
||||
without hardware. B1 is the first regression test.
|
||||
|
||||
## C. Backends and model management
|
||||
|
||||
- [ ] **C1. Multiple ASR backends behind the worker protocol.** (L)
|
||||
Keep Redux as the default, allow additional backends (whisper.cpp, faster-whisper,
|
||||
sherpa-onnx Parakeet, …) as separate worker executables speaking the existing
|
||||
`Y`/`T`/`R`/`E` framing. Define a small backend manifest (id, display name,
|
||||
launcher, pinned model files + hashes + attribution, license text, CPU/GPU
|
||||
requirements) so nothing is hardcoded in C++ or `model_files.py`.
|
||||
*Done when:* Redux is expressed as a manifest, and a second backend can be added
|
||||
without touching `worker.cpp`/`runtime.cpp`.
|
||||
- [ ] **C2. Model/backend state and a chooser in the UI.** (L) Depends on C1 + B4.
|
||||
Settings page listing backends/models with state (not installed / installed and
|
||||
verified / loading / ready / failed), the active one marked, and selection that
|
||||
restarts the worker. Target user is non-technical: an **Install** button on the
|
||||
panel runs the installer's machine-readable mode (D2) as a child process and shows
|
||||
progress/errors in the panel, so nobody needs a terminal. The click is the consent;
|
||||
there are still no implicit or background downloads, and the panel states what
|
||||
will be downloaded and how large it is before it starts. Persist the choice in
|
||||
`config.json`.
|
||||
- [ ] **C3. Model status CLI.** (S) `frameyap --list-models` / `--check-model ID`
|
||||
(offline, hash-verifies installed files) so the UI and installer share one
|
||||
implementation.
|
||||
|
||||
## D. Installer
|
||||
|
||||
- [ ] **D1. Installer with a binary-or-source choice.** (L)
|
||||
`install.sh` offers *prebuilt archive* (checksummed) or *build from source*
|
||||
(checks toolchain/deps via `install-preflight.sh`, builds in a private dir), then
|
||||
continues automatically through install after the user's choices. Retain rollback,
|
||||
idempotency, no sudo, no Steam AppID, opt-in autolaunch.
|
||||
- [ ] **D2. Model-agnostic, attended-or-unattended operation.** (M)
|
||||
Every prompt has a flag (`--mode binary|source`, `--backend ID`, `--model-dir`,
|
||||
`--yes`, `--autolaunch`/`--no-autolaunch`, `--without-model`, `--print-plan`,
|
||||
`--json` output) so a model/agent can run it non-interactively; interactive
|
||||
prompts only run on a TTY and print the equivalent flags they chose. Exit codes
|
||||
and messages must be machine-readable.
|
||||
- [ ] **D3. Publish a first prebuilt ARM64 archive.** (M) Depends on A4. Follow the
|
||||
release checklist in `docs/packaging.md`; do not advertise the one-command route
|
||||
until the archive and its checksum are actually published and tested from a clean
|
||||
account.
|
||||
|
||||
## E. Naming and versioning
|
||||
|
||||
- [x] **E1. Name: keep "FrameYap" for v0.1.** Frame (the hardware) + yap (speech)
|
||||
says what it is; no rename churn before the first tag. If a hardware-neutral
|
||||
project name is wanted later (e.g. plain "Yap", with FrameYap as the Steam Frame
|
||||
front end), decide it before the cross-window work in G, not now.
|
||||
- [ ] **E2. Rename/explain UI terms.** (S) Delivery actions become **Type** (text +
|
||||
space) and **Type + Enter**; align overlay buttons, `--check-controls` output,
|
||||
README, help text and `docs/overlay.md`, and keep `UiAction::Enter` internal only
|
||||
if labels are consistent. "Quick chat" → "Quick phrases". Add one-line Settings
|
||||
explanations for "Hold Quit" and "Lasers anytime". Explain the "Parakeet Redux"
|
||||
vs `moondream` naming once in `docs/worker.md`.
|
||||
- [ ] **E3. Version scheme `MAJOR.MINOR.YYYYMMDDHHMM`.** (S)
|
||||
e.g. `0.1.202609241530`: valid semver (numeric patch, no leading zeros), sorts
|
||||
correctly, keeps the build date visible, URL/filename-safe. The version field is
|
||||
just that string. The old `-gHASH` (which commit) and `-dirty` (uncommitted
|
||||
changes) suffixes were only for telling developer builds apart, so they move out
|
||||
of the version: `--version` prints `frameyap 0.1.202609241530`, and for a dev
|
||||
build adds a second line like `git abc12345 (uncommitted changes)`. Release
|
||||
archives are built from a clean tag, so users never see it. Update the CMake
|
||||
version regex/`FRAMEYAP_VERSION`, `tests/cli.cmake`, `package-release.py`,
|
||||
installer version checks and docs. `SOURCE_DATE_EPOCH` still drives the
|
||||
timestamp. Tag releases `v0.1.<timestamp>`.
|
||||
|
||||
## F. Code structure (non-urgent)
|
||||
|
||||
- [ ] **F1. Move `--check-*` diagnostics out of `main.cpp`** (S) — ~70 lines of
|
||||
inline UI plus hand-rolled per-mode argument checks; use a `check.cpp` and a
|
||||
table-driven option parser.
|
||||
- [ ] **F2. Split `overlay.cpp`'s `Impl`** (M) — ~40 loosely related members (drag
|
||||
state, save-failure flags, counters, pose caches): separate drag, persistence and
|
||||
diagnostics.
|
||||
|
||||
## G. Other windows (post-release)
|
||||
|
||||
Text delivery already works into a WezTerm window on Frame (owner-tested; this is
|
||||
not the recorded live acceptance in H). Deeper integration of other windows with
|
||||
this app is future design and out of scope for v0.1.
|
||||
|
||||
- [ ] **G1. Validate browser text fields via the Gamescope input path.** (M)
|
||||
Highest-priority target. Record which fields accept Type / Type + Enter (plain
|
||||
inputs, textareas, rich editors, password fields should be expected to differ) and
|
||||
document the results honestly.
|
||||
- [ ] **G2. Later, if needed:** a KDE desktop-mode backend behind `DeliveryLease`,
|
||||
and a uinput backend as a last resort. Not scheduled; uinput needs `/dev/uinput`
|
||||
access and types with no focus check, which conflicts with the project's
|
||||
no-sudo/udev rule and per-window authorization.
|
||||
|
||||
## H. Carried over from the earlier TODO (hardware validation)
|
||||
|
||||
- [x] Basic menu launch on Frame: the user found FrameYap in Steam's **Non-Steam**
|
||||
section; the panel showed and Quit worked. This does not validate transcription,
|
||||
recording, text delivery, cold startup, or which shortcut discovery mechanism Steam
|
||||
used.
|
||||
- [ ] Verify the exact missing-runtime panel message and Steam's shortcut persistence
|
||||
across a normal restart (without restarting sessions just for the test). Confirm no
|
||||
runtime/model environment override before future live checks.
|
||||
- [ ] Live acceptance on Frame: microphone → reviewed text → real target delivery,
|
||||
Auto Insert with speech, physical resize.
|
||||
|
||||
---
|
||||
|
||||
## Decisions
|
||||
|
||||
- Redux runtime: treated as usable for local inference per upstream's Kestrel
|
||||
README; not bundled in our archives (A1 done).
|
||||
- Multiple backends + a model chooser UI: wanted (C1–C3). The panel can trigger the
|
||||
install on an explicit click, for non-technical users.
|
||||
- Name: keep FrameYap. v0.1 is a first small release, not a POC.
|
||||
- Close-mic-when-idle: setting, default off (B3).
|
||||
- Labels: Type / Type + Enter (E2).
|
||||
- Installer: binary or source, continues automatically after choices, fully flag-
|
||||
driven for agent use (D1–D2).
|
||||
- Version: `MAJOR.MINOR.YYYYMMDDHHMM`, git hash only in dev-build `--version` output
|
||||
(E3).
|
||||
- Other windows: browser text fields first (G1); deeper integration is future design.
|
||||
|
||||
## Open questions
|
||||
|
||||
None currently blocking.
|
||||
+10
-16
@@ -1,4 +1,4 @@
|
||||
# FrameYap POC: implementation and validation
|
||||
# Build, scope and validation
|
||||
|
||||
This is a standalone native application, not a plugin. Default builds/tests never
|
||||
initialize OpenVR, open a microphone, run ASR, download files or inject input.
|
||||
@@ -67,17 +67,12 @@ This is a compact prototype panel, not yet the proposed polished miniature statu
|
||||
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
|
||||
impact and target application compatibility require further headset work.
|
||||
|
||||
## Critical runtime licensing boundary
|
||||
## Inference runtime
|
||||
|
||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0.
|
||||
The installed **kestrel-kernels 0.7.0** license is different: proprietary, requiring
|
||||
an M87 Labs written agreement for use; copying/redistribution depends on that
|
||||
agreement. PyPI availability is not permission. See [third-party notes](third-party.md).
|
||||
|
||||
Do not publish a bundled Redux runtime, imply a public release is ready, or rerun
|
||||
inference while applicable permission is unresolved. Our adapter is implemented;
|
||||
that does not resolve distribution rights. An alternative runtime would be a
|
||||
separately scoped and independently licensed implementation—not a silent model swap.
|
||||
Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0. Inference
|
||||
runs through the `moondream` Python package and its Kestrel runtime, which you
|
||||
install in your own environment; builds/tests never fetch it. See
|
||||
[third-party notes](third-party.md).
|
||||
|
||||
## Developer native build
|
||||
|
||||
@@ -126,7 +121,7 @@ an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or
|
||||
`~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files
|
||||
are edited and no runtime/session restart is performed.
|
||||
|
||||
## Deliberate launch, once runtime permission is settled
|
||||
## Deliberate launch
|
||||
|
||||
Explicit setup downloads only the pinned, openly licensed model:
|
||||
|
||||
@@ -137,12 +132,12 @@ python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model"
|
||||
`fetch-model.py` is idempotent for matching hashes, rejects existing mismatched
|
||||
files, and is never invoked by build/tests or first utterance.
|
||||
|
||||
Provide your independently authorized CPU Python environment (moondream 2.4.0,
|
||||
Provide your own CPU Python environment (moondream 2.4.0,
|
||||
kestrel 0.8.0) and local weights:
|
||||
|
||||
```sh
|
||||
./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \
|
||||
--python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
|
||||
--python /path/to/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
|
||||
--model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head
|
||||
```
|
||||
|
||||
@@ -161,5 +156,4 @@ sending existing users' audio to a service.
|
||||
`scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV;
|
||||
no recording, device input or automatic download. `--show-text` is a separate
|
||||
explicit disclosure of that fixture's transcript. Use only with an authorized
|
||||
runtime. See the dated [POC record](evidence/poc-cpu-overlay-2026-09-24.md) for
|
||||
measurements and their limits; they are not headset/performance acceptance.
|
||||
runtime. Benchmark numbers from earlier trials are not headset/performance acceptance.
|
||||
+9
-13
@@ -1,14 +1,12 @@
|
||||
# FrameYap native dictation overlay — proposal
|
||||
|
||||
Standalone project design; see
|
||||
[provenance](provenance.md) and [historical Frame probes](evidence/frameyap-apis-2026-09-24.md).
|
||||
This document is the full target design, not a blanket implementation claim.
|
||||
The native POC now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||
Standalone project design. This document is the full target design, not a blanket
|
||||
implementation claim. The native app now implements overlay/actions, bounded SDL3 capture, a persistent
|
||||
Redux adapter, review-first Gamescope insertion and a user-local archive installer.
|
||||
Opt-in conservative Xwayland focus tracking is implemented locally but not yet
|
||||
accepted for live automatic typing on Frame. Polished status-chip UX and full hardware
|
||||
acceptance remain proposed. See [current scope](poc.md), [device observations](evidence/poc-cpu-overlay-2026-09-24.md)
|
||||
and [runtime licensing boundary](third-party.md).
|
||||
acceptance remain proposed. See [build and scope](build.md) and [third-party notes](third-party.md).
|
||||
Future work is tracked in [TODO.md](../TODO.md).
|
||||
|
||||
## Recommendation
|
||||
|
||||
@@ -105,11 +103,9 @@ not an application-rendered hand-pose animation loop. Do not promise a particula
|
||||
GPU cost until measured.
|
||||
|
||||
A scene renderer's canvas/MSDF resources are not an OpenVR overlay backend and
|
||||
are not imported here. The Inconsolata TTF used by kouseki is independently
|
||||
bundled under its retained OFL; the panel renderer is original FrameYap code.
|
||||
The neon HUD frame is a visual reference, not an engine dependency. Further
|
||||
source/asset reuse requires an explicit license-reviewed extraction, never a
|
||||
runtime path into the engine checkout.
|
||||
are not imported here. The bundled Inconsolata TTF is under its retained OFL; the panel renderer is
|
||||
original FrameYap code. Further source/asset reuse requires an explicit
|
||||
license-reviewed extraction, never a runtime path into another project's checkout.
|
||||
|
||||
### Controller bindings
|
||||
|
||||
@@ -261,7 +257,7 @@ processing timeout. No shell commands in IPC and no input authority in the worke
|
||||
no CUDA device/runtime assumption. Do not copy the desktop's x86 venv.
|
||||
- Weights are ~178 MB; Torch, kernels, temporary conversion and activations mean
|
||||
install size/RSS will be larger. Measure cold load, peak RSS and package size.
|
||||
Review runtime redistribution licensing separately from model attribution.
|
||||
Keep runtime notices separate from model attribution.
|
||||
|
||||
Microphone access does not mute VRChat or any other social-voice app. Shared
|
||||
PipeWire capture may let both hear the same utterance; the overlay must not claim
|
||||
@@ -312,4 +308,4 @@ Hardware-free tests should cover state transitions, bounded PCM/transcripts,
|
||||
worker framing/timeout/cancellation, duplicate/stale replies and focus generations
|
||||
with fakes. Separate opt-in Frame checks cover protocol availability, owned-window
|
||||
insertion, microphones, inference and overlays. No fixture establishes physical
|
||||
headset acceptance. See [provenance](provenance.md) for historical source anchors.
|
||||
headset acceptance.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Anchored resize / Auto Insert deployment — 2026-09-24
|
||||
|
||||
Historical observation from an explicitly coordinated Frame-only install and
|
||||
owned-target fixture. It does not authorize further microphone tests or prove
|
||||
actual speech-to-text Auto Insert, grip usability, arbitrary app delivery, or
|
||||
human headset acceptance.
|
||||
|
||||
- Source milestones: `19a3db7` adds session-only corner-anchored resize;
|
||||
`c5e925c` adds default-off, fail-closed Xwayland Auto Insert.
|
||||
- Local default/UI/native hardware-free suites passed: 13/13, 14/14, 18/18.
|
||||
Native Xvfb focus-loss test also passed 12 repeated runs. ARM64 native build
|
||||
from the `c5e925c` Git archive passed hardware-free CTest 18/18 on Frame.
|
||||
No normal test initialized OpenVR, recorded audio, or injected input.
|
||||
- Native-only, external-runtime archive version
|
||||
`2026-09-24T201841Z-gc5e925cb` was staged with explicit licensed SDL/OpenVR
|
||||
files. SHA-256 archive:
|
||||
`c6bef87ec86fc0903247be714aec9c5784c3fa5ac788858dad9f50f1958d6f22`.
|
||||
The pre-install checksum check succeeded; system libxcb was available and no
|
||||
staged binary dependencies were unresolved.
|
||||
- No FrameYap process was running when the installer checked. The idempotent
|
||||
user-local installer selected the new version, retained previous
|
||||
`2026-09-24T191513Z-g3c03f88f`, and backed up the existing config. Installed
|
||||
`--version` matched the archive version; installed/staged binary SHA-256 both
|
||||
`973fd119701093470b2ee3560df7e8049d968e1bab49f63f5242dabf055aa4ec`.
|
||||
Auto Insert remained `false`; no FrameYap process was left running.
|
||||
- A separate temporary ARM64 test helper used the new `FocusGuard` and existing
|
||||
Gamescope IME `TextInput` against one project-owned Xterm receiver on display
|
||||
`:0`. It checked the expected window ID against X keyboard focus, X active
|
||||
window and Gamescope focused-window both before arming and after acquiring
|
||||
the IME lease. The helper queued the literal ASCII fixture `fy-safe-probe`
|
||||
without Enter; the private, disposable terminal receiver read **exactly 13
|
||||
bytes**, matching the fixture. Both temporary processes exited. No microphone,
|
||||
ASR model, arbitrary destination, speech transcript, or app UI Auto Insert
|
||||
workflow was exercised by that helper.
|
||||
|
||||
Compositor processing and the owned terminal's exact bytes support this narrow
|
||||
fixture-delivery claim. The full microphone → worker → focus guard → Auto Insert
|
||||
flow remains untested on Frame, as do physical resize interactions, input during
|
||||
focus changes in actual games, and native Wayland targets. X focus checks are
|
||||
not atomic with delivery; review remains the fallback. No public release or
|
||||
bundled ASR runtime was published.
|
||||
@@ -1,58 +0,0 @@
|
||||
# Frame bindings and explicit Enter deployment — 2026-09-24
|
||||
|
||||
Historical evidence, not current device status or permission for further live tests.
|
||||
|
||||
## Implemented and checked
|
||||
|
||||
User approved right X hold-to-talk, B Cancel, A Insert + space, Y pending Insert
|
||||
+ Enter; existing grip gestures remain. A Bindings tab shows OpenVR origin names
|
||||
and requests the runtime binding editor. Insert ensures a trailing ASCII space;
|
||||
explicit Enter consumes pending review, queues its text, releases that IME lease,
|
||||
then acquires a new lease for Submit. Failed text never proceeds to Submit; an
|
||||
uncertain send is not retried. No transcription completion auto-submits.
|
||||
|
||||
Read-only inspection of the installed Frame controller profile confirmed A/B/X/Y
|
||||
on the right and a D-pad on the left. It references runtime-owned left/right SVG
|
||||
diagrams for SteamVR's editor. FrameYap does not redistribute those diagrams or
|
||||
claim a generic OpenVR button-glyph API. Editor display is not yet human-accepted.
|
||||
|
||||
Local default CTest: 13/13. Local native hardware-free CTest: 16/16. Synthetic
|
||||
Bindings canvas was inspected for layout. These do not prove actual input delivery.
|
||||
|
||||
## Native install
|
||||
|
||||
Committed application source: `7a3a5451`, including wrist geometry/config work
|
||||
`569ba13` and `e61bf73`. A Git source archive (excluding unrelated dirty workspace
|
||||
files) built on Frame with existing standalone dependencies; ARM64 native
|
||||
hardware-free CTest passed 16/16. No runtime/model dependencies were downloaded.
|
||||
|
||||
Installed and verified version: `2026-09-24T184918Z-g7a3a5451`.
|
||||
Native-only archive SHA-256:
|
||||
`9fa7991298731b27fc6dbf6d805167ee006e0b8ab93c2b859e2fad61007832a5`.
|
||||
`current` selected this version and `previous` retained
|
||||
`2026-09-24T183121Z-g69f5de5a`. Installer checked the supplied checksum. Installed
|
||||
`--version` matched; `ldd` resolved bundled SDL/OpenVR and system dependencies.
|
||||
The authorized runtime/model paths configuration hash remained unchanged.
|
||||
|
||||
The installer preserved existing disabled action values. With the user's approval
|
||||
of the new controls, a separate exact-byte-backed-up config update enabled the
|
||||
X/B/A/Y mappings and returned experimental input priority to normal. Lasers anytime
|
||||
was already off. Saved `right-wrist` mount was retained; wrist config now uses
|
||||
0.30 m width, zero roll and (0, 0.18, 0.089) offset.
|
||||
|
||||
A running FrameYap process appeared during configuration; only its exact verified
|
||||
installed executable PID was terminated gracefully under the restart approval.
|
||||
No SteamVR, SSH or user-session process was stopped. The restarted process's
|
||||
`/proc/PID/exe` matched the new installed version. Startup reported normal priority,
|
||||
Lasers anytime off and panel shown. This is API/process evidence, not headset
|
||||
visibility or delivered-input acceptance. A normal app launch warms its configured
|
||||
worker and opens/discards idle microphone samples; no recording or input-delivery
|
||||
test was initiated by the assistant.
|
||||
|
||||
## Open acceptance
|
||||
|
||||
The wearer then reported **"wrist is backwards"**. Wrist orientation is therefore
|
||||
not accepted; whether this means inverted text or a panel facing away was awaiting
|
||||
clarification at this record. Do not count passing pose tests as comfort/orientation
|
||||
acceptance. Actual B/A/Y delivery, text-plus-Enter ordering at a real target, and
|
||||
SteamVR binding-editor behavior remain human-led checks.
|
||||
@@ -1,50 +0,0 @@
|
||||
# Debug diagnostics and right-wrist correction — 2026-09-24
|
||||
|
||||
Historical observation, not current availability or permission to run tests.
|
||||
|
||||
The wearer clarified that the right-wrist panel faced away (its back was visible).
|
||||
Commit `29f3729d` reverses right-wrist panel-right and panel-front, preserving
|
||||
panel-up, center and size. Offline pose tests check both hands, orthonormality
|
||||
and positive determinant over several rolls; this is not physical acceptance.
|
||||
|
||||
The wearer also reported repeated generic transcription failures. The retained
|
||||
app log contained startup status only. Inspection found that the Python worker
|
||||
replaced every request exception with `transcription failed`, and both the native
|
||||
parent and Python suppressed stderr. No underlying exception had been retained;
|
||||
the root cause therefore remained unknown. The user declined a public-clip
|
||||
inference trial and chose to retry speech themselves after diagnostics were added.
|
||||
|
||||
Commit `6592e1af` adds default-safe stage/category errors and explicitly opt-in
|
||||
advanced debugging through config/Settings. Full worker stdout/stderr, tracebacks
|
||||
and transcripts may appear in private bounded logs when enabled; no raw clip
|
||||
archive is created. The native receiver allowlists error labels before showing
|
||||
or logging the non-debug error. Settings changes restart the owned worker and
|
||||
discard current work; defaults remain off. See [diagnostic policy](../worker.md#advanced-debugging).
|
||||
|
||||
## Checks and deployment
|
||||
|
||||
- Local default CTest: 13/13; local native hardware-free CTest: 16/16.
|
||||
- Inspected a synthetic Settings canvas containing the toggle and privacy warning.
|
||||
- ARM64 native build from Git archive `6592e1af`: hardware-free CTest 16/16.
|
||||
Includes fake-child stderr/protocol isolation, log bounds/rotation/permissions,
|
||||
unsafe path refusal, opt-in/off behavior and safe-error privacy tests.
|
||||
- Installed version: `2026-09-24T190806Z-g6592e1af`.
|
||||
- Native-only archive SHA-256:
|
||||
`eac25ae6e36970301e5cb67614eaa4053d20b79711397b462e2caf1359c09709`.
|
||||
- Checksum-verified installer selected the version above; installed `--version`
|
||||
and the restarted process's executable path matched. Previous version retained:
|
||||
`2026-09-24T184918Z-g7a3a5451`. The intermediate wrist-only archive was not installed.
|
||||
- Runtime/model paths configuration hash unchanged. Installer backed up the
|
||||
previous config, added `advanced_debug: false`, and retained normal priority,
|
||||
approved X/B/A/Y mappings and the wrist size/offset settings.
|
||||
- The saved mount was now `left-wrist` (changed since the earlier right-wrist
|
||||
observation); installation preserved it rather than choosing for the wearer.
|
||||
- No FrameYap process was present immediately before this install. Relaunched
|
||||
only FrameYap. No SteamVR/SSH/user-session process was stopped. Startup reported
|
||||
normal priority and Lasers anytime off; this does not establish panel visibility.
|
||||
|
||||
No public-fixture inference, assistant-triggered microphone recording or input
|
||||
injection was performed. Normal authorized app startup warms the configured worker
|
||||
and opens/discards idle microphone samples. Advanced logging was left **off** for
|
||||
the wearer to enable. Actual failure diagnosis, Settings-toggle behavior on Frame,
|
||||
and physical wrist acceptance still require the user's next trial.
|
||||
@@ -1,67 +0,0 @@
|
||||
# Experimental overlay input priority — 2026-09-24
|
||||
|
||||
Source commit: `69f5de5a`.
|
||||
Installed native ARM64 version: `2026-09-24T183121Z-g69f5de5a`.
|
||||
|
||||
The user confirmed the Vulkan flicker fix, reported controller actions becoming
|
||||
unavailable in system laser/dashboard interaction states, and agreed to try
|
||||
OpenVR's experimental priority mechanism. The investigation concerns action
|
||||
delivery across modes, independent of any specific physical button.
|
||||
|
||||
## Implementation and preparation
|
||||
|
||||
FrameYap config `input_priority` accepts `normal` (default) or `experimental`.
|
||||
Experimental requests `k_nActionSetOverlayGlobalPriorityMin` (`0x01000000`) for
|
||||
the existing action set through `UpdateActionState`, including while system
|
||||
laser mode or the dashboard is active. The request covers the sources bound to
|
||||
FrameYap actions; bindings and action paths are unchanged. The installer
|
||||
preserves the selection and backs up invalid config before repair.
|
||||
|
||||
SteamVR separately permits global input priority through its Developer setting.
|
||||
The current Frame's saved `steamvr.globalActionSetPriority` was already true;
|
||||
its runtime default was false. This was a targeted file read, not an effective
|
||||
runtime API query. FrameYap now reads this permission through `IVRSettings` and
|
||||
reports it separately from its priority request at startup. It does not write
|
||||
the SteamVR setting.
|
||||
|
||||
The controls-only diagnostic reports all six actions' activity, press state and
|
||||
pose/role acceptance, alongside dashboard visibility, our Lasers anytime flag,
|
||||
`IsInputAvailable`, panel visibility and the application focus gate. This can
|
||||
distinguish runtime action inactivity from application rejection. The laser flag
|
||||
records our request; it is not a detector for all system laser activation.
|
||||
|
||||
## Verification and deployment
|
||||
|
||||
- Local default build and CTest: 13/13 passed.
|
||||
- Local native build and hardware-free CTest: 16/16 passed. The fake Wayland
|
||||
test used sandbox escalation to bind its local Unix socket.
|
||||
- Frame ARM64 Release build from a checksum-verified archive of the source
|
||||
commit above: 16/16 hardware-free CTest checks passed.
|
||||
- The package was installed through the existing installer. Installed
|
||||
`--version` matched and its executable hash matched the staged executable.
|
||||
- FrameYap's config was set to `experimental` under the application lock, with
|
||||
an exact backup; all other config values were verified unchanged by that edit.
|
||||
- No OpenVR probe, microphone capture, inference or text delivery was started.
|
||||
No application, SSH or session process was stopped.
|
||||
|
||||
Native archive SHA-256:
|
||||
`40a9445427469286e8997563bc5598ace7769ba4c13b1a9290911c821d0e419c`.
|
||||
Installed executable SHA-256:
|
||||
`da842efcad926660efc66c03d843592573380b30e563e797df2944a0d6d77da7`.
|
||||
|
||||
The prior Vulkan version `2026-09-24T181314Z-g4c043c99` is retained. To end the
|
||||
priority experiment on the new build, set `input_priority` to `normal` and
|
||||
relaunch. To roll back to the older binary, first restore the pre-upgrade config
|
||||
backup `config.json.backup-sih_efaa`: that binary predates the new config key.
|
||||
The separate `config.json.backup-priority-2026-09-24T183121Z-g69f5de5a` preserves
|
||||
the upgraded config before selecting experimental priority.
|
||||
|
||||
## Acceptance boundary
|
||||
|
||||
The user was told the new build is ready for headset testing. Compare the same
|
||||
bindings with the dashboard open/closed and Lasers anytime on/off; check both
|
||||
pointer interaction and controller press/release, including mode transitions.
|
||||
Higher priority may consume input used by a scene or the dashboard. This record
|
||||
does not claim that actions now arrive in every state or that simultaneous
|
||||
dashboard interaction is accepted. Dated deployment evidence does not establish
|
||||
future device availability or authorize future live runs.
|
||||
@@ -1,29 +0,0 @@
|
||||
# Guided Frame focus probe — 2026-09-24
|
||||
|
||||
Historical observation, not authorization for later live input or proof of target safety.
|
||||
The wearer consented to an opt-in, disposable-target focus trial. Two project-owned
|
||||
`xmessage` windows were created on Xwayland `:0` for 65 seconds, then closed by
|
||||
the owning finite command. No microphone, transcript, input injection or SteamVR
|
||||
session cleanup was used. Other SSH/user processes were left untouched.
|
||||
|
||||
An earlier read-only snapshot showed `/tmp/.X11-unix/X0` and `X1`, Gamescope
|
||||
socket `gamescope-0`, and `_NET_ACTIVE_WINDOW` matching
|
||||
`GAMESCOPE_FOCUSED_WINDOW` on `:0`. Brief snapshots of an owned test window
|
||||
also showed disagreement between these root properties, so either property
|
||||
alone is insufficient to authorize automatic typing.
|
||||
|
||||
The wearer selected A/B and reported doing several focus changes. Window A was
|
||||
`0x3e00022`, B was `0x4200022` in that run. A 120 ms sampled observer saw
|
||||
X keyboard focus, `_NET_ACTIVE_WINDOW` and `GAMESCOPE_FOCUSED_WINDOW` agree at
|
||||
A, change to B, then return to A several times. A transition to a third window
|
||||
`0x3c00003` was also observed. At other moments the X keyboard-focus/active
|
||||
window IDs changed while Gamescope's focused-window property still named A.
|
||||
The root property is a window ID encoded as CARDINAL, not a generation token.
|
||||
|
||||
These samples establish *observable correlation for these two owned Xwayland
|
||||
windows*, not continuous seat identity across all targets, a focus-loss event
|
||||
stream, transcript quality, delivered input, native Wayland coverage or headset
|
||||
acceptance. The offline fail-closed observer subscribes to X property changes
|
||||
and focus-out on the exact armed window; its own live behavior and actual IME
|
||||
delivery still require a separate disposable-target validation. There is still
|
||||
a non-atomic gap between final focus check and compositor input processing.
|
||||
@@ -1,44 +0,0 @@
|
||||
# Historical Steam Frame API probes — 2026-09-24
|
||||
|
||||
These observations were made on one device and SteamOS build, not rerun as part
|
||||
of this repository's scaffold. They do not prove current availability or headset
|
||||
acceptance. No probe code, user audio, credentials or runtime binaries are shipped.
|
||||
See the [design](../design.md) for proposed implementation and safety gates.
|
||||
|
||||
## Environment
|
||||
|
||||
SteamOS 0.4.0 (VR variant), AArch64, glibc 2.39, Python 3.12.3;
|
||||
Gamescope 3.16.28-2. Linux ARM64 package availability is not evidence of
|
||||
inference speed or compatibility with this device.
|
||||
|
||||
## Observations and limits
|
||||
|
||||
| Surface | Historical observation | Limit |
|
||||
| --- | --- | --- |
|
||||
| OpenVR | Native `VRApplication_Overlay` initialization succeeded; `IVROverlay_028`, `IVRInput_011`, `IVRSystem_026` and `IVRApplications_008` were accepted. | No rendered overlay, binding or autolaunch test. |
|
||||
| Gamescope IME | `gamescope_input_method_manager` v3 advertised; v2 binding accepted and returned `done(serial=1)`. | Discovery is not delivered input. |
|
||||
| Unicode delivery | A separate disposable X11/XIM receiver on the device accepted an exact mixed-script Unicode fixture via `set_string` and `commit` after checking focus on its owned window. | One Xwayland receiver, not general games, native Wayland or PC-streamed targets. No Enter was sent. |
|
||||
| libei | Sender connected; seat advertised KEYBOARD, not TEXT. | No bound device or delivered key test. |
|
||||
| XTEST / uinput | XTEST advertised; `/dev/uinput` was openable by the test account. | No injected XTEST key or virtual device. |
|
||||
| Audio | PipeWire input device enumerated. | No microphone recording or model inference in this probe. |
|
||||
| Portal | Existing portal introspection exposed no RemoteDesktop, InputCapture, Clipboard, ScreenCast or GlobalShortcuts interface. | Not a guarantee for future OS releases. |
|
||||
|
||||
The Xwayland receiver was destroyed after the test. No packages, services or
|
||||
Steam settings were changed. No microphone audio was recorded. This is a
|
||||
historical API probe, not a live availability check or install smoke test.
|
||||
|
||||
## Implementation cautions
|
||||
|
||||
Gamescope's [input-method protocol](https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml)
|
||||
is private and version-sensitive; see its [implementation](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/ime.cpp).
|
||||
The serial is not a verified focus-generation guard; a roundtrip is not a text
|
||||
consumption acknowledgement. The inspected [`gamescope-type` example](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/Apps/gamescope_type.c)
|
||||
interprets newline as Submit: do not use it as a transcript pipe.
|
||||
Generic virtual-keyboard and data-control Wayland globals were not advertised
|
||||
in this probe. Focus tracking and Steam keyboard coexistence need separate tests.
|
||||
|
||||
The proposed Redux model is pinned to revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c`; the weight file was
|
||||
177,774,490 bytes in an earlier local inspection. ARM64 Python 3.12 wheels
|
||||
appeared available, but no native model load or inference was performed on the
|
||||
Frame. Runtime redistribution terms require separate review.
|
||||
@@ -1,31 +0,0 @@
|
||||
# Free controller grab / layout lock deployment — 2026-09-24
|
||||
|
||||
Historical deployment evidence; physical interaction remains user-led acceptance.
|
||||
The user authorized continuing after the [raw-overlay incident analysis](raw-overlay-teardown-crash-2026-09-24.md), using controller/visual testing instead of another raw-overlay probe.
|
||||
|
||||
- Source: `0d4cfbfdb74aaaa8e56be844122ac51eeb8fe365`. Includes full controller-relative
|
||||
position/rotation, saved `lock_layout`, corrected top-left mask coordinates, and
|
||||
earlier gradient, clock/date and wrist-fade commits.
|
||||
- Local default suite: 15/15 passed. Local native suite: 20/20 passed.
|
||||
Native ARM64 Release build on Frame: 20/20 hardware-free tests passed.
|
||||
- Exact source archive SHA-256 (verified before extraction):
|
||||
`57aa17045ddfee0fc8f6f1bb2bd4cbf19b082cf62321edaded2fb40ebd19e9d2`.
|
||||
- Installed version: `2026-09-24T210207Z-g0d4cfbfd`.
|
||||
Native-only external-runtime package SHA-256:
|
||||
`85574b1645b68e57a609c3e46b8068613ac5bc64a25fbc8b7ed7c1e4bcb17f9a`.
|
||||
Installed/staged binary SHA-256 both matched:
|
||||
`4d256feb1b23b649e4720fb3ca11eab786229a445c22d4fc31b209ffe7e6b187`.
|
||||
- No existing FrameYap process was running at install. The managed installer
|
||||
retained rollback and backed up the existing config before filling missing
|
||||
preferences. No ASR runtime/model was bundled or downloaded.
|
||||
- Installed `--version` and the running executable path were verified after
|
||||
launching the normal Vulkan runtime (PID 78418 at that check). Startup reported
|
||||
dashboard visible, normal input priority, Lasers anytime off and panel-shown=N.
|
||||
The latter is not visual acceptance; the selected wrist's angle/tracking gate
|
||||
can keep it hidden. The user was asked to face the wrist toward them if needed.
|
||||
- No raw-overlay probe, deliberate recording or synthesized input was run during
|
||||
this deployment. No SteamVR, Gamescope, SSH or terminal session was restarted.
|
||||
|
||||
Requested human checks: corner-bracket hit/resize; free depth/rotation and stable
|
||||
release; Settings lock hides/disables both handles and unlock restores them.
|
||||
At this record's creation those checks were requested, not yet reported passed.
|
||||
@@ -1,44 +0,0 @@
|
||||
# External grab/scale handles — 2026-09-24
|
||||
|
||||
Historical observation from an explicitly authorized Frame install/relaunch.
|
||||
This record is not permission for later hardware runs and does not establish
|
||||
physical drag/release behavior or human headset acceptance.
|
||||
|
||||
- Source commit: `33533ba8a2c85ccef3a388da4c2155e7dbbc14e0`.
|
||||
- The user's Frame screenshot showed a thin grab underline below the terminal
|
||||
and an external lower-right corner bracket. FrameYap independently draws that
|
||||
layout in transparent RGBA margins; no Steam private UI code/assets are bundled.
|
||||
The screenshot remains outside Git.
|
||||
- Local default, FreeType UI and native hardware-free suites passed 14/14,
|
||||
15/15 and 19/19 respectively. A synthetic panel preview was visually inspected;
|
||||
renderer tests check transparent gaps, opaque handle centers, antialiased alpha,
|
||||
cursor ownership and suppression of stale control approvals.
|
||||
- Controller-ray math tests cover stationary stability, scaling on independent
|
||||
axes, rotated/relative geometry, out-of-bounds intersections and invalid rays.
|
||||
The original per-event resize feedback path was removed. Grab currently
|
||||
translates in the panel plane, not depth or orientation.
|
||||
- Exact source archive SHA-256:
|
||||
`a8b8d4387de1fdd5cb1031e8905a6d6766e89616ceff3b62f902ea192a70e07b`.
|
||||
The checksum was verified before extraction on Frame. Native ARM64 Release
|
||||
build and hardware-free CTest passed 19/19; no test initialized OpenVR,
|
||||
recorded audio or injected input.
|
||||
- Installed version: `2026-09-24T203812Z-g33533ba8`.
|
||||
Native-only external-runtime package SHA-256:
|
||||
`922c49de4188cba2e58bae829c02d5e9b8af3a33dba5bc6619b469389c8e73da`.
|
||||
No model or ASR runtime was bundled/downloaded. Staged `ldd` had no unresolved
|
||||
dependencies. Installed and staged binary SHA-256 both matched:
|
||||
`d77e4bec5b585a12a06885086c90c7d0e3b5b447d857b39a7f8586dfa7ce8518`.
|
||||
- No old FrameYap binary was running at installation time. The managed installer
|
||||
selected the new version and retained `2026-09-24T201841Z-gc5e925cb` as previous.
|
||||
Installed `--version` and the relaunched process's executable path were verified.
|
||||
Only FrameYap was launched; no SSH, SteamVR or terminal session was stopped.
|
||||
- Startup reported normal input priority, Lasers anytime off, and
|
||||
`panel-shown=Y`. This establishes successful initialization/show request,
|
||||
including the new intersection-mask call, not visual/physical acceptance.
|
||||
The normal runtime was left running for user testing; no recording or text
|
||||
delivery was deliberately triggered by this check.
|
||||
|
||||
Remaining acceptance: actual source-device reporting, grab/scale tracking,
|
||||
release outside the mask, wrist/head behavior, transparency and comfortable
|
||||
hit-target sizes. When legacy trigger release is not observable, dragging cancels
|
||||
on loss of hover rather than relying on an outside MouseButtonUp event.
|
||||
@@ -1,105 +0,0 @@
|
||||
# Overlay flicker: source review and upstream reports — 2026-09-24
|
||||
|
||||
Local source review and public web research only. No Frame connection, OpenVR
|
||||
initialization, microphone capture or input delivery was performed for this
|
||||
investigation. No runtime fix or headset update is claimed. The earlier
|
||||
[controls-only observations](ui-click-flicker-2026-09-24.md) remain separate
|
||||
evidence; their device availability and permission do not carry forward.
|
||||
|
||||
## Does FrameYap recreate the panel on interaction?
|
||||
|
||||
The inspected `src/overlay.cpp` creates one OpenVR overlay in `Overlay::Impl`'s
|
||||
constructor. Its only `DestroyOverlay` call is in cleanup, including startup
|
||||
failure cleanup. Pointer handling, tab changes and `draw()` do not recreate that
|
||||
handle. `PanelSurface` retains one fixed-size RGBA vector; normal redraws replace
|
||||
its pixels, not its dimensions. This native panel is an OpenVR overlay, not an
|
||||
SDL/X11/Wayland desktop window.
|
||||
|
||||
`draw()` calls `SetOverlayRaw` when `PanelSurface::render()` reports changed
|
||||
content. Hover and button down do not invalidate the canvas; action clicks in
|
||||
`--check-controls` only log diagnostics. Tabs and placement notes can still
|
||||
repaint. In normal dictation, action-induced status changes repaint too.
|
||||
|
||||
The earlier event-counter trial recorded one show, zero hides and no hidden
|
||||
events during interaction. The earlier static-canvas trial nevertheless had a
|
||||
wearer report of whole-panel disappearance on action clicks. Thus application
|
||||
handle recreation is unsupported by the code, and uploads cannot yet explain
|
||||
all reported flicker. A compositor texture replacement or composition problem
|
||||
could look like window recreation while the API overlay remains alive; this is
|
||||
a hypothesis, not an observation of SteamVR internals.
|
||||
|
||||
## Relevant primary sources
|
||||
|
||||
- **OpenVR #772, April 2018:** a Linux C++ overlay author reported the entire
|
||||
overlay disappearing between `SetOverlayRaw` updates, with different behavior
|
||||
in the two eyes. Contributor Joe Ludwig advised using an OpenGL or Vulkan
|
||||
texture with `SetOverlayTexture` for frequent updates, describing substantial
|
||||
raw-upload latency and CPU/memory cost. This is a close match for redraw
|
||||
flashes, but is historical guidance, not a Frame measurement or confirmation
|
||||
that an overlay handle is destroyed.
|
||||
[Report and recommendation](https://github.com/ValveSoftware/openvr/issues/772#issuecomment-380539744).
|
||||
- **OpenVR #941, November 2018:** Ludwig reiterated that raw uploads are a poor
|
||||
video path and recommended a graphics texture. This corroborates the API
|
||||
recommendation; it is not an independent reproduction of our click-only case.
|
||||
[Maintainer response](https://github.com/ValveSoftware/openvr/issues/941#issuecomment-440004776).
|
||||
- **SteamVR 2.17.1 beta discussion, June 5, 2026:** Desktop+ developer
|
||||
`elvissteinjr` reported severe flickering with cursor override and the default
|
||||
cursor blob, plus problems with transparency and overlay ordering. FrameYap's
|
||||
inspected path does not use cursor override, so this is evidence of related
|
||||
compositor trouble, not an exact reproduction or a confirmed Frame bug.
|
||||
[Firsthand reports, comments 7 and 10](https://steamcommunity.com/app/250820/eventcomments/572665855469650962/).
|
||||
- **Valve's SteamVR 2.17 release notes, September 10, 2026:** include fixes for
|
||||
dashboard/overlay cursor visibility and `MinimalControlBar` handling. These
|
||||
establish intervening changes after the June report; they do not identify a
|
||||
fix for FrameYap. Record the actual Frame runtime build before comparing it
|
||||
with these reports. OpenVR SDK v2.15.6 does not identify the running SteamVR
|
||||
version.
|
||||
[Official announcement feed](https://steamcommunity.com/app/250820/announcements/?l=english).
|
||||
|
||||
The GitHub web viewer omitted issue comments during this review; the linked
|
||||
responses were checked through GitHub's public issues/comments API as well.
|
||||
|
||||
## What existing counters can and cannot establish
|
||||
|
||||
The pinned SDK defines `ImageLoaded` as completion of a raw/file image load,
|
||||
not overlay creation. It separately defines `OverlayCreated` and
|
||||
`OverlayDestroyed`. Shown/hidden events reflect API visibility, not proof that
|
||||
every headset frame contains the panel.
|
||||
[OpenVR v2.15.6 event definitions](https://github.com/ValveSoftware/openvr/blob/v2.15.6/headers/openvr.h#L853-L895).
|
||||
|
||||
Our existing totals do not timestamp each click, identify every hit target,
|
||||
record lifecycle events, or resolve the named overlay again. Consequently they
|
||||
cannot distinguish an internal compositor resource change from a render-order
|
||||
problem. A constant application handle alone would not distinguish these either.
|
||||
|
||||
## Next controlled comparison, proposed only
|
||||
|
||||
First extend the finite controls-only probe with monotonic timestamps, hit
|
||||
targets, raw-upload/image-load sequence numbers, and the current handle plus
|
||||
read-only `FindOverlay` results. Record created/destroyed events with their
|
||||
target handles: cursor or dashboard overlays must not be counted as FrameYap
|
||||
recreation. Keep diagnostics off the canvas. Record the runtime version and
|
||||
whether the flash affects one eye, both eyes, just the cursor or the whole panel.
|
||||
|
||||
Then compare one variable at a time with a contemporaneous wearer report:
|
||||
|
||||
| Trial | Purpose |
|
||||
| --- | --- |
|
||||
| Static canvas, default laser, diagnostic action clicks | Reproduce interaction without new raw uploads after startup settles. |
|
||||
| Same static canvas, only `HideLaserIntersection` enabled | Test whether the compositor's cursor blob participates; this deliberately removes cursor feedback. |
|
||||
| Same panel placed clear of dashboard surfaces | Test overlap/ordering without a renderer change. |
|
||||
| Scheduled content updates with no pointing or clicking | Test raw-image replacement independently of interaction. |
|
||||
| Same content updates through a persistent GPU texture | Compare the raw path with `SetOverlayTexture`, retaining and synchronizing the texture correctly. |
|
||||
|
||||
`HideLaserIntersection` suppresses the cursor blob; it does not disable mouse
|
||||
input. `VisibleInDashboard` permits visibility there, whereas
|
||||
`MakeOverlaysInteractiveIfVisible` activates global laser mode. These have
|
||||
different effects and should not be changed together to diagnose flicker.
|
||||
[Pinned flag definitions](https://github.com/ValveSoftware/openvr/blob/v2.15.6/headers/openvr.h#L3740-L3757).
|
||||
|
||||
A persistent GPU texture is a justified rendering experiment for content
|
||||
updates, but cannot be promised to fix a static overlay blinking on clicks.
|
||||
If static-click flicker survives the cursor/placement comparisons, the resulting
|
||||
minimal reproduction is useful for an upstream compositor report. No report has
|
||||
been submitted. Any native probe change still needs offline checks, authorized
|
||||
deployment/version verification and an opt-in human headset check.
|
||||
@@ -1,38 +0,0 @@
|
||||
# Captured PCM range rejection — 2026-09-24
|
||||
|
||||
Historical observations; not permission for additional recording or inference.
|
||||
|
||||
After the wearer enabled advanced debugging and retried speech, the private
|
||||
worker log showed successful requests followed by an inference-stage exception:
|
||||
`ValueError: PCM must contain finite samples in [-1, 1]`, raised by the runtime's
|
||||
`_float_pcm` validation. No captured speech or full private log is reproduced here.
|
||||
Both native submission and Python clip reading already rejected nonfinite values;
|
||||
finite amplitude overshoot was not bounded. The precise source of that overshoot
|
||||
(microphone gain, capture processing or resampling) was not measured.
|
||||
|
||||
Commit `019b813` saturates finite microphone samples to `[-1, 1]` after SDL
|
||||
conversion, preserving in-range samples and clip length. It does not rescale
|
||||
whole clips or accept NaN/infinity. `Worker::submit` independently validates the
|
||||
range before clip-file/IPC mutation. Fake-device and IPC regression tests cover
|
||||
both signs of overshoot, extreme finite values, exact boundaries, unchanged
|
||||
ordinary samples, nonfinite rejection and subsequent capture recovery.
|
||||
|
||||
Local default CTest passed 13/13; local native hardware-free CTest passed 16/16.
|
||||
The coordinated native Git snapshot also includes `3c03f88f`, which changes the
|
||||
Bindings button to open SteamVR's editor directly. Its source diff was reviewed
|
||||
before the combined build; no competing installer was run.
|
||||
|
||||
ARM64 native hardware-free CTest: 16/16. Installed/verified build:
|
||||
`2026-09-24T191513Z-g3c03f88f`.
|
||||
Native-only package SHA-256:
|
||||
`d8433bc6e9e84ee9b54728d99cf1f7631dd999d6aee562a54cc0d04cad484716`.
|
||||
Installed `--version`, `current` selection and the relaunched executable path
|
||||
matched. `previous` retains `2026-09-24T190806Z-g6592e1af`.
|
||||
|
||||
User config and authorized runtime/model paths hashes were unchanged across
|
||||
installation; advanced debugging remained enabled by the user. No FrameYap
|
||||
process was running at the pre-install check. Only FrameYap was launched;
|
||||
no SteamVR, SSH or user-session process was stopped. The new diagnostic log
|
||||
was mode 0600. No assistant-triggered recording, private-audio replay, public
|
||||
fixture inference or input injection was performed. Actual speech retry after
|
||||
this fix, recognition quality and headset interaction remain wearer-led checks.
|
||||
@@ -1,103 +0,0 @@
|
||||
# FrameYap POC observations — 2026-09-24
|
||||
|
||||
Dated observations on one user-authorized Steam Frame, not proof of future device
|
||||
availability, full dictation acceptance or runtime redistribution rights. No audio,
|
||||
weights, private transcript/log files, credentials or runtime binaries are included.
|
||||
The final native package below was built from source commit **`d60d8d2`**.
|
||||
|
||||
## CPU compatibility trial (before runtime-license review)
|
||||
|
||||
Isolated user-local environment: ARM64, glibc 2.39, Python 3.12.3,
|
||||
moondream 2.4.0 / kestrel 0.8.0 / kernels 0.7.0 / native 0.1.8,
|
||||
Torch **2.8.0+cpu** (`torch.version.cuda is None`). Model weights/config/tokenizer
|
||||
verified against pinned Redux revision `fad622f25f303105c20d70e201bcc477c88b620c`.
|
||||
No dense model substitution or desktop/cloud inference was used.
|
||||
|
||||
A one-second synthetic silence trial loaded in 5.645 s; three decodes took
|
||||
0.187 / 0.149 / 0.167 s and returned no text. Peak process RSS in that trial was
|
||||
989,844 KiB (~967 MiB). This is not peak RSS for arbitrary 20-second speech clips.
|
||||
|
||||
Public 11.0-second JFK speech fixture:
|
||||
<https://github.com/openai/whisper/blob/v20250625/tests/jfk.flac>.
|
||||
Source FLAC SHA-256 `63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715`.
|
||||
Converted to PCM16/mono/16 kHz WAV with ffmpeg; WAV SHA-256
|
||||
`0c397b12d7dbdd89e4fd0d9a0840c14a2c5c3707758562755cda96a931b25a91`.
|
||||
The persistent worker used bounded file/pipe IPC; no microphone was opened.
|
||||
|
||||
| CPU threads | Worker load | Five request times (seconds) | Median | RTF | Audio/compute ratio |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2 | 5.190 s | 1.550, 1.619, 1.587, 1.619, 1.659 | 1.619 s | 0.147 | 6.8× realtime |
|
||||
| 4 | 4.655 s | 1.196, 1.158, 1.143, 1.249, 1.502 | 1.196 s | 0.109 | 9.2× realtime |
|
||||
|
||||
The first result reproduced the fixture's known sentence; all five replies in
|
||||
each run were correlated and 108 UTF-8 bytes. These are medians/maxima from one
|
||||
public clip, not representative accuracy/WER, p95, streaming realtime performance,
|
||||
or compositor/thermal acceptance. Load time is excluded from RTF. Two threads
|
||||
remain the default pending scene-contention measurements.
|
||||
|
||||
An unconstrained dependency install initially selected CUDA Torch/NVIDIA wheels;
|
||||
these were replaced/removed from the owned venv before measurement. No system
|
||||
packages were changed. Later inspection found the proprietary kernel license's
|
||||
separate-agreement requirement. Permission was declared unresolved; further
|
||||
inference and proprietary-runtime packaging were paused pending vendor clarification.
|
||||
See [license boundary](../third-party.md). The acquired development environment
|
||||
remains user-local and is not included in the native-only installation.
|
||||
|
||||
## Native runtime / controller / overlay checks
|
||||
|
||||
- Native CMake build succeeded on Frame using standalone OpenVR SDK v2.15.6,
|
||||
explicitly built SDL 3.2.16, system Wayland and FreeType. No unrelated app tree
|
||||
or environment was linked/copied. No sudo or system/session restart.
|
||||
- Installed executable connected to `gamescope-0`, bound the IME and received
|
||||
ready/done state; each discovery check disconnected **without text or Enter**.
|
||||
- OpenVR accepted the raw panel. The user reported seeing the five-second window.
|
||||
This confirms visibility, not readability/comfort or successful pointer clicks.
|
||||
- Device controller properties identified `frame_controller` and its input profile.
|
||||
The profile exposes grip `click`; authored default bindings were added. A
|
||||
controls-only run reported both grip actions tracked/active. **No physical
|
||||
gesture delivery was observed in that run.** Double-tap/hold/loss behavior is
|
||||
currently verified by deterministic unit tests, not a human controller trial.
|
||||
- The ARM loader initially tried `/data/work/openvrpaths.vrpath`. Selecting the
|
||||
existing standard user registry fixed initialization; code now does so only
|
||||
when no explicit override is supplied.
|
||||
- Registration initially returned API success but did not install the app.
|
||||
Frame's manifest parser requires **`binary_path_linux_arm`**. Adding it fixed
|
||||
registration. Code now verifies `IsApplicationInstalled`, not return status alone.
|
||||
- Repeated Add, Remove and re-Add of **only** `local.frameyap.overlay` succeeded;
|
||||
autolaunch queried **off**. The final installed panel check also succeeded after
|
||||
explicit identification with that registered key. Menu-driven launch and cold
|
||||
SteamVR startup were not exercised.
|
||||
|
||||
## Installer and final state
|
||||
|
||||
Native-only artifacts `v0.1.0-poc1`/`poc2` exercised installation, same-version
|
||||
rerun, upgrade, rollback and forward selection. A producer-prefix SDL RUNPATH leak
|
||||
was found and removed; final ELF RUNPATH is only `$ORIGIN/../lib` and bundled
|
||||
SDL/OpenVR resolve inside the installed tree.
|
||||
|
||||
Explicit OpenVR unregister followed by `--uninstall --unregistered` succeeded.
|
||||
The final `v0.1.0-poc3` artifact was then installed twice and registered with
|
||||
autolaunch off. It includes native code, SDL/OpenVR, Hack font/notices and the
|
||||
worker adapter, **no ASR runtime or weights**. Installed size: ~8.1 MiB.
|
||||
|
||||
Local archive SHA-256:
|
||||
`90aab107feea2cb501bf815806bdc1d04d84bb6f87f7c795b60ad43c7fdc0634`.
|
||||
The artifact is retained only in the device's project-owned development directory;
|
||||
no GitHub release/upload was performed. Installed launcher is `~/.local/bin/frameyap`.
|
||||
Actual native CLI and installer both refused an independently held install lock.
|
||||
At the final check, no owned `frameyap` process or transient `frameyap-*` runtime
|
||||
directory remained. Installation/registration remain; nothing was autostarted.
|
||||
|
||||
Final verification on source `d60d8d2`: default local build **8/8 CTests**;
|
||||
optional native local build **9/9**; native ARM64 build **9/9**. Suites include
|
||||
13 installer cases, worker framing/hash/cancellation/deadlines, gestures, state,
|
||||
instance locking, and an isolated fake Wayland server. Those tests do not connect
|
||||
to the live compositor or record audio. `git diff --check` passed locally.
|
||||
|
||||
## Still open
|
||||
|
||||
Runtime permission and any public distribution; live microphone quality/device
|
||||
selection; physical tap/hold/buttons; actual text/Enter delivery into disposable
|
||||
and real target apps; focus/held-modifier behavior; quick typing's focus observer;
|
||||
scene/dashboard collisions; compositor cost, battery/thermal and headset comfort.
|
||||
No live microphone recording or input injection occurred in this POC pass.
|
||||
@@ -1,100 +0,0 @@
|
||||
# Raw-overlay probe → compositor/session crash — 2026-09-24
|
||||
|
||||
Incident analysis from read-only boot, journal, coredump metadata and SteamVR log
|
||||
inspection after the user reported a headset reboot and loss of their terminal.
|
||||
No repeat crash experiment was performed during analysis. This record does not
|
||||
authorize a probe, restart or deployment; live reproduction can destroy the desktop
|
||||
session even when the test overlay is hidden.
|
||||
|
||||
## Established timeline (UTC)
|
||||
|
||||
- Linux stayed on the same boot throughout; uptime was about seven hours when
|
||||
checked at 20:55. This was **not an OS reboot**.
|
||||
- 20:50:24.955: the normal FrameYap process called `VR_Shutdown`, then disconnected.
|
||||
Its later absence explains the read-only probe's `FindOverlay` failure; that
|
||||
failure is not evidence that SteamVR had already crashed.
|
||||
- 20:51:42.264: the final `mask-probe` process started as `VRApplication_Overlay`.
|
||||
- 20:51:42.279: it connected to the existing compositor (PID 2316).
|
||||
- 20:51:42.284: it called `VR_Shutdown`; compositor logs record its pipe disconnect.
|
||||
- 20:51:42.357: SteamVR's crash reporter was already processing a compositor dump.
|
||||
Coredump metadata timestamps the crash at 20:51:42, signal **SIGBUS (7)**.
|
||||
- 20:51:42.563: systemd-coredump began processing it; completion was logged at
|
||||
20:51:44.195. The completion time is not the initial fault time.
|
||||
- 20:51:47: `steamvr.service` failed and scheduled a restart.
|
||||
- 20:51:57: Gamescope failed to stop within ten seconds; systemd killed its
|
||||
process group, including Xwayland, and restarted the graphical session.
|
||||
This accounts for the headset appearing to reboot and terminal-session loss.
|
||||
|
||||
Read-only `systemctl --user show` established the coupling:
|
||||
`steamvr.service` has `Restart=always` and requires `gamescope-session.service`;
|
||||
`gamescope-session.service` has `PartOf=steamvr.service graphical-session.target`
|
||||
and a ten-second stop timeout. The agent did not issue a system reboot or kill
|
||||
SSH/terminal processes; the service recovery cascade performed the session kills.
|
||||
|
||||
## Trigger sequence and result
|
||||
|
||||
The temporary native ARM64 probe used OpenVR SDK 2.15.6; client startup reported
|
||||
runtime 2.17.10. It created its own hidden overlay, set an absolute pose, 1 m width,
|
||||
mouse input and 1080×780 mouse scale, then:
|
||||
|
||||
1. Allocated a zero-filled 1080×780×4-byte RGBA buffer (3,369,600 bytes).
|
||||
2. Called `SetOverlayRaw`; returned `VROverlayError_None`.
|
||||
3. Applied one rectangular intersection mask, first at (996,32,68,68), then at
|
||||
(996,680,68,68). Both calls returned success.
|
||||
4. Called `ComputeOverlayIntersection` for three synthetic rays per mask. All
|
||||
six returned false, including the intended positive cases. Therefore this
|
||||
probe **did not validate either mask coordinate convention**.
|
||||
5. Immediately called `DestroyOverlay` and `VR_Shutdown`, without waiting for an
|
||||
image-loaded event or compositor upload completion. It never showed the overlay.
|
||||
|
||||
The normal FrameYap renderer uses a persistent Vulkan texture, not this raw-upload
|
||||
path. Free-grab/lock changes were still local and were not running on the device.
|
||||
The final probe source was retained at `/tmp/frameyap-mask-probe.cpp` locally and
|
||||
`~/frameyap-poc/mask-probe.cpp` on Frame at analysis time; do not rerun casually.
|
||||
|
||||
## Crash evidence and interpretation
|
||||
|
||||
The faulting compositor thread's available stack began:
|
||||
|
||||
```text
|
||||
__memcpy_sve libc.so.6 + 0xa0008
|
||||
vrcompositor + 0x298cb8
|
||||
vrcompositor + 0x6eaf4
|
||||
vrcompositor + 0x11bcf0
|
||||
vrcompositor + 0x11c0c4
|
||||
vrcompositor + 0x14b9b4
|
||||
vrcompositor + 0x14f9e0
|
||||
vrcompositor + 0x183bf0
|
||||
start_thread
|
||||
```
|
||||
|
||||
The compositor core exists but was inaccessible to the unprivileged account;
|
||||
no privilege escalation or core extraction was attempted. No GPU reset, kernel
|
||||
panic or OOM event appeared in the inspected kernel interval 20:50–20:52.
|
||||
SteamVR's own crash reporter automatically uploaded its minidump, reporting
|
||||
CrashID `bp-f1d750cf-8943-4834-8873-9eec02260924`; the agent did not initiate that
|
||||
upload or send a dump separately.
|
||||
|
||||
**Strongly supported trigger:** the hidden raw-overlay probe and its immediate
|
||||
teardown. The compositor fault followed its disconnect within the same second.
|
||||
|
||||
**Leading unproven mechanism:** an asynchronous raw-image copy outliving its
|
||||
shared-memory backing during overlay destruction/client shutdown. SIGBUS in
|
||||
`memcpy` is consistent with an invalid/truncated mapped backing object. It does
|
||||
not prove that mechanism: raw-upload buffer sizing/limits or another compositor
|
||||
memory-handling fault remain alternatives. Symbols, fault-address/mapping data,
|
||||
or a controlled comparison are needed to distinguish them.
|
||||
|
||||
## Consequences for further work
|
||||
|
||||
- Hidden overlays are not isolated from compositor upload/lifetime machinery.
|
||||
- API success and a clean probe exit do not establish compositor completion.
|
||||
- Keep normal rendering on the existing Vulkan path; do not reintroduce
|
||||
`SetOverlayRaw` as a convenient live diagnostic shortcut.
|
||||
- A future explicitly authorized experiment should separate raw upload from
|
||||
immediate teardown, vary only one factor at a time, and capture process/journal
|
||||
evidence from a session outside the headset graphical service. A longer-lived
|
||||
probe is an experiment, not a proven workaround; adding a sleep is not a fix.
|
||||
- Tmux protects a remote coding process from a terminal/SSH disconnect. It does
|
||||
not itself isolate the headset compositor, and a tmux server inside a killed
|
||||
service group could still die. Confirm where the server lives before replaying.
|
||||
@@ -1,59 +0,0 @@
|
||||
# FrameYap click flicker investigation — 2026-09-24
|
||||
|
||||
Dated controls-only observations on one user-authorized Frame. This is not
|
||||
headset acceptance or proof that a compositor update will behave identically.
|
||||
No microphone was opened and no text/Enter was delivered. Both checks used the
|
||||
finite `--check-controls --mount world` mode; clicks were diagnostic only.
|
||||
|
||||
## What changed
|
||||
|
||||
The earlier diagnostic painted pointer counts, action text and controller state
|
||||
onto the panel, causing `SetOverlayRaw` uploads even for clicks that would not
|
||||
otherwise alter its pixels. Commit `b5499ec` logs those values to the terminal
|
||||
instead; it keeps the panel static for Record/Cancel/Insert/Enter clicks, while
|
||||
navigation and mount changes still repaint. A subsequent commit, `c59c8b1`,
|
||||
counts application raw uploads, show/hide calls and OpenVR visibility/focus/image
|
||||
events without adding repaints.
|
||||
|
||||
The rendering loop in `src/runtime.cpp` and `src/overlay.cpp` is single-threaded:
|
||||
it draws before polling input and processes click actions before the next draw.
|
||||
No separate logic/render thread race was found in this path. That alone does not
|
||||
identify SteamVR's compositor behavior.
|
||||
|
||||
## Controls-only trials
|
||||
|
||||
- Static-canvas build `2026-09-24T173900Z-gb5499ec` was built on Frame from a
|
||||
clean snapshot; 11/11 hardware-free ARM64 tests passed. Its native-only archive
|
||||
was checksum-verified and installed; the previous version was retained. The
|
||||
30-second probe exited normally with 22 pointer downs, 22 ups and 8 action/
|
||||
mount/recenter hits. The wearer reported that the **whole overlay still
|
||||
disappeared on every action click**, including clicks that leave the canvas
|
||||
static. This contradicts full raw uploads being the *sole* cause of the flash.
|
||||
- Event-counter build `2026-09-24T174700Z-gc59c8b1` was built on Frame from a
|
||||
clean snapshot; 12/12 hardware-free ARM64 tests passed. It was checksum-verified,
|
||||
installed and version-checked. The 30-second probe exited normally with 7
|
||||
pointer downs, 7 ups and 2 action/mount/recenter hits. From start to finish,
|
||||
application `ShowOverlay` calls stayed at **1**, `HideOverlay` calls at **0**,
|
||||
`VREvent_OverlayShown` at **1**, and `VREvent_OverlayHidden` at **0**.
|
||||
`SetOverlayRaw` calls rose from 2 initial uploads to 7 during the trial;
|
||||
image-loaded events reached 7 with no image-failed events. Four overlay and
|
||||
four global focus-change notifications were observed, but not on every click;
|
||||
input-focus-captured and gamepad-focus-lost counters stayed at zero. Pointer
|
||||
events occurred without intervening raw uploads or hide/show events. This
|
||||
second probe had no separate contemporaneous headset visibility report.
|
||||
|
||||
The second probe did not log hit targets; its five later raw uploads may include
|
||||
tab, mount or placement-note changes and are not a per-action-click count. The
|
||||
log does **not** establish that SteamVR
|
||||
never briefly occluded the panel: it only shows that the app did not call hide
|
||||
and received no hidden event in that finite trial. The first trial's visual
|
||||
report plus static-click rendering behavior point toward dashboard laser/input
|
||||
compositing rather than a FrameYap repaint race, but the exact compositor cause
|
||||
remains unverified. A persistent GPU texture alone cannot explain or guarantee a
|
||||
fix for flicker observed when no texture is uploaded.
|
||||
|
||||
The installed diagnostic version at the end of this investigation was
|
||||
`2026-09-24T174700Z-gc59c8b1` (native-only, external ASR runtime unchanged); an
|
||||
independent controls-only right-X binding trial occurred between these two
|
||||
builds. No session or SSH process was terminated. No further hardware remedy is
|
||||
claimed here.
|
||||
@@ -1,61 +0,0 @@
|
||||
# Persistent Vulkan overlay deployment — 2026-09-24
|
||||
|
||||
Source commit: `4c043c99`.
|
||||
Installed native ARM64 version: `2026-09-24T181314Z-g4c043c99`.
|
||||
|
||||
The user requested replacing raw uploads with a GPU texture and supplied the
|
||||
current Frame SSH target and key for native build/install verification. These
|
||||
are dated observations, not future device availability or permission to launch
|
||||
hardware checks.
|
||||
|
||||
## Implementation
|
||||
|
||||
The overlay now uses `SetOverlayTexture` with a persistent Vulkan RGBA8 image.
|
||||
The CPU panel rasterizer still produces the pixels. One staging allocation,
|
||||
image and command buffer are reused; redraws do not recreate them. SteamVR
|
||||
selects the physical device and required instance/device extensions. Transfers
|
||||
share one FrameYap-owned graphics queue with SteamVR, and GPU resources survive until
|
||||
`VR_Shutdown` completes. No desktop surface, swapchain or raw-upload fallback
|
||||
was introduced. See [rendering details](../overlay.md).
|
||||
|
||||
## Verification
|
||||
|
||||
- Default local offline build: 13/13 CTest checks passed.
|
||||
- Local native build: all 16 checks passed; the fake Wayland protocol check
|
||||
required permission to bind its local test socket outside the sandbox.
|
||||
- Native ARM64 Release build on Frame from a checksum-verified Git archive:
|
||||
16/16 hardware-free checks passed, including the Vulkan fake-driver test.
|
||||
- Explicit offscreen Vulkan check on the workstation's RTX 4090, with
|
||||
`VK_LAYER_KHRONOS_validation` enabled: eight exact 1000×680 RGBA readbacks,
|
||||
one stable image, no validation messages.
|
||||
- The same explicit offscreen check on Frame's `Turnip Adreno (TM) 750`:
|
||||
eight exact 1000×680 RGBA readbacks and one stable image. This exercised the
|
||||
actual image upload/layout/readback path, without OpenVR initialization.
|
||||
- Staged and installed native dependency resolution succeeded. Vulkan resolves
|
||||
to the system loader; SDL/OpenVR resolve inside FrameYap's own `lib/`.
|
||||
- The native archive was checksum-verified and installed. `current` selects
|
||||
the version above; `previous` retains `2026-09-24T174700Z-gc59c8b1`.
|
||||
Installed `--version` matched and its executable hash matched the staged
|
||||
binary. No app, SSH or user-session process was terminated.
|
||||
|
||||
Native archive SHA-256:
|
||||
`d6f0173b25a879c02f0ee67063c881a23dff874a71674826ddd7666d0980759d`.
|
||||
Installed executable SHA-256:
|
||||
`761ea0e4d2dc235c5f056d8d944cb68392713c246aaec008d0b6c235954ef645`.
|
||||
|
||||
## Acceptance boundary
|
||||
|
||||
The updated headset installation is verified, but this session did not launch
|
||||
an OpenVR visual/controls probe or collect a wearer report. GPU readback does
|
||||
not establish SteamVR texture acceptance, orientation, click behavior or a
|
||||
flicker fix. No microphone, inference or text/Enter delivery was exercised.
|
||||
The next human headset check should compare both static action clicks and
|
||||
content-changing tabs using the installed Vulkan build's `--check-controls`.
|
||||
|
||||
## Subsequent wearer report — 2026-09-24
|
||||
|
||||
After testing the installed Vulkan build, the user confirmed that the flicker
|
||||
is fixed. They separately reported controller shortcuts becoming unavailable
|
||||
in system laser/dashboard interaction states while pointer clicks work. This
|
||||
is wearer confirmation of the visual fix and a distinct input-routing issue;
|
||||
it does not establish microphone, transcription or text-delivery acceptance.
|
||||
@@ -2,10 +2,11 @@
|
||||
|
||||
**User goal:** install from GitHub with a `curl … | bash`-style command, without a
|
||||
Steam store AppID. An idempotent archive installer and native-only local artifacts
|
||||
are now implemented/tested; no public release is published. The full bundled-ASR
|
||||
experience remains blocked on runtime permission. This document retains the target
|
||||
design; see [current packaging](packaging.md), [POC evidence](evidence/poc-cpu-overlay-2026-09-24.md)
|
||||
and [third-party boundary](third-party.md).
|
||||
are now implemented/tested; no public release is published. A bundled-ASR
|
||||
experience is not yet offered. This document retains the target
|
||||
design; see [current packaging](packaging.md) and [third-party notes](third-party.md).
|
||||
Planned installer work (binary-or-source choice, flag-driven operation) is in
|
||||
[TODO.md](../TODO.md).
|
||||
|
||||
The read-only `scripts/install-preflight.sh` checks whether a host appears suitable
|
||||
for the **proposed** Linux ARM64 glibc package format and has the expected basic
|
||||
@@ -66,16 +67,15 @@ confirmation from stdin while the installer itself is arriving through that pipe
|
||||
no external application libraries/assets and no system Python modification.
|
||||
- Model fetched during explicit installation/setup, with pinned revision/hash and
|
||||
attribution. A documented `--without-model` option can defer the large download.
|
||||
No surprise first-utterance downloads. Licensing may require obtaining particular
|
||||
runtime components from their vendor instead of redistributing them in our tarball.
|
||||
No surprise first-utterance downloads. Runtime components are installed from
|
||||
their own package index rather than redistributed in our tarball.
|
||||
- Install under `$XDG_DATA_HOME/frameyap` (default `~/.local/share/...`),
|
||||
configuration under `$XDG_CONFIG_HOME/frameyap`, optional launcher in
|
||||
`~/.local/bin`; transient audio stays in a private `$XDG_RUNTIME_DIR` directory.
|
||||
- No sudo, OS read-only-root changes, package-manager installs, udev changes,
|
||||
`/dev/uinput` permission changes or modifications to unrelated launchers.
|
||||
- No Steam store AppID, Steamworks SDK, root service or network ASR dependency.
|
||||
- Distribution license/third-party notices must be resolved before publication;
|
||||
the small model file alone is not the full runtime or redistribution permission.
|
||||
- Third-party notices for everything we redistribute must be included with a release.
|
||||
|
||||
## Installer lifecycle and safety
|
||||
|
||||
|
||||
+7
-13
@@ -1,4 +1,4 @@
|
||||
# Native OpenVR POC panel
|
||||
# Native OpenVR panel
|
||||
|
||||
`src/overlay.hpp` provides RAII OpenVR ownership and `registration()`. The panel
|
||||
only emits UI actions; `src/runtime.cpp` owns audio, transcription and insertion.
|
||||
@@ -12,7 +12,7 @@ normal build or test.
|
||||
Explicit development dependencies: Valve OpenVR SDK v2.15.6, Vulkan headers/loader
|
||||
and FreeType 2. The native runtime needs a compatible system Vulkan driver.
|
||||
Configure/build must not fetch them. The default font is the bundled Inconsolata
|
||||
Regular, also used by kouseki; its OFL and extraction provenance are included in
|
||||
Regular; its OFL and notices are included in
|
||||
[third-party notes](third-party.md). `--font FILE` overrides the JSON selection.
|
||||
A missing selected font falls back to bundled Inconsolata, then a system DejaVu
|
||||
Sans face if present. Glyph coverage depends on the selected face; full CJK
|
||||
@@ -24,7 +24,7 @@ margins for a thin grab underline and an external L-shaped scale handle. `src/ov
|
||||
persistent Vulkan RGBA8 image and submits it with `SetOverlayTexture`. The image,
|
||||
staging allocation and command buffer are reused; tabs do not create extra
|
||||
overlays or render targets. The rounded mint-to-blue perimeter,
|
||||
shallow curved accent, and dark cards borrow kouseki's VR visual language. Rounded
|
||||
shallow curved accent, and dark cards form the panel's visual language. Rounded
|
||||
preview, status and control surfaces use independently rasterized antialiased edges
|
||||
and restrained baked neon halos rather than GPU bloom. The recording indicator and
|
||||
selected controls remain distinguishable by their labels, not color alone. Rounded
|
||||
@@ -43,13 +43,10 @@ transfer before reusing staging memory. Image barriers finish in
|
||||
[OpenVR's Vulkan contract](https://github.com/ValveSoftware/openvr/wiki/Vulkan).
|
||||
The queue is used on the overlay thread; GPU resources outlive `VR_Shutdown`.
|
||||
The device selection, texture description and persistent panel-upload patterns
|
||||
were compared with kouseki's `openvr_session.cpp` and `vulkan_renderer.cpp` at
|
||||
`738569f4c41ff4c8fc9edd5bfff9c861957ea39e`; FrameYap owns this implementation.
|
||||
are FrameYap's own implementation.
|
||||
GPU setup/submission errors stop startup or the run with an explicit error.
|
||||
This replaces the raw-upload rendering path; headset flicker acceptance still
|
||||
requires an on-device comparison. The
|
||||
[Vulkan deployment record](evidence/vulkan-overlay-2026-09-24.md) documents the
|
||||
native installation and offscreen GPU checks separately from headset acceptance.
|
||||
requires an on-device comparison. Native installation and offscreen GPU checks are separate from headset acceptance.
|
||||
|
||||
The header shows local time and date instead of the former on-device/review
|
||||
and current-mount labels. It updates when the displayed minute or date changes,
|
||||
@@ -280,16 +277,13 @@ and panel-front (controller +Z and -X), keeping panel-up unchanged so it faces
|
||||
inward with upright, unmirrored text. The controller-relative center is
|
||||
(0, 0.18, 0.089) m, approximating the compact HUD's surface center: its
|
||||
0.12 m wrist lift, 0.09 m bottom anchor and ~0.03 m panel-center correction;
|
||||
Z combines the fallback 0.054 m wrist calibration and 0.035 m finger-back offset. This copies placement geometry, not
|
||||
VR Workspace's avatar-dependent wrist calibration. Wrist-mounted panels now use
|
||||
the same *behavior* as kouseki's watch HUD: fully visible while their entire
|
||||
Z combines the fallback 0.054 m wrist calibration and 0.035 m finger-back offset. Wrist-mounted panels are fully visible while their entire
|
||||
orientation is within 60° of an upright, viewer-facing panel; linear opacity
|
||||
fade from 60° to 75°, then hidden (including laser interaction). Pitch, yaw and
|
||||
roll contribute together; turning the wrist away or moving the head around it
|
||||
changes the angle. OpenVR's overlay alpha changes without rerendering the panel.
|
||||
World and head mounts do not fade. Missing headset tracking hides a wrist panel;
|
||||
a lost wrist still uses the existing world-space fallback. This was implemented
|
||||
independently with no kouseki library or runtime dependency. Headset readability,
|
||||
a lost wrist still uses the existing world-space fallback. Headset readability,
|
||||
fade feel and interaction at the threshold still need live acceptance.
|
||||
|
||||
To tune the selected wrist, set `wrist` in `config.json` as in the example above:
|
||||
|
||||
+10
-15
@@ -1,9 +1,8 @@
|
||||
# Release packaging and idempotent user-local installer
|
||||
|
||||
**No GitHub release is published.** Native-only local artifacts have been installed
|
||||
and reinstalled on Frame. The end-to-end bundled-ASR release remains blocked on
|
||||
runtime permission; see [third-party notes](third-party.md). The installer never
|
||||
pretends the proprietary runtime is included when it is not.
|
||||
and reinstalled on Frame. The installer never pretends an ASR runtime is included when it is not;
|
||||
see [third-party notes](third-party.md).
|
||||
|
||||
## Producer
|
||||
|
||||
@@ -21,11 +20,11 @@ model/* # optional pinned public weights + attribution
|
||||
runtime/bin/python3 # ONLY for an authorized bundled-runtime artifact
|
||||
```
|
||||
|
||||
For the current **external-runtime** POC, `scripts/stage-native-poc.py --help`
|
||||
For the current **external-runtime** package, `scripts/stage-native-poc.py --help`
|
||||
documents explicit inputs. It invokes `cmake --install` on an existing native build,
|
||||
copies SDL/OpenVR and an explicitly licensed font, and retains notices. It does
|
||||
not build, download, run the app, or copy a proprietary ASR runtime. The native
|
||||
POC relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
||||
not build, download, run the app, or copy an ASR runtime. The native
|
||||
app relies on Frame's system Vulkan loader/driver, Wayland, libxcb, FreeType,
|
||||
libstdc++ and glibc; audit `ldd` on the installed binary.
|
||||
SDL/OpenVR resolve inside its own `lib/`, not a producer
|
||||
prefix. ARM64/glibc packaging is not a claim of compatibility with arbitrary Linux.
|
||||
@@ -46,10 +45,8 @@ the installer still refuses a reused tag whose contents have changed. Historic
|
||||
|
||||
`--external-runtime` refuses a runtime directory and records
|
||||
`runtime: external-authorized-python` in `release.json`. The installer explicitly
|
||||
reports that ASR is not supplied. Without that flag, a complete independently
|
||||
licensed, compatible isolated CPU Python runtime is required. **Do not use that
|
||||
bundled route for Kestrel without permission covering redistribution.** Staging
|
||||
validation is not a license grant or an inference test.
|
||||
reports that ASR is not supplied. Without that flag, a complete compatible isolated CPU Python runtime is required
|
||||
in the archive. Staging validation is not an inference test.
|
||||
|
||||
The producer refuses overwrites and emits `frameyap-VERSION-linux-aarch64.tar.gz`
|
||||
plus `.sha256` containing `HASH FILENAME`. Archive extraction rejects traversal,
|
||||
@@ -129,7 +126,7 @@ Runtime/check/registration modes and installer share an exclusive nonblocking
|
||||
Selection of a completed `current` is atomic; `previous` is retained.
|
||||
`sh install.sh --rollback` switches to the prior validated version. Foreign/modified
|
||||
wrappers, untracked install files and inconsistent ownership metadata are refused.
|
||||
Same-user malicious concurrent filesystem mutation is outside the POC threat model.
|
||||
Same-user malicious concurrent filesystem mutation is outside the current threat model.
|
||||
|
||||
The generated `frameyap.vrmanifest` uses `local.frameyap.overlay`, **not a store
|
||||
AppID**. Linux ARM requires `binary_path_linux_arm`; both Linux fields are written.
|
||||
@@ -178,8 +175,7 @@ not hand-edit Steam's shortcut database.
|
||||
|
||||
This check does not validate microphone capture, transcription, controller input,
|
||||
text delivery or cold SteamVR startup. With a configured inference runtime, the
|
||||
menu launch attempts to load the model; defer that test until runtime licensing is
|
||||
resolved or independent authorization is established. If an
|
||||
menu launch attempts to load the model; defer that test unless you intend to load the model. If an
|
||||
environment/configuration unexpectedly supplies a runtime/model, do not perform
|
||||
this inert launcher check.
|
||||
|
||||
@@ -189,6 +185,5 @@ an acknowledgement, not a hidden SteamVR edit. Only owned files are removed;
|
||||
config stays, models move to `saved-models/VERSION`, conflicts/untracked files abort.
|
||||
|
||||
Offline tests: `python3 -m unittest discover -s tests -p test_installer.py`.
|
||||
They use temporary homes/local fixtures. Dated live-device observations are in the
|
||||
[POC record](evidence/poc-cpu-overlay-2026-09-24.md). When editing the Python helper,
|
||||
They use temporary homes/local fixtures. When editing the Python helper,
|
||||
run `python3 scripts/sync-installer.py`; tests enforce embedded installer parity.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Technical provenance and limits
|
||||
|
||||
FrameYap is a standalone native POC. No unrelated application's source, build
|
||||
tree, assets or Python environment is included. No model weights or runtime
|
||||
binaries are in Git. The included Gamescope XML's provenance and separately
|
||||
licensed build inputs are recorded in [third-party notes](third-party.md).
|
||||
The dated [device probe](evidence/frameyap-apis-2026-09-24.md) records limited historical observations; its fixture code and private logs are
|
||||
not part of this repository. No current device availability or release install
|
||||
can be inferred from those observations.
|
||||
|
||||
## Model and preliminary measurements
|
||||
|
||||
Proposed model: [moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux/tree/fad622f25f303105c20d70e201bcc477c88b620c),
|
||||
revision `fad622f25f303105c20d70e201bcc477c88b620c`. An earlier local
|
||||
inspection measured a 177,774,490-byte weight file. Its model card identifies
|
||||
CC-BY-4.0; separate runtime/kernel redistribution terms must be reviewed before
|
||||
packaging. No license for those artifacts is granted by this repository.
|
||||
|
||||
A preliminary, unpublished desktop benchmark used Python 3.12.13, moondream
|
||||
2.4.0, kestrel 0.8.0 and four CPU threads. On an i7-13700K, warm decode
|
||||
median/p95 were 61/137 ms and raw WER 11.11% on just 25 reviewed clips / 189
|
||||
words. This is not a general accuracy estimate or Frame latency prediction. The
|
||||
CPU-selected desktop process also occupied GPU memory; GPU-free operation was
|
||||
not established in that earlier desktop trial. A later isolated ARM64 CPU-only
|
||||
public-clip trial is recorded in [POC observations](evidence/poc-cpu-overlay-2026-09-24.md);
|
||||
it does not establish live-microphone/headset acceptance or distribution permission.
|
||||
|
||||
## Public platform references
|
||||
|
||||
- [OpenVR 2.15.6](https://github.com/ValveSoftware/openvr/releases/tag/v2.15.6)
|
||||
- [Gamescope 3.16.28 input-method protocol](https://github.com/ValveSoftware/gamescope/blob/3.16.28/protocol/gamescope-input-method.xml)
|
||||
- [Gamescope IME implementation](https://github.com/ValveSoftware/gamescope/blob/3.16.28/src/ime.cpp)
|
||||
+14
-26
@@ -20,9 +20,7 @@ images, protected kernels or another application's assets were extracted for the
|
||||
|
||||
## Bundled font and UI reference
|
||||
|
||||
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of the typeface used
|
||||
by kouseki's editor and VR canvas, extracted from its `assets/fonts` directory at
|
||||
checkout revision `738569f4c41ff4c8fc9edd5bfff9c861957ea39e`.
|
||||
`assets/fonts/Inconsolata-Regular.ttf` is an unmodified copy of Inconsolata Regular.
|
||||
SHA-256: `e0267abf9d734e2b9f766f8cb7a496b552c57cdfeacfa0efdc5bfd21940ae145`.
|
||||
Copyright 2006 The Inconsolata Project Authors; **SIL Open Font License 1.1**,
|
||||
retained in `assets/fonts/OFL-Inconsolata.txt` (line endings and trailing whitespace
|
||||
@@ -32,10 +30,8 @@ OFL, not MIT, and is not sold by itself. Upstream: <https://github.com/googlefon
|
||||
The TTF is unchanged. No MSDF atlas, icons, engine code or renderer dependencies
|
||||
were copied. Unicode coverage is finite; missing glyphs use the face's notdef glyph.
|
||||
|
||||
Visual references: kouseki's `apps/vr_workspace/hud.cpp` (rounded mint-to-blue
|
||||
perimeter and curved accent) and `menu_tablet.hpp` (dark cards, highlighted
|
||||
selection). FrameYap implements those design ideas independently on a single
|
||||
CPU RGBA surface. Building, installing and running require no kouseki checkout.
|
||||
The panel's visual style (rounded mint-to-blue perimeter, dark cards, highlighted
|
||||
selection) is implemented independently on a single CPU RGBA surface.
|
||||
CMake installs the font and OFL with assets; native staging defaults to that
|
||||
font, places the launcher copy at `fonts/font.ttf`, and includes its license in
|
||||
`THIRD_PARTY_NOTICES.txt`. Custom staging fonts still require an explicit license.
|
||||
@@ -56,7 +52,7 @@ font, places the launcher copy at `fonts/font.ttf`, and includes its license in
|
||||
- Compiler runtime, libc minimum and transitive shared libraries require a release
|
||||
dependency audit. Passing a developer build is not a portable-runtime guarantee.
|
||||
|
||||
## Redux weights and proprietary runtime are different
|
||||
## Redux weights and inference runtime
|
||||
|
||||
Public model: <https://huggingface.co/moondream/parakeet-redux>, exact revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c`, model card **CC-BY-4.0**. Attribution:
|
||||
@@ -66,24 +62,16 @@ sizes and SHA-256 hashes are recorded in `python/frameyap/model_files.py`.
|
||||
`fetch-model.py` fetches and retains the original model card alongside the files.
|
||||
No weights are committed to this repository.
|
||||
|
||||
The inspected `kestrel-kernels==0.7.0` wheel license identifies it as proprietary
|
||||
M87 Labs software and says use requires a separate written agreement. Copying and
|
||||
redistribution are restricted by that agreement. This includes its protected CPU
|
||||
payload, not just CUDA. The Python wrapper and model card do not override those
|
||||
terms. No attempt was made to unpack/decrypt/reverse-engineer protected kernels.
|
||||
Inference uses the `moondream` Python package and its `kestrel` / `kestrel-kernels`
|
||||
dependencies, installed by the user from PyPI into their own environment. Upstream's
|
||||
Kestrel README states: "Local inference is free and requires no API key"
|
||||
(<https://github.com/m87-labs/kestrel>); finetuned-model inference needs a Moondream
|
||||
API key and is not used here. FrameYap's release archives do not vendor or bundle
|
||||
these packages; they are installed from PyPI onto the user's machine, either by the
|
||||
user or by the installer at the user's request.
|
||||
|
||||
**Release blocker:** permission covering use and redistribution has not been
|
||||
established here. The initial on-device compatibility measurements preceded this
|
||||
license review; further inference/bundling was paused when the issue was found.
|
||||
The acquired packages remain isolated under the device's project-owned development
|
||||
directory, not in Git or a published artifact. Do not advertise the GitHub runtime
|
||||
bundle as available or automatically download/install those packages for end users.
|
||||
Resolve permission with the vendor, or separately scope an independently licensed
|
||||
runtime for the same weights. Do not silently substitute a dense/heavier model.
|
||||
|
||||
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) also
|
||||
need their own notice/license inventory before publication. The public model's
|
||||
small size is neither total runtime size nor redistribution permission.
|
||||
Other runtime packages (Torch CPU, numpy, tokenizer/native extensions, etc.) need
|
||||
their own notice/license inventory before a bundled release.
|
||||
|
||||
## CPU trial dependency choice
|
||||
|
||||
@@ -95,7 +83,7 @@ Torch reported `torch.version.cuda is None`. Do not repeat an unconstrained
|
||||
`pip install moondream` as a CPU setup recipe.
|
||||
|
||||
For eventual packaging research, a standalone CPython 3.12.14 ARM64 distribution
|
||||
was downloaded but not bundled with the proprietary runtime:
|
||||
was downloaded but not bundled with any ASR runtime:
|
||||
<https://github.com/astral-sh/python-build-standalone/releases/tag/20260901>,
|
||||
`cpython-3.12.14+20260901-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz`,
|
||||
SHA-256 `577b4bec0793ad1ff0cbff9adbd0df078eddde38a4c41bf5d83ad381a85ee39d`.
|
||||
|
||||
+4
-6
@@ -46,7 +46,7 @@ thread-pool variables set before import. It uses
|
||||
`md.photon("moondream/parakeet-redux", model_path=<absolute local directory>,
|
||||
device="cpu", cpu_threads=threads)` and persistent
|
||||
`transcribe(audio=<numpy float32>, sample_rate=16000)["text"]`.
|
||||
Install an **isolated** Python runtime with the separately reviewed
|
||||
Install an **isolated** Python runtime with the
|
||||
moondream 2.4.0, kestrel 0.8.0 and compatible CPU dependencies; provide
|
||||
preinstalled local weights from revision
|
||||
`fad622f25f303105c20d70e201bcc477c88b620c` and pass its directory
|
||||
@@ -58,11 +58,9 @@ flags do not prove every third-party internal is unable to access a network.
|
||||
Runtime/build/tests perform no downloads; the separate explicit setup utility
|
||||
`scripts/fetch-model.py` can provision the public pinned weights.
|
||||
|
||||
**Licensing blocker:** the observed kestrel-kernels 0.7.0 license requires a
|
||||
separate M87 Labs agreement; do not treat wheel availability as permission for use
|
||||
or bundling. See [third-party notes](third-party.md). No public runtime bundle has
|
||||
been released. Limited ARM64 measurements are in the [POC record](evidence/poc-cpu-overlay-2026-09-24.md),
|
||||
not a claim of complete headset acceptance.
|
||||
Release archives do not bundle the runtime; see [third-party notes](third-party.md). No public
|
||||
runtime bundle has been released. Limited ARM64 measurements were taken during development;
|
||||
they are not a claim of complete headset acceptance.
|
||||
|
||||
## Advanced debugging
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stage a native-only POC from an explicit native build and licensed files.
|
||||
|
||||
No downloads, compiler invocation, proprietary ASR runtime, registration or launch.
|
||||
No downloads, compiler invocation, ASR runtime, registration or launch.
|
||||
System Vulkan loader/driver, Wayland/FreeType/libstdc++/glibc remain platform prerequisites.
|
||||
"""
|
||||
import argparse
|
||||
@@ -43,7 +43,7 @@ def main():
|
||||
notices = ["FrameYap native-only POC. No ASR runtime or model is included.\n",
|
||||
"Original FrameYap code: MIT. System Vulkan/Wayland/FreeType/libstdc++/glibc are not bundled.\n",
|
||||
"Bundled libraries: Valve OpenVR and unmodified SDL3; font license included below.\n",
|
||||
"This package does not grant any rights to kestrel-kernels or provide a functioning ASR environment.\n"]
|
||||
"This package does not include Kestrel or provide a functioning ASR environment.\n"]
|
||||
for label, file in (("FrameYap", root / "LICENSE"), ("OpenVR", args.openvr_license),
|
||||
("SDL3", args.sdl_license), ("Font", args.font_license)):
|
||||
notices.extend([f"\n--- {label} ---\n", file.read_text()])
|
||||
|
||||
@@ -171,7 +171,7 @@ struct PanelSurface::Impl {
|
||||
}
|
||||
}
|
||||
void frame() {
|
||||
// Independently rasterized version of kouseki's HUD visual language:
|
||||
// Independently rasterized neon-frame HUD style:
|
||||
// rounded mint-to-blue perimeter and a second shallow curved accent.
|
||||
for (int y = 0; y < CH; ++y) for (int x = 0; x < CW; ++x) {
|
||||
if (x > 34 && x < CW - 34 && y > 34 && y < CH - 34) continue;
|
||||
|
||||
Reference in new issue
Block a user