baketnk--frame-yap/docs/build.md

# Build, scope and validation

This is a standalone native application, not a plugin. Default builds/tests never
initialize OpenVR, open a microphone, run ASR, download files or inject input.

## Implemented

- Opt-in OpenVR RGBA overlay with world-space default and selectable wrist/head
  mounts, status, recording timer, paginated UTF-8 preview and explicit controls.
  A saved, default-off Lasers anytime setting requests system-wide laser mouse
  mode while FrameYap is visible; it may affect games, and is not an input override.
- Remappable SteamVR actions. The default Steam Frame binding maps right X
  (hold to record, release to transcribe) to the existing PTT action using the
  observed `frame_controller` profile. Right B cancels, A requests Type (text +
  space), and Y opens/cycles Quick phrases. Type + Enter is the explicit overlay
  control or left-grip double-tap (Enter alone if no preview/phrase). The Bindings button
  requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
  actions were inactive in the observed dashboard check; do not rely on them.
  If left grip becomes active, two short taps request explicit Enter.
  For grip gestures, first squeeze <=250 ms; second squeeze begins <=350 ms
  after first release.
  Activity/tracking loss cancels a held recording and requires neutral rearm.
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
  20 second maximum. In the native app the microphone stream opens after model
  warm-up, stays running between utterances, and discards idle samples; PTT does
  not open/pause/close the device **by default**. Settings → Close mic when idle
  (`"close_mic_when_idle": true`, default **false**) closes it between clips and
  reopens on PTT; repeated transitions caused an audio spike on Frame and may
  add latency or clip the first syllable. Quit, worker restart, device failure
  or capture failure also closes it. Keeping it open does not promise glitch-free
  playback on every audio stack. Other apps may still
  transmit your voice: this app does **not** mute VRChat or any other app.
- Local Models chooser and bounded manifest-driven dispatcher: selecting a listed
  backend restarts the worker and invalidates pending clip/review/focus authority;
  a missing or invalid model disables recording. A separate Install then Confirm
  Install exposes pinned source/size/license/attribution and exact manifest-byte
  SHA-256 before installer handoff. These are local implementations, not a tested
  installed UI or an approved additional inference engine. Only Redux is supplied.
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
  cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
  Model imports are lazy and loading is offline. A correlated bad transcript/
  `E` reply is a request-level failure: Record can retry without dropping the
  ready model. Normal repeats and microphone failures retain the loaded model;
  microphone device failure releases the stream for explicit retry. A cancelled
  in-flight request or broken worker protocol may require reloading. No
  cloud/desktop fallback.
- Gamescope IME v2 generated bindings and a tick-driven paced delivery queue:
  at most 24 Unicode codepoints per commit, at least 150 ms between commits,
  including separate actions and explicit Enter. This is a proposed workaround
  for the finite temporary keymap, not a target-consumption acknowledgment. Clipboard is
  untouched. The full literal is retained while pending; Cancel/Quit, focus loss
  or model/debug changes discard unsent input without retry or Enter. Manual
  Type captures a verified Xwayland target; Auto insert keeps its original guard.
  The IME is retained only through the task and cooldown (shutdown can wait the
  remaining <=150 ms). Pacing does not sleep in the UI loop, but an individual
  Wayland roundtrip can still reach its bounded one-second timeout. Type adds
  a trailing space if it fits, without doubling an existing one. If a validated
  transcript fills the 4096-byte bound and lacks a trailing space, Type preserves
  all its bytes and
  queues it **without** the usual space; it does not signal a separate error.
  Destination consumption and repeated-delivery behavior remain unaccepted.
- User-local installer with checked binary-archive or explicitly provisioned
  source-build mode, safe extraction, atomic current-version selection, retained
  rollback, runtime/install lock, foreign-file refusal and explicit
  unregister-before-uninstall acknowledgement. Source mode needs a local
  compiler, SDK and libraries. An explicit `--install-runtime --yes` step
  pip-installs the pinned CPU Python runtime into a user-local venv.

## Deliberately not claimed

**Review-first by default.** An opt-in Auto insert setting now watches the
Xwayland active window, Gamescope focused-window property and exact keyboard
focus from PTT start to IME lease acquisition. Loss/regain, disagreement,
unavailable focus or a held keyboard key falls back to manual review. No
automatic Enter or inferred speech commands. This focus guard has offline
tests but live microphone → automatic insertion is **not yet accepted** on Frame;
it cannot identify arbitrary native Wayland targets. There remains a race
between the final focus check and global input processing. Text delivery is
reported as **input queued**, not application consumption or message delivery.
A request is consumed once even if transport completion is uncertain; no retries.
Unavailable IME acquisition leaves the preview intact.

No VAD, always-listening mode, desktop transcription service, keyboard emulation
fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip bindings
are not guaranteed globally active in every scene/dashboard state, and the app
never enables SteamVR's experimental overlay overrides on your behalf.

This is one compact panel, not a separate miniature status chip. Front-prefix
loss on repeated submissions (P1) is under separate investigation, **not fixed**
by the chooser/worker documentation or any speculative delivery change. Font
coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact
and target application compatibility require further headset work.

## Inference runtime

Redux weights at `2bf128600aac4b16946f7ed8372e56117fe5e23b` are CC-BY-4.0. Pinned
file sizes/hashes and attribution live in `assets/backends/redux.json`; offline
`--list-models`/`--check-model redux --model-dir /absolute/model` (or
`scripts/model-status.py`) verify without inference or downloads. Manifest
schema/verification are in `python/frameyap/model_files.py`. The local generic
dispatcher resolves a manifest's in-release Python/executable launcher and
checks request/reply correlation; a new manifest still needs its own
compatible offline runtime and independent tests. Native `--run` flags `--backend ID`, `--model-store /absolute/store` and
`--manifest-dir /absolute/manifests` are wired through
the installed launcher as an explicit override, not a provisioning command.
Inference uses the `moondream` Python package and Kestrel runtime, separately
provisioned in your own environment or by the explicit `install.sh --install-runtime --yes`;
builds and tests never pip-install them. See
[third-party notes](third-party.md).

## Developer native build

Requirements: Linux, CMake/C++20, SDL3 >=3.2, Wayland client + scanner, FreeType,
Vulkan headers/loader (plus the system GPU driver at runtime),
and a deliberately provisioned standalone OpenVR **v2.15.6** SDK. No CMake fetches.

```sh
cmake -S . -B build-native -DFRAMEYAP_NATIVE=ON \
  -DOPENVR_ROOT=/absolute/path/to/openvr
cmake --build build-native -j2
ctest --test-dir build-native --output-on-failure
```

On ARM64, select the SDK's `lib/linuxarm64/libopenvr_api.so` explicitly with
`-DOPENVR_LIBRARY=...` if necessary. Do not use an x86 library or another
application's build/runtime. Installable binary has `$ORIGIN/../lib` RUNPATH;
producer must audit and bundle its compatible dependency closure.

The device development trial built SDL **3.2.16** under a project-owned user
prefix with audio enabled and video/render/GPU/joystick/haptic/sensor/camera/tests
examples disabled (`SDL_UNIX_CONSOLE_BUILD=ON`). No OS package modifications.
That is a producer workflow, not an end-user compiler requirement.

## Explicit hardware checks (not CTest)

```sh
./build-native/frameyap --check-input --socket gamescope-0
./build-native/frameyap --check-overlay --assets "$PWD/assets" --mount world
./build-native/frameyap --check-controls --assets "$PWD/assets" --mount world
```

The first acquires/releases an IME without text/actions. The second displays a
five-second inert panel. The third displays a 30-second diagnostic panel and
reports pointer actions, SteamVR grip and PTT action activity, tracking and
read-only legacy grip state, **without microphone or input injection**. Mount
clicks in check modes do not save a preference. A SteamVR error code or inactive
action means the gesture cannot be accepted; raw grip reads do not authorize a
fallback binding.
Action handles alone are not proof that gestures were delivered. Checks must be explicitly launched
while the user expects the panel. Normal CLI/help/version remain inert.

The ARM64 OpenVR loader's default `/data/work/openvrpaths.vrpath` failed on the
observed device. FrameYap respects `VR_PATHREG_OVERRIDE`; when absent, it selects
an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or
`~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files
are edited and no runtime/session restart is performed.

## Deliberate launch

Explicit setup downloads only the pinned model:

```sh
python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model"
```

`fetch-model.py` is idempotent for matching hashes, rejects existing mismatched
files, and is never invoked by build/tests or first utterance.

Provide your own CPU Python environment (moondream 2.4.0,
kestrel 0.8.0) and local weights:

```sh
./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \
  --python /path/to/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
  --model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head
```

Use a disposable text destination first. `--run` loads the model but does not
record until an explicit recording control. Click Type only after focusing your
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
and terminates only the owned worker. Installer upgrades refuse an active app.

The `Controller` receives injected `ControllerAudio`, `ControllerWorker`,
`ControllerFocus` and delivery factory interfaces; hardware-free fakes can test
state, retry, authorization and mic lifetime without initializing OpenVR or
recording speech. The worker API is intentionally small: a fork can replace the
worker implementation or add its own model/API integration without changing
overlay and delivery code.
The default product remains local-only Redux; extending a fork does not authorize
sending existing users' audio to a service.

## Benchmarks

`scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV;
no recording, device input or automatic download. `--show-text` is a separate
explicit disclosure of that fixture's transcript. Use only with an authorized
runtime. Benchmark numbers from earlier trials are not headset/performance acceptance.