baketnk--frame-yap/docs/poc.md

# FrameYap POC: implementation and validation

This is a standalone native application, not a plugin. Default builds/tests never
initialize OpenVR, open a microphone, run ASR, download files or inject input.

## Implemented

- Opt-in OpenVR RGBA overlay with world-space default and selectable wrist/head
  mounts, status, recording timer, paginated UTF-8 preview and explicit controls.
  A saved, default-off Lasers anytime setting requests system-wide laser mouse
  mode while FrameYap is visible; it may affect games, and is not an input override.
- Remappable SteamVR actions. The default Steam Frame binding maps right X
  (hold to record, release to transcribe) to the existing PTT action using the
  observed `frame_controller` profile. Right B cancels, A inserts + space, and Y
  inserts pending text + Enter (or Enter only with no preview). The Bindings button
  requests SteamVR's remapping editor directly. Grip bindings remain, but both grip
  actions were inactive in the observed dashboard check; do not rely on them.
  If left grip becomes active, two short taps request explicit Enter.
  For grip gestures, first squeeze <=250 ms; second squeeze begins <=350 ms
  after first release.
  Activity/tracking loss cancels a held recording and requires neutral rearm.
- SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum,
  20 second maximum. In the native app the microphone stream opens after model
  warm-up, stays running between utterances, and discards idle samples; PTT does
  not open/pause/close the device. Quit, worker restart, device failure or capture
  failure closes it. This avoids repeated capture-device transitions but does not
  promise glitch-free playback on every audio stack. Other apps may still
  transmit your voice: this app does **not** mute VRChat or any other app.
- Persistent local Redux worker, correlated bounded pipes, private tmpfs clips,
  cancellation/reaping and deadlines; exact pinned model SHA-256 verification.
  Model imports are lazy and loading is offline. Normal repeats, request-local
  transcription failures and microphone failures retain the loaded model;
  microphone device failure releases the stream for explicit retry. A cancelled
  in-flight request or broken worker protocol may require reloading. No
  cloud/desktop fallback.
- Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable
  handling, UTF-8/control validation and explicit Submit action for Enter.
  Insert ensures a trailing space without doubling an existing one. Enter
  first inserts pending review, releases the text lease, then acquires a fresh
  lease for Submit. Failed/uncertain text never proceeds to Submit; failed
  Submit acquisition never replays text. A full 4096-byte transcript without
  room for a space is preserved with an error, never silently truncated.
- Idempotent user-local release-archive installer: SHA-256, safe extraction,
  atomic current-version selection, retained rollback, runtime/install lock,
  foreign-file refusal and explicit unregister-before-uninstall acknowledgement.

## Deliberately not claimed

**Review-first by default.** An opt-in Auto insert setting now watches the
Xwayland active window, Gamescope focused-window property and exact keyboard
focus from PTT start to IME lease acquisition. Loss/regain, disagreement,
unavailable focus or a held keyboard key falls back to manual review. No
automatic Enter or inferred speech commands. This focus guard has offline
tests but live microphone → automatic insertion is **not yet accepted** on Frame;
it cannot identify arbitrary native Wayland targets. There remains a race
between the final focus check and global input processing. Text delivery is
reported as **input queued**, not application consumption or message delivery.
A request is consumed once even if transport completion is uncertain; no retries.
Unavailable IME acquisition leaves the preview intact.

No VAD, always-listening mode, desktop transcription service, keyboard emulation
fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip bindings
are not guaranteed globally active in every scene/dashboard state, and the app
never enables SteamVR's experimental overlay overrides on your behalf.

This is a compact prototype panel, not yet the proposed polished miniature status
chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery
impact and target application compatibility require further headset work.

## Critical runtime licensing boundary

Redux weights at `fad622f25f303105c20d70e201bcc477c88b620c` are CC-BY-4.0.
The installed **kestrel-kernels 0.7.0** license is different: proprietary, requiring
an M87 Labs written agreement for use; copying/redistribution depends on that
agreement. PyPI availability is not permission. See [third-party notes](third-party.md).

Do not publish a bundled Redux runtime, imply a public release is ready, or rerun
inference while applicable permission is unresolved. Our adapter is implemented;
that does not resolve distribution rights. An alternative runtime would be a
separately scoped and independently licensed implementation—not a silent model swap.

## Developer native build

Requirements: Linux, CMake/C++20, SDL3 >=3.2, Wayland client + scanner, FreeType,
Vulkan headers/loader (plus the system GPU driver at runtime),
and a deliberately provisioned standalone OpenVR **v2.15.6** SDK. No CMake fetches.

```sh
cmake -S . -B build-native -DFRAMEYAP_NATIVE=ON \
  -DOPENVR_ROOT=/absolute/path/to/openvr
cmake --build build-native -j2
ctest --test-dir build-native --output-on-failure
```

On ARM64, select the SDK's `lib/linuxarm64/libopenvr_api.so` explicitly with
`-DOPENVR_LIBRARY=...` if necessary. Do not use an x86 library or another
application's build/runtime. Installable binary has `$ORIGIN/../lib` RUNPATH;
producer must audit and bundle its compatible dependency closure.

The device development trial built SDL **3.2.16** under a project-owned user
prefix with audio enabled and video/render/GPU/joystick/haptic/sensor/camera/tests
examples disabled (`SDL_UNIX_CONSOLE_BUILD=ON`). No OS package modifications.
That is a producer workflow, not an end-user compiler requirement.

## Explicit hardware checks (not CTest)

```sh
./build-native/frameyap --check-input --socket gamescope-0
./build-native/frameyap --check-overlay --assets "$PWD/assets" --mount world
./build-native/frameyap --check-controls --assets "$PWD/assets" --mount world
```

The first acquires/releases an IME without text/actions. The second displays a
five-second inert panel. The third displays a 30-second diagnostic panel and
reports pointer actions, SteamVR grip and PTT action activity, tracking and
read-only legacy grip state, **without microphone or input injection**. Mount
clicks in check modes do not save a preference. A SteamVR error code or inactive
action means the gesture cannot be accepted; raw grip reads do not authorize a
fallback binding.
Action handles alone are not proof that gestures were delivered. Checks must be explicitly launched
while the user expects the panel. Normal CLI/help/version remain inert.

The ARM64 OpenVR loader's default `/data/work/openvrpaths.vrpath` failed on the
observed device. FrameYap respects `VR_PATHREG_OVERRIDE`; when absent, it selects
an existing standard `$XDG_CONFIG_HOME/openvr/openvrpaths.vrpath` (or
`~/.config/openvr/openvrpaths.vrpath`) before initializing OpenVR. No Steam files
are edited and no runtime/session restart is performed.

## Deliberate launch, once runtime permission is settled

Explicit setup downloads only the pinned, openly licensed model:

```sh
python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model"
```

`fetch-model.py` is idempotent for matching hashes, rejects existing mismatched
files, and is never invoked by build/tests or first utterance.

Provide your independently authorized CPU Python environment (moondream 2.4.0,
kestrel 0.8.0) and local weights:

```sh
./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \
  --python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
  --model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head
```

Use a disposable text destination first. `--run` loads the model but does not
record until an explicit recording control. Click Insert only after focusing your
intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery
and terminates only the owned worker. Installer upgrades refuse an active app.

The worker API is intentionally small: a fork can replace the worker implementation
or add its own model/API integration without changing overlay and delivery code.
The default product remains local-only Redux; extending a fork does not authorize
sending existing users' audio to a service.

## Benchmarks

`scripts/benchmark-worker.py` accepts a supplied nonprivate PCM16/16 kHz/mono WAV;
no recording, device input or automatic download. `--show-text` is a separate
explicit disclosure of that fixture's transcript. Use only with an authorized
runtime. See the dated [POC record](evidence/poc-cpu-overlay-2026-09-24.md) for
measurements and their limits; they are not headset/performance acceptance.