FrameYap POC: implementation and validation

This is a standalone native application, not a plugin. Default builds/tests never initialize OpenVR, open a microphone, run ASR, download files or inject input.

Implemented

Opt-in OpenVR RGBA overlay with world-space default and selectable wrist/head mounts, status, recording timer, paginated UTF-8 preview and explicit controls.
Remappable SteamVR actions. Default Steam Frame grip bindings use the observed frame_controller profile. Right: short tap, then hold the second squeeze to record; release to transcribe. Left: two short taps request explicit Enter. First squeeze <=250 ms; second squeeze begins <=350 ms after first release. Activity/tracking loss cancels a held recording and requires neutral rearm.
SDL3 default recording device, mono float32 conversion at 16 kHz, 200 ms minimum, 20 second maximum. Microphone is closed outside actual capture. Other apps may still transmit your voice: this app does not mute VRChat or any other app.
Persistent local Redux worker, correlated bounded pipes, private tmpfs clips, cancellation/reaping and deadlines; exact pinned model SHA-256 verification. Model imports are lazy and loading is offline. Normal repeats, request-local transcription failures and microphone failures retain the loaded model; microphone device streams close outside capture. A cancelled in-flight request or broken worker protocol may require reloading. No cloud/desktop fallback.
Gamescope IME v2 generated bindings, per-action short-lived lease, unavailable handling, UTF-8/control validation and explicit separate Submit action for Enter.
Idempotent user-local release-archive installer: SHA-256, safe extraction, atomic current-version selection, retained rollback, runtime/install lock, foreign-file refusal and explicit unregister-before-uninstall acknowledgement.

Deliberately not claimed

Review-first only. No automatic insertion or inferred commands. A transcript must be explicitly inserted into the current focused destination. We do not yet implement the proposed Xwayland focus-generation observer or safe quick typing. There remains a race with focus changes after user approval. Text delivery is reported as input queued, not application consumption or message delivery. A request is consumed once even if transport completion is uncertain; no retries. Unavailable IME acquisition leaves the preview intact.

No VAD, always-listening mode, desktop transcription service, keyboard emulation fallback, streaming-PC bridge, or automatic Enter. No general undo. Grip bindings are not guaranteed globally active in every scene/dashboard state, and the app never enables SteamVR's experimental overlay overrides on your behalf.

This is a compact prototype panel, not yet the proposed polished miniature status chip. Font coverage/complex shaping, ergonomics, compositor cost, thermal/battery impact and target application compatibility require further headset work.

Critical runtime licensing boundary

Redux weights at fad622f25f303105c20d70e201bcc477c88b620c are CC-BY-4.0. The installed kestrel-kernels 0.7.0 license is different: proprietary, requiring an M87 Labs written agreement for use; copying/redistribution depends on that agreement. PyPI availability is not permission. See third-party notes.

Do not publish a bundled Redux runtime, imply a public release is ready, or rerun inference while applicable permission is unresolved. Our adapter is implemented; that does not resolve distribution rights. An alternative runtime would be a separately scoped and independently licensed implementation—not a silent model swap.

Developer native build

Requirements: Linux, CMake/C++20, SDL3 >=3.2, Wayland client + scanner, FreeType, and a deliberately provisioned standalone OpenVR v2.15.6 SDK. No CMake fetches.

cmake -S . -B build-native -DFRAMEYAP_NATIVE=ON \
  -DOPENVR_ROOT=/absolute/path/to/openvr
cmake --build build-native -j2
ctest --test-dir build-native --output-on-failure

On ARM64, select the SDK's lib/linuxarm64/libopenvr_api.so explicitly with -DOPENVR_LIBRARY=... if necessary. Do not use an x86 library or another application's build/runtime. Installable binary has $ORIGIN/../lib RUNPATH; producer must audit and bundle its compatible dependency closure.

The device development trial built SDL 3.2.16 under a project-owned user prefix with audio enabled and video/render/GPU/joystick/haptic/sensor/camera/tests examples disabled (SDL_UNIX_CONSOLE_BUILD=ON). No OS package modifications. That is a producer workflow, not an end-user compiler requirement.

Explicit hardware checks (not CTest)

./build-native/frameyap --check-input --socket gamescope-0
./build-native/frameyap --check-overlay --assets "$PWD/assets" --mount world
./build-native/frameyap --check-controls --assets "$PWD/assets" --mount world

The first acquires/releases an IME without text/actions. The second displays a five-second inert panel. The third displays a 30-second diagnostic panel and reports pointer actions, SteamVR action activity, tracking and read-only legacy grip state, without microphone or input injection. Mount clicks in check modes do not save a preference. A SteamVR error code or inactive grip means the gesture cannot be accepted; raw grip reads do not authorize a fallback binding. Action handles alone are not proof that gestures were delivered. Checks must be explicitly launched while the user expects the panel. Normal CLI/help/version remain inert.

The ARM64 OpenVR loader's default /data/work/openvrpaths.vrpath failed on the observed device. FrameYap respects VR_PATHREG_OVERRIDE; when absent, it selects an existing standard $XDG_CONFIG_HOME/openvr/openvrpaths.vrpath (or ~/.config/openvr/openvrpaths.vrpath) before initializing OpenVR. No Steam files are edited and no runtime/session restart is performed.

Deliberate launch, once runtime permission is settled

Explicit setup downloads only the pinned, openly licensed model:

python3 scripts/fetch-model.py --destination "$HOME/.local/share/frameyap-model"

fetch-model.py is idempotent for matching hashes, rejects existing mismatched files, and is never invoked by build/tests or first utterance.

Provide your independently authorized CPU Python environment (moondream 2.4.0, kestrel 0.8.0) and local weights:

./build-native/frameyap --run --assets "$PWD/assets" --font /path/font.ttf \
  --python /authorized/runtime/bin/python3 --worker "$PWD/python/frameyap/worker.py" \
  --model "$HOME/.local/share/frameyap-model" --threads 2 --socket gamescope-0 --head

Use a disposable text destination first. --run loads the model but does not record until an explicit recording control. Click Insert only after focusing your intended text field. Quit or SIGINT/SIGTERM closes capture, invalidates delivery and terminates only the owned worker. Installer upgrades refuse an active app.

The worker API is intentionally small: a fork can replace the worker implementation or add its own model/API integration without changing overlay and delivery code. The default product remains local-only Redux; extending a fork does not authorize sending existing users' audio to a service.

Benchmarks

scripts/benchmark-worker.py accepts a supplied nonprivate PCM16/16 kHz/mono WAV; no recording, device input or automatic download. --show-text is a separate explicit disclosure of that fixture's transcript. Use only with an authorized runtime. See the dated POC record for measurements and their limits; they are not headset/performance acceptance.

7.6 KiB Raw Blame History