FrameYap

Standalone, on-device voice typing for Steam Frame. MIT licensed. Early release (v0.1 in progress).

Implemented: native OpenVR overlay, remappable controller actions, bounded SDL3 capture, persistent local Parakeet Redux worker, preview/explicit insertion through Gamescope, and an idempotent user-local installer. No Steam store AppID, sudo, desktop ASR server, cloud fallback or unrelated application dependency.

Status: native ARM64 build, CPU inference on a public clip, overlay visibility, Gamescope discovery and native-only installation have been exercised on Frame. Live microphone → reviewed text → real target delivery is not yet accepted.

Inference runtime: Redux weights are CC-BY-4.0 and run locally through the moondream Python package (its Kestrel runtime states that local inference is free and needs no API key). The build and tests never download it; the installer or you install it from PyPI into a Python environment. See third-party notes. No GitHub release is published yet.

Controls

• Right X (default Frame binding): hold to record; release to transcribe. Repeated presses reached the controls-only diagnostic on Frame; live mic capture through this shortcut still needs guided acceptance. This PTT action is remappable through SteamVR bindings.
• Right B: cancel/discard (or close quick chat). Right A: insert reviewed text with a trailing space. Right Y: open quick chat; press again to cycle its highlighted choice. Submit on the overlay (or double-tap left grip when active) sends that choice verbatim then Enter; outside quick chat it inserts pending review + Enter, or sends Enter alone when there is no text. Edit the quick_inputs list in $XDG_CONFIG_HOME/frameyap/config.json (restart to apply). Nothing submits automatically. Auto Insert is opt-in and off by default.
• Overlay: Record/Stop, Cancel, paginated preview, Insert, Submit and Hold Quit (hold the button for 0.9 seconds before releasing). Review and settings share one Inconsolata/neon-framed surface. The header shows local time/date; Settings selects 12/24-hour time and date format/off. Hold the thin bar below the panel to freely move and rotate it with your controller; release to leave it at that pose. Drag the external lower-right bracket to scale it (world, head or wrist), keeping the upper-left anchored. Both handles use the app's gradient in transparent margins, like Steam's window handles. Settings → Lock grab/scale hides and disables both handles; the lock is saved. Position/size changes last for this run only. Wrist mounting fades the panel as its full orientation turns away from an upright viewer-facing pose (60°–75°), hiding interaction past that angle; world and head mounting do not fade. Bindings requests SteamVR's binding editor directly. Existing SteamVR overrides may supersede defaults. World-space by default; settings offer left wrist, right wrist and head mounting, plus recenter. Dashboard lasers provide clickable controls. Settings → Lasers anytime is an opt-in, default-off system-wide laser mode while the panel is visible; it may affect games and is separate from experimental input overrides.
• Theme and controls: optional $XDG_CONFIG_HOME/frameyap/config.json selects panel colors, a font path and Frame controller button mappings; missing fonts fall back to bundled Inconsolata. The installer creates/checks this file and backs it up before repairs. See overlay configuration.
• Advanced debugging: Settings toggle / "advanced_debug": true in config. Off by default. Restarts the worker and discards current work; full exceptions, worker output and transcripts go to private, bounded local logs. No raw audio archive. See diagnostics.
• Experimental input priority: set "input_priority": "experimental" in that config and enable SteamVR's Developer option Enable global input from overlays. FrameYap then requests priority for its bound controller sources. This may consume controls used by games or the dashboard; coexistence on Frame is under test. The default is "normal"; restart FrameYap after changing it.
• Review by default: focus your destination, then press Insert (text + space) or explicitly Enter (text + space, then Enter). Settings → Auto insert is off by default: when enabled, it queues text + space only if Xwayland keyboard focus, active window and Gamescope focus match continuously from recording through delivery. Any uncertainty leaves a preview for manual Insert; it never sends Enter. This is not yet validated for live transcription on Frame. Maximum clip 20 seconds; accidental taps under 200 ms are discarded.
• While the native app is Ready, it keeps the mic device open and discards idle audio instead of opening/closing on every PTT. Quit releases the device. Other apps may still hear/transmit your voice; FrameYap does not mute them.

Physical gesture timing, global bindings during games, mic capture, target-app compatibility and headset comfort still need coordinated validation. Successful API initialization is not delivered input or human acceptance.

Build and test offline

CMake 3.20+, C++20 compiler. Python 3.10+ runs the additional hardware-free tests.

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure
./build/frameyap --help

Default build has no hardware backends. It never downloads packages/models or initializes SteamVR, a microphone or input injection. Native dependencies and explicit launch/check commands are documented in the build guide. --version uses an ISO-like UTC build timestamp (with Git hash when available), not a numbered release; use that same tag when packaging the binary.

Installation

The real installer accepts a versioned, checksummed prebuilt ARM64 archive:

sh install.sh --archive /path/to/frameyap-VERSION-linux-aarch64.tar.gz \
  --sha256 ARCHIVE_SHA256 --version VERSION

This is the local-artifact command shape, not an available public download. Installation is user-local, retains rollback, refuses active-app upgrades and foreign files, and does not launch or register automatically. Registration uses OpenVR application key local.frameyap.overlay, not a Steam store AppID. Autolaunch is opt-in. An installed desktop entry can be selected manually as a non-Steam shortcut. A basic launch from Steam's Non-Steam section opened the panel on one Frame; registration alone did not show an entry in the first checked dashboard menu. See packaging and lifecycle. For a native-only install, menu-driven inference needs a Python runtime you provide and the pinned model. Configure their absolute paths in ~/.config/frameyap/paths.conf as described in the packaging guide; they are never fetched or bundled implicitly.

A pinned GitHub one-command route is implemented in install.sh --version TAG, but do not advertise or run it as a working public installation until a vetted release exists. Native-only artifacts support the overlay/checks without an end-user compiler; a bundled-ASR distribution is not yet offered.

Project map

• Build guide: implemented boundaries, build, controls, explicit tests.
• Design: full target design; some features remain proposed.
• Installer design and packaging.
• Overlay, worker protocol, dependency/license inventory.
• TODO: release plan and open work items.

Dated device-test records are kept locally (untracked) and are not authority for current device availability. Live microphone → reviewed text → target delivery has not been formally accepted; see Status above.

No recordings, transcripts, private logs, model weights or runtime binaries are committed. The worker boundary is intentionally small for forks experimenting with other models/APIs; the default remains local-only Redux.