mirror of
https://github.com/baketnk/frame-yap.git
synced 2026-10-04 22:00:03 +02:00
docs: user-facing README; move technical detail to docs/development.md
README now covers install, launch, controls, troubleshooting, privacy and update/uninstall for users. Validation status, version format, CLI and full control semantics move to docs/development.md. TODO records the v0.1.202609251524 release and the owner's fresh curl | sh install. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
baketnkandClaude Opus 5.5 committed
1 parent
3f451087ca
commit
b8eeb50b39
3 files changed
+105
-38
No files matched your search
@@ -1,48 +1,66 @@
|
||||
# FrameYap
|
||||
|
||||
Voice typing on Steam Frame, with recognition on the headset rather than a desktop or cloud server.
|
||||
Hold a controller button to record, review the transcript, then deliberately type it into the focused app.
|
||||
Standalone OpenVR overlay; no Steam store AppID or sudo. First release: v0.1.202609251524.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format.
|
||||
- For voice recognition, a **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0, CPU Torch 2.8.0) and the pinned local Parakeet Redux model. Neither is bundled; each has its own explicit install step: `sh install.sh --install-model --backend redux --yes` and `sh install.sh --install-runtime --yes` (needs Python 3.10–3.13 with venv; about 200 MB download). Use `--print-plan` first to see exactly what each fetches. There is no fallback ASR service.
|
||||
- A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](docs/build.md).
|
||||
On-device voice typing for Steam Frame. Hold a button, speak, check the text, and type it into whatever app has focus. Speech recognition runs on the headset itself; nothing is sent to the cloud.
|
||||
|
||||
## Install
|
||||
|
||||
On the Frame, open a terminal (desktop mode's Konsole, or SSH) and run:
|
||||
1. On the Frame, switch to **desktop mode** and open **Konsole** (or connect over SSH).
|
||||
2. Paste this and press Enter:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/baketnk/frame-yap/releases/latest/download/install.sh | sh
|
||||
```
|
||||
```sh
|
||||
curl -fsSL https://github.com/baketnk/frame-yap/releases/latest/download/install.sh | sh
|
||||
```
|
||||
|
||||
You can download and read `install.sh` first; it is one self-contained file. The installer pins its own release (currently `v0.1.202609251524`) and verifies the archive's SHA-256. It asks three y/n questions: install FrameYap, download the Parakeet Redux speech model (about 180 MB, CC-BY-4.0), and pip-install the CPU Python runtime (moondream/Kestrel with CPU Torch; about 200 MB download, roughly 1–1.5 GB on disk). Everything goes under your home directory: no sudo, compiler or Steam store AppID. Afterwards, launch **FrameYap** from the desktop application menu.
|
||||
3. Answer **y** to the three questions:
|
||||
- install FrameYap,
|
||||
- download the speech model (about 180 MB),
|
||||
- install the speech runtime (about 200 MB download, roughly 1–1.5 GB on disk).
|
||||
|
||||
The release archive contains FrameYap, SDL3 and the OpenVR client library only; Kestrel, Torch, moondream and the model weights are fetched on your machine from PyPI and Hugging Face. For automation, the same steps are `sh install.sh --yes`, `sh install.sh --install-model --backend redux --yes` and `sh install.sh --install-runtime --yes`; add `--print-plan --json` to preview any step. Maintainers build release archives on Linux ARM64 with `sh scripts/build-release.sh WORKDIR VERSION`. See [packaging](docs/packaging.md) and [install design](docs/install-design.md).
|
||||
The downloads can take a few minutes with little output; that is normal.
|
||||
|
||||
## Controls (default Frame binding)
|
||||
Everything installs into your home folder. No sudo, no compiler, no Steam store page. Prefer to read the script first? Download [`install.sh`](https://github.com/baketnk/frame-yap/releases/latest/download/install.sh) and open it; it is a single file.
|
||||
|
||||
| Button / control | Action |
|
||||
## Launch
|
||||
|
||||
Open **FrameYap** from the desktop application menu, or in the headset from your Steam library's **Non-Steam** section. The panel appears in front of you; drag the bar under it to move it, or the corner handle to resize.
|
||||
|
||||
## Use it
|
||||
|
||||
| Button | What it does |
|
||||
| --- | --- |
|
||||
| Right X, hold / release | Record while held; release to transcribe. |
|
||||
| Right B | Cancel/discard, or close the Quick phrases picker. |
|
||||
| Right A | **Type:** queue reviewed text, normally with a trailing space. With nothing to review, a press queues Enter alone (so a quick double press types then submits). |
|
||||
| Right Y | Open **Quick phrases**; press again to cycle the selection. |
|
||||
| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. |
|
||||
| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. |
|
||||
| Overlay Type / Type + Enter | The same deliberate text / Enter actions. |
|
||||
| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). |
|
||||
| **X** (right), hold | Record while held; let go to transcribe. |
|
||||
| **A** (right) | Type the text into the focused app. With nothing to type, A presses **Enter** (so A, A types and submits). |
|
||||
| **B** (right) | Cancel / discard. |
|
||||
| **Y** (right) | Quick phrases; press again to pick the next one. |
|
||||
| Left grip, double-tap | Type + Enter. |
|
||||
|
||||
Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Type and Type + Enter now require a verified Xwayland target and use paced direct typing; Cancel stops remaining batches, never undoing prior input. The clipboard stays untouched. Check the destination before typing. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](docs/overlay.md).
|
||||
You can also click the panel's buttons with the laser pointer. Nothing is ever typed or submitted without you pressing a button. Buttons can be remapped with **Bindings** on the panel.
|
||||
|
||||
## Status / not yet validated
|
||||
The header shows battery levels for your controllers and headset. The **Buttons paused** badge means the Steam menu is open: controller buttons go to Steam, so use the pointer instead (or close the menu).
|
||||
|
||||
Native ARM64 build, CPU inference, overlay visibility, Gamescope API discovery and native-only installation have been exercised on Frame. The owner uses live microphone → reviewed transcript → paced delivery into real apps day to day, and the earlier front-prefix loss on repeated submissions (P1) is no longer observed. General app compatibility (for example browser fields), Auto insert with speech, physical resize, battery/thermal cost and a clean-account install are not yet validated. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it.
|
||||
## Troubleshooting
|
||||
|
||||
The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](docs/overlay.md), [worker](docs/worker.md), [design](docs/design.md).
|
||||
- **Panel says Unavailable, or shows an error while loading:** the speech model or runtime is missing (for example, you answered **n** during install). Run these, then restart FrameYap:
|
||||
|
||||
Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.<timestamp>` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](docs/third-party.md); [TODO](TODO.md).
|
||||
```sh
|
||||
sh ~/.local/share/frameyap/current/bin/install.sh --install-model --backend redux --yes
|
||||
sh ~/.local/share/frameyap/current/bin/install.sh --install-runtime --yes
|
||||
```
|
||||
- **Text went to the wrong place:** FrameYap types into the currently focused window. Click the target text field first.
|
||||
- **Controller buttons do nothing:** check for the **Buttons paused** badge (see above).
|
||||
|
||||
No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection.
|
||||
## Privacy
|
||||
|
||||
Recognition runs locally with the Parakeet Redux model. The installer downloads FrameYap from GitHub, the model from Hugging Face and the Python packages (moondream, Kestrel, CPU Torch) from PyPI and PyTorch; after that, FrameYap works offline. By default the microphone stays open while FrameYap is running and idle audio is thrown away; turn on **Settings → Close mic when idle** to close it between recordings (it may clip the first syllable). Recordings and transcripts are not kept, unless you turn on **Advanced debug**, whose logs may contain speech.
|
||||
|
||||
## Update and uninstall
|
||||
|
||||
- **Update:** run the install command again and answer **y** to FrameYap. Answer **n** to the model and runtime questions unless the release notes say they changed.
|
||||
- **Settings:** `~/.config/frameyap/config.json` (Quick phrases, theme, placement). Restart FrameYap after editing.
|
||||
- **Uninstall:** `sh ~/.local/share/frameyap/current/bin/install.sh --uninstall --unregistered`. Your settings and downloaded model are kept. Delete `~/.local/share/frameyap` and `~/.config/frameyap` to remove everything.
|
||||
|
||||
## More
|
||||
|
||||
- [Development notes](docs/development.md): build from source, validation status, full control details.
|
||||
- [Third-party components and licenses](docs/third-party.md). FrameYap itself is [MIT licensed](LICENSE).
|
||||
- The release archive contains FrameYap, SDL3 and the OpenVR client library only; the speech runtime and model are downloaded on your machine, not redistributed.
|
||||
@@ -50,11 +50,14 @@ accepted; local code/tests cannot establish a fixed delivery regression.
|
||||
|
||||
## A. First release (v0.1) blockers
|
||||
|
||||
Remaining release gates (2026-09-25): D4 is implemented; next a
|
||||
fresh clean install on Frame (install → `--install-model` → `--install-runtime`) that also checks the missing-runtime panel message
|
||||
(H), then tag `v0.1.<timestamp>` and update the README install section.
|
||||
**v0.1.202609251524 released (2026-09-25).** Built on the Frame with
|
||||
`scripts/build-release.sh`, published on GitHub, and installed fresh on the
|
||||
Frame by the owner via `curl | sh` (install → model → pip runtime), followed
|
||||
by successful live voice typing. The README is now user-facing; technical
|
||||
detail lives in `docs/development.md`. Still unchecked: the exact
|
||||
missing-runtime panel message and the battery readouts on Frame.
|
||||
|
||||
- [x] **A8. UI polish batch (local; deploy pending).** Removed the fixed
|
||||
- [x] **A8. UI polish batch (shipped in v0.1.202609251524).** Removed the fixed
|
||||
"Type adds a space" hint, grew the review card to six lines to close the empty
|
||||
band, added L/HMD/R battery readouts and a Buttons ready/paused dashboard chip,
|
||||
and made Type with nothing to review send Enter. Which batteries Frame reports
|
||||
@@ -213,6 +216,6 @@ this app is future design and out of scope for v0.1.
|
||||
|
||||
## Open questions
|
||||
|
||||
The clean-account Frame install (including D4's ARM64 pip run and the
|
||||
missing-runtime message) is the remaining unresolved gate; checked source tasks
|
||||
do not imply it.
|
||||
The fresh Frame install passed (2026-09-25). Open follow-ups: the
|
||||
missing-runtime panel message, which batteries Frame reports, browser text
|
||||
fields (G1), and whether the installer's y/n defaults suit first-time users.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Development notes
|
||||
|
||||
Technical details moved out of the user-facing README. See also
|
||||
[build](build.md), [overlay](overlay.md), [worker](worker.md),
|
||||
[packaging](packaging.md), [install design](install-design.md),
|
||||
[design](design.md) and [third-party notes](third-party.md).
|
||||
|
||||
## Requirements (detail)
|
||||
|
||||
- Steam Frame with usable SteamVR/OpenVR and Gamescope for the **native** overlay and text delivery; Linux ARM64/glibc for the current installer payload format.
|
||||
- For voice recognition, a **CPU Python runtime** (moondream 2.4.0 / Kestrel 0.8.0, CPU Torch 2.8.0) and the pinned local Parakeet Redux model. Neither is bundled; each has its own explicit install step: `sh install.sh --install-model --backend redux --yes` and `sh install.sh --install-runtime --yes` (needs Python 3.10–3.13 with venv; about 200 MB download). Use `--print-plan` first to see exactly what each fetches. There is no fallback ASR service.
|
||||
- A local source build needs CMake 3.20+, C++20 and explicit native libraries/SDK; the default hardware-free build needs only CMake and C++20. See [build requirements](build.md).
|
||||
|
||||
## Controls (full semantics)
|
||||
|
||||
| Button / control | Action |
|
||||
| --- | --- |
|
||||
| Right X, hold / release | Record while held; release to transcribe. |
|
||||
| Right B | Cancel/discard, or close the Quick phrases picker. |
|
||||
| Right A | **Type:** queue reviewed text, normally with a trailing space. With nothing to review, a press queues Enter alone (so a quick double press types then submits). |
|
||||
| Right Y | Open **Quick phrases**; press again to cycle the selection. |
|
||||
| Left grip, double-tap | **Type + Enter:** queue the selected phrase verbatim + Enter, pending review (normally + space) then Enter, or Enter alone if neither exists. |
|
||||
| Overlay Record / Stop | Click-to-start/stop alternative to the PTT binding. |
|
||||
| Overlay Type / Type + Enter | The same deliberate text / Enter actions. |
|
||||
| Overlay Hold Quit | Hold for 0.9 seconds, then release to quit (prevents accidental exit). |
|
||||
|
||||
Controls are remappable in SteamVR. Grip gestures may be unavailable with the dashboard open; pointer controls are an alternative. At the 4096-byte transcript limit, Type preserves the full text without appending a space if none fits. No speech commands, automatic Enter or automatic submit. Review is the default; Settings → Auto insert is opt-in, normally text + space only under continuously observed Xwayland focus. Type and Type + Enter now require a verified Xwayland target and use paced direct typing; Cancel stops remaining batches, never undoing prior input. The clipboard stays untouched. Check the destination before typing. Edit literal Quick phrases (`quick_inputs`) in `$XDG_CONFIG_HOME/frameyap/config.json`, then restart. See [overlay and settings](overlay.md).
|
||||
|
||||
## Validation status
|
||||
|
||||
Native ARM64 build, CPU inference, overlay visibility, Gamescope API discovery and native-only installation have been exercised on Frame. The owner uses live microphone → reviewed transcript → paced delivery into real apps day to day, and the earlier front-prefix loss on repeated submissions (P1) is no longer observed. General app compatibility (for example browser fields), Auto insert with speech, physical resize, and battery/thermal cost are not yet validated. On 2026-09-25 the owner removed the previous install, ran the published v0.1.202609251524 `curl | sh` installer on the Frame (FrameYap, model and pip runtime) and voice-typed successfully; the ARM64 runtime install is therefore exercised, but the missing-runtime panel message and battery readouts on Frame were not separately checked. Local code/build status does not mean the device was updated. A completed Gamescope IME call means *input queued*, not that an app consumed or submitted it.
|
||||
|
||||
The native app normally keeps the mic device open while Ready and discards idle audio; it never mutes other applications' microphones. Settings → **Close mic when idle** (default OFF) closes it between clips, but reopening on PTT can cause an audio spike, delay or first-syllable clipping. **Lasers anytime** (default OFF) requests system-wide lasers while the panel is visible, potentially affecting games; it is not SteamVR's experimental input override. Review, settings, placement and diagnostic details: [overlay](overlay.md), [worker](worker.md), [design](design.md).
|
||||
|
||||
Version output is numeric `frameyap MAJOR.MINOR.YYYYMMDDHHMM` (currently `0.1`); a development build may print `git HASH` and optionally `(uncommitted changes)` on a **separate** line. The UTC timestamp is set at configuration time (`SOURCE_DATE_EPOCH` can supply it); release archives must be built from a clean `v0.1.<timestamp>` tag. Offline model inventory and pinned SHA-256 checks in a source-tree build: `./build/frameyap --list-models`, `./build/frameyap --check-model redux --model-dir /absolute/model`, or `python3 scripts/model-status.py` with the same options. Packaging retains the verifier script for installed CLI use, which still needs a clean-account artifact check. Native `--run` accepts `--backend ID`, `--model-store /absolute/dir`, `--manifest-dir /absolute/dir` overrides; the installed launcher passes explicit flags through to the binary. These locally wired paths do not imply an installed release was tested or a model/runtime was supplied. No model or runtime is downloaded by status checks or on normal launch. [Dependency licenses and outstanding release audit](third-party.md); [TODO](../TODO.md).
|
||||
|
||||
No recordings, transcripts, private logs, weights or CPU runtime binaries are committed. No default build/test downloads or initializes SteamVR, microphone or input injection.
|
||||
|
||||
## Release builds
|
||||
|
||||
Release archives are built on Linux ARM64 from a clean `v0.1.<timestamp>` tag
|
||||
with `sh scripts/build-release.sh WORKDIR VERSION`, which fetches SHA-256-pinned
|
||||
SDL3 and OpenVR SDK sources, builds, tests, stages and packages. The release
|
||||
commit sets `RELEASE_VERSION` in `scripts/install_payload.py` (then
|
||||
`python3 scripts/sync-installer.py`); upload the archive, its `.sha256` and
|
||||
`install.sh` to the GitHub release.
|
||||
Reference in new issue
Block a user