mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 01:00:18 +02:00
saphidandClaude Opus 5.5
094c12c6d8
Real-Frame testing (2026-09-29) found an unworn headset enters standby within seconds; SetOverlayRaw then returns RequestFailed (23) and the movie died. The player now drops frames during standby, keeps audio and pacing, re-sends stills and the theatre surround after waking, and only errors after five minutes without an accepted frame. A Stop arriving while the player is already shutting down is ignored, so a finished video stays 'ended' instead of 'error: Stopped'. The status now reports the layout's real source (filename/metadata). Docs record the end-to-end device matrix (API, web UI, CLI). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
177 lines
9.6 KiB
Markdown
177 lines
9.6 KiB
Markdown
# Movies, stereo photos and splats
|
||
|
||
Frame Control has its own Frame-side OpenVR player. It uses SteamOS's ffmpeg
|
||
and V4L2 hardware decoder, Python and SteamVR. **No separate player or viewer
|
||
is required.** Chromium and immersive WebXR are not in this playback path.
|
||
|
||
## Use it
|
||
|
||
In **Tools → Media in the headset**, send a file, choose its layout and press
|
||
**Play**. **Theatre** gives it a larger screen and an 85% black surround.
|
||
**Stop** removes both. Refresh reads the library and the player's state.
|
||
The screen follows your head; it isn't a saved world-space panel.
|
||
|
||
The command-line route uses the same player:
|
||
|
||
```sh
|
||
scripts/push-vr-video.sh --launch --theatre ~/Movies/film_SBS.mp4
|
||
scripts/push-vr-video.sh --launch --layout ou ~/Pictures/stereo.png
|
||
scripts/push-vr-video.sh --launch capture.splat
|
||
scripts/push-vr-video.sh --list
|
||
scripts/push-vr-video.sh --stop
|
||
```
|
||
|
||
Uploads live in `~/Videos/FrameControl/<id>/` on the Frame. Each upload gets
|
||
its own directory, so sending another file with the same name doesn't replace
|
||
it. The UI's existing upload limit is 8 GiB. Transfers use the app's rsync/scp
|
||
path; resumable large uploads are not implemented here yet. Existing
|
||
`~/Videos/VR` files and Proton prefixes are left alone. The script no longer
|
||
launches DeoVR, links a Proton prefix, or accepts directories.
|
||
|
||
| Media | Supported preview |
|
||
|---|---|
|
||
| Movies | H.264 / H.265, mono or left-first SBS / top-first OU; audio through the Frame's PulseAudio-compatible server |
|
||
| Stereo photos | PNG / JPEG containing both eyes, SBS or OU |
|
||
| Gaussian splats | Common 32-byte `.splat` records, 1–20,000 Gaussians; a stationary stereo preview |
|
||
|
||
Auto layout reads delimited filename tags: `_SBS`, `_HSBS`, `_LR`, `_OU`,
|
||
`_TB`, `_HOU`, `_HTB`, `_FSBS`, `_FOU`, `_FTB`. SBS/OU without `F` means
|
||
half-resolution packing. Full packing preserves each eye's original aspect.
|
||
It also accepts FFmpeg's `stereo_mode` metadata (`left_right`, `top_bottom`,
|
||
`mono`); left/right and top/bottom metadata are treated as full packing.
|
||
Choose an explicit layout if that assumption doesn't match the file.
|
||
Unknown or conflicting tags ask for a choice, rather than silently flattening
|
||
stereo. The explicit selector always wins. Layout is not guessed from resolution.
|
||
|
||
## What was verified
|
||
|
||
**Verified remotely, 2026-09-28:** SteamOS 0.4.1, BUILD_ID
|
||
`20260925.6191901`, SteamVR 2.18.1. Nobody wore the headset for these checks.
|
||
All test media were generated by us. No paid content or DRM was involved.
|
||
|
||
- Stock `ffmpeg` `h264_v4l2m2m` decoded 150 frames of our 1920×1080 H.264
|
||
test in 0.128 s (decode only); converting all frames to RGBA took 0.242 s.
|
||
- Our ffmpeg → Python → OpenVR path submitted all 150 frames and exited 0.
|
||
The 1280×720 prototype took 4.775 s; the full 1920×1080 player took
|
||
4.618 s. These are wall times, not in-headset frame-rate measurements.
|
||
Finishing a 5 s clip early showed those probes weren't paced, so the final
|
||
player paces output at 30 fps: all 150 frames then completed in **5.025 s**.
|
||
- The H.265 hardware decoder also completed the generated 1080p clip
|
||
(60 frames, exit 0); this is a short compatibility check, not a 4K/8K benchmark.
|
||
- Our actual player, launched through Frame Control's media API, displayed
|
||
SBS and OU photos with red only in the left-eye capture and cyan only in
|
||
the right. OpenVR's `SideBySide_Parallel` flag separates the eyes; we
|
||
rearrange OU rows ourselves.
|
||
- A generated 48-Gaussian `.splat` rendered in our own CPU renderer and appeared
|
||
in both eyes with separate perspective projections.
|
||
- Theatre's owned dark surround and screen appeared in captures. Steam's
|
||
dashboard remained available over them. Stop removed the owned overlays
|
||
and ended the dedicated user service. No global SteamVR setting was changed.
|
||
- A generated H.264/AAC clip reached the Pulse audio output and completed
|
||
all 100 video frames with exit 0. This proves the output path, not audible
|
||
quality or lip sync.
|
||
|
||
|
||
|
||
|
||
|
||
**Verified end to end, 2026-09-29** (same build; headset unworn): uploads
|
||
through the HTTP API, the web UI and `scripts/push-vr-video.sh`, all played by
|
||
the owned player. Our generated test files:
|
||
|
||
| Case | Result |
|
||
|---|---|
|
||
| H.264 half-SBS 1920×1080 with AAC, theatre | 240/240 frames in 8.09 s; audio stream "Frame Control Media" in PulseAudio |
|
||
| H.265 half-OU 1920×1080 | 180/180 frames in 6.03 s; red left eye, cyan right |
|
||
| H.264 full-SBS 3840×1080, `stereo_mode=left_right` only | Detected from metadata; 150/150 frames in 5.03 s |
|
||
| H.264 1280×720, explicit 2D | 150/150 frames in 5.02 s |
|
||
| SBS PNG, OU JPEG (theatre) | Correct eye in each capture |
|
||
| 3,000-Gaussian `.splat` | Rendered in about 5 s, then held until Stop |
|
||
| 2D file on Auto, `_SBS_OU` file, HEIC, VP9 | Refused with the documented message |
|
||
| Second Play while one runs | Refused: "Stop the current media…" |
|
||
|
||
Stop always left the unit inactive, and no player process remained.
|
||
|
||
**Standby (verified):** an unworn Frame turns its displays off a few seconds
|
||
after it wakes. `SetOverlayRaw` then returns `RequestFailed` (23). The
|
||
first run's movie died there. The player now drops frames while the headset
|
||
is in standby, keeps the audio and its clock going, and resumes the picture
|
||
when the headset wakes. The 8 s movie above dropped 40 frames and finished.
|
||
Stills and the theatre surround are re-sent after waking. Five minutes
|
||
without an accepted frame is reported as an error. Headset-view captures
|
||
taken during standby show a flat dark frame, not our screen.
|
||
|
||
|
||
|
||
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
|
||
`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139)
|
||
in the basic appsink probe and the OpenVR probe. We do not ship that route.
|
||
The ffmpeg path above completed instead.
|
||
|
||
## Limits and blockers
|
||
|
||
- **Not verified:** worn-headset comfort, sound quality/lip sync, long movies,
|
||
4K/8K decode, HDR, controller interaction or behaviour on other OS builds.
|
||
Output is bounded to 1920×1080 packed pixels before OU rearrangement.
|
||
- **Not implemented:** pause, seeking, subtitles, playlists, right-eye-first
|
||
layouts, non-square pixel correction, VR180/360 projection and fisheye.
|
||
This preview is a flat stereo screen, not a dome player.
|
||
- **Native spatial-photo blocker:** HEIC/HEIF/AVIF/MPO stereo-container
|
||
extraction isn't implemented in our viewer. We reject these rather than
|
||
displaying one image and calling it spatial. Export both eyes to PNG/JPEG
|
||
first. This is a current implementation gap, not a claim that the Frame
|
||
cannot support these containers.
|
||
- **Large/immersive splat blocker:** the owned CPU rasterizer projects 3D
|
||
covariance into each eye and alpha-composites Gaussians, but renders a
|
||
fixed view at 320×240 per eye. It caps each Gaussian footprint at 32 pixels
|
||
and normalizes the scene to a two-metre box. Large scenes, PLY/SPZ, live
|
||
head-position parallax and navigation need a GPU scene renderer; this
|
||
version rejects files above 20,000 records. It is a stereo preview, not
|
||
an immersive walk-through.
|
||
- A single player can run at a time. It stops at EOF, on **Stop**, or after
|
||
four hours in any case, so longer movies are cut off. Still images remain
|
||
until Stop or that timeout. There is no delete action yet: remove old
|
||
uploads from `~/Videos/FrameControl/` over SSH. The log is
|
||
`~/.local/share/frame-control/media/player.log` on the Frame.
|
||
- **Panel/stream theatre remains outside this media slice:** SteamVR's
|
||
`vrcmd --dock-overlay` accepts `theater`, but docking an existing panel
|
||
and its dimming behaviour were not verified here. The shared headset had
|
||
workspace and stream tests active. We did not reposition their panels.
|
||
Integration with [#22](https://github.com/saphid/frame-control/issues/22)
|
||
and [#31](https://github.com/saphid/frame-control/issues/31) can use the
|
||
owned rendering hook below; no second stream/window manager is introduced.
|
||
|
||
## Integration hook
|
||
|
||
`POST /api/upload` with `X-Mode: media` and `X-Filename` sends a file and
|
||
returns its `id`. `POST /api/media` accepts:
|
||
|
||
```json
|
||
{"action":"play","id":"<32 hex characters>/film_SBS.mp4","layout":"auto","theatre":true}
|
||
```
|
||
|
||
Other actions are `list`, `status`, `stop`. These use the normal `X-Frame-UI`
|
||
guard. Play reports **starting**, not a claim that frames reached the headset;
|
||
read status for `playing`, `ended`, `stopped` or `error`.
|
||
The own-code files are copied to `~/.local/share/frame-control/media/` and
|
||
run in the `frame-control-media.service` systemd user unit. Stop affects only
|
||
that unit, including its decoder child.
|
||
|
||
For a Frame-side stream producer, `frame_media_player.Overlay` exposes
|
||
`create(key, width, distance, stereo=False, aspect=1, order=1)`,
|
||
`pixels(handle, rgba, width, height)` and `close()`. Use an owned unique key,
|
||
pass interleaved RGBA bytes (left/right halves for stereo) and always close in
|
||
`finally`. `create` uses a head-relative transform; it does not move another
|
||
app's panel. The player demonstrates a separate black surround overlay.
|
||
`frame_media.stereo_pixels` converts OU to SBS. This is the narrow rendering
|
||
hook for stream/workspace work; it doesn't capture or manage a Mac/PC stream.
|
||
|
||
## Optional separate app
|
||
|
||
You can independently install **DeoVR Video Player** (free Steam app 837380)
|
||
if you prefer its VR180/360 features. Earlier tests on SteamOS 0.3.0,
|
||
build `20260922.6101926` (2026-09-25), verified its OpenVR initialization and
|
||
8K H.265 streamed VR180 decoding under Proton. Frame Control's media features
|
||
neither install nor launch it, and do not depend on it. Its behaviour and
|
||
file-naming conventions are not evidence about our player.
|