mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 14:00:34 +02:00
saphidandClaude Opus 5.5
f81f70ed5d
- Stop is a no-op when the collected player unit is already gone (raw systemctl stop exits 5 on the Frame; verified 2026-09-29). - Surface systemd-run stderr when the player can't start. - Keep the copy error if the cleanup ssh also fails; reject upload names that the play path can never accept. - Allow 60 s for play (ffprobe 30 s + systemd-run 15 s remote). - Docs: four-hour cap is unconditional; no delete action yet; fix a garbled timing sentence. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
149 lines
8.0 KiB
Markdown
149 lines
8.0 KiB
Markdown
# Movies, stereo photos and splats
|
||
|
||
Frame Control has its own Frame-side OpenVR player. It uses SteamOS's ffmpeg
|
||
and V4L2 hardware decoder, Python and SteamVR. **No separate player or viewer
|
||
is required.** Chromium and immersive WebXR are not in this playback path.
|
||
|
||
## Use it
|
||
|
||
In **Tools → Media in the headset**, send a file, choose its layout and press
|
||
**Play**. **Theatre** gives it a larger screen and an 85% black surround.
|
||
**Stop** removes both. Refresh reads the library and the player's state.
|
||
The screen follows your head; it isn't a saved world-space panel.
|
||
|
||
The command-line route uses the same player:
|
||
|
||
```sh
|
||
scripts/push-vr-video.sh --launch --theatre ~/Movies/film_SBS.mp4
|
||
scripts/push-vr-video.sh --launch --layout ou ~/Pictures/stereo.png
|
||
scripts/push-vr-video.sh --launch capture.splat
|
||
scripts/push-vr-video.sh --list
|
||
scripts/push-vr-video.sh --stop
|
||
```
|
||
|
||
Uploads live in `~/Videos/FrameControl/<id>/` on the Frame. Each upload gets
|
||
its own directory, so sending another file with the same name doesn't replace
|
||
it. The UI's existing upload limit is 8 GiB. Transfers use the app's rsync/scp
|
||
path; resumable large uploads are not implemented here yet. Existing
|
||
`~/Videos/VR` files and Proton prefixes are left alone. The script no longer
|
||
launches DeoVR, links a Proton prefix, or accepts directories.
|
||
|
||
| Media | Supported preview |
|
||
|---|---|
|
||
| Movies | H.264 / H.265, mono or left-first SBS / top-first OU; audio through the Frame's PulseAudio-compatible server |
|
||
| Stereo photos | PNG / JPEG containing both eyes, SBS or OU |
|
||
| Gaussian splats | Common 32-byte `.splat` records, 1–20,000 Gaussians; a stationary stereo preview |
|
||
|
||
Auto layout reads delimited filename tags: `_SBS`, `_HSBS`, `_LR`, `_OU`,
|
||
`_TB`, `_HOU`, `_HTB`, `_FSBS`, `_FOU`, `_FTB`. SBS/OU without `F` means
|
||
half-resolution packing. Full packing preserves each eye's original aspect.
|
||
It also accepts FFmpeg's `stereo_mode` metadata (`left_right`, `top_bottom`,
|
||
`mono`); left/right and top/bottom metadata are treated as full packing.
|
||
Choose an explicit layout if that assumption doesn't match the file.
|
||
Unknown or conflicting tags ask for a choice, rather than silently flattening
|
||
stereo. The explicit selector always wins. Layout is not guessed from resolution.
|
||
|
||
## What was verified
|
||
|
||
**Verified remotely, 2026-09-28:** SteamOS 0.4.1, BUILD_ID
|
||
`20260925.6191901`, SteamVR 2.18.1. Nobody wore the headset for these checks.
|
||
All test media were generated by us. No paid content or DRM was involved.
|
||
|
||
- Stock `ffmpeg` `h264_v4l2m2m` decoded 150 frames of our 1920×1080 H.264
|
||
test in 0.128 s (decode only); converting all frames to RGBA took 0.242 s.
|
||
- Our ffmpeg → Python → OpenVR path submitted all 150 frames and exited 0.
|
||
The 1280×720 prototype took 4.775 s; the full 1920×1080 player took
|
||
4.618 s. These are wall times, not in-headset frame-rate measurements.
|
||
Finishing a 5 s clip early showed those probes weren't paced, so the final
|
||
player paces output at 30 fps: all 150 frames then completed in **5.025 s**.
|
||
- The H.265 hardware decoder also completed the generated 1080p clip
|
||
(60 frames, exit 0); this is a short compatibility check, not a 4K/8K benchmark.
|
||
- Our actual player, launched through Frame Control's media API, displayed
|
||
SBS and OU photos with red only in the left-eye capture and cyan only in
|
||
the right. OpenVR's `SideBySide_Parallel` flag separates the eyes; we
|
||
rearrange OU rows ourselves.
|
||
- A generated 48-Gaussian `.splat` rendered in our own CPU renderer and appeared
|
||
in both eyes with separate perspective projections.
|
||
- Theatre's owned dark surround and screen appeared in captures. Steam's
|
||
dashboard remained available over them. Stop removed the owned overlays
|
||
and ended the dedicated user service. No global SteamVR setting was changed.
|
||
- A generated H.264/AAC clip reached the Pulse audio output and completed
|
||
all 100 video frames with exit 0. This proves the output path, not audible
|
||
quality or lip sync.
|
||
|
||
|
||
|
||
|
||
|
||
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
|
||
`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139)
|
||
in the basic appsink probe and the OpenVR probe. We do not ship that route.
|
||
The ffmpeg path above completed instead.
|
||
|
||
## Limits and blockers
|
||
|
||
- **Not verified:** worn-headset comfort, sound quality/lip sync, long movies,
|
||
4K/8K decode, HDR, controller interaction or behaviour on other OS builds.
|
||
Output is bounded to 1920×1080 packed pixels before OU rearrangement.
|
||
- **Not implemented:** pause, seeking, subtitles, playlists, right-eye-first
|
||
layouts, non-square pixel correction, VR180/360 projection and fisheye.
|
||
This preview is a flat stereo screen, not a dome player.
|
||
- **Native spatial-photo blocker:** HEIC/HEIF/AVIF/MPO stereo-container
|
||
extraction isn't implemented in our viewer. We reject these rather than
|
||
displaying one image and calling it spatial. Export both eyes to PNG/JPEG
|
||
first. This is a current implementation gap, not a claim that the Frame
|
||
cannot support these containers.
|
||
- **Large/immersive splat blocker:** the owned CPU rasterizer projects 3D
|
||
covariance into each eye and alpha-composites Gaussians, but renders a
|
||
fixed view at 320×240 per eye. It caps each Gaussian footprint at 32 pixels
|
||
and normalizes the scene to a two-metre box. Large scenes, PLY/SPZ, live
|
||
head-position parallax and navigation need a GPU scene renderer; this
|
||
version rejects files above 20,000 records. It is a stereo preview, not
|
||
an immersive walk-through.
|
||
- A single player can run at a time. It stops at EOF, on **Stop**, or after
|
||
four hours in any case, so longer movies are cut off. Still images remain
|
||
until Stop or that timeout. There is no delete action yet: remove old
|
||
uploads from `~/Videos/FrameControl/` over SSH. The log is
|
||
`~/.local/share/frame-control/media/player.log` on the Frame.
|
||
- **Panel/stream theatre remains outside this media slice:** SteamVR's
|
||
`vrcmd --dock-overlay` accepts `theater`, but docking an existing panel
|
||
and its dimming behaviour were not verified here. The shared headset had
|
||
workspace and stream tests active. We did not reposition their panels.
|
||
Integration with [#22](https://github.com/saphid/frame-control/issues/22)
|
||
and [#31](https://github.com/saphid/frame-control/issues/31) can use the
|
||
owned rendering hook below; no second stream/window manager is introduced.
|
||
|
||
## Integration hook
|
||
|
||
`POST /api/upload` with `X-Mode: media` and `X-Filename` sends a file and
|
||
returns its `id`. `POST /api/media` accepts:
|
||
|
||
```json
|
||
{"action":"play","id":"<32 hex characters>/film_SBS.mp4","layout":"auto","theatre":true}
|
||
```
|
||
|
||
Other actions are `list`, `status`, `stop`. These use the normal `X-Frame-UI`
|
||
guard. Play reports **starting**, not a claim that frames reached the headset;
|
||
read status for `playing`, `ended`, `stopped` or `error`.
|
||
The own-code files are copied to `~/.local/share/frame-control/media/` and
|
||
run in the `frame-control-media.service` systemd user unit. Stop affects only
|
||
that unit, including its decoder child.
|
||
|
||
For a Frame-side stream producer, `frame_media_player.Overlay` exposes
|
||
`create(key, width, distance, stereo=False, aspect=1, order=1)`,
|
||
`pixels(handle, rgba, width, height)` and `close()`. Use an owned unique key,
|
||
pass interleaved RGBA bytes (left/right halves for stereo) and always close in
|
||
`finally`. `create` uses a head-relative transform; it does not move another
|
||
app's panel. The player demonstrates a separate black surround overlay.
|
||
`frame_media.stereo_pixels` converts OU to SBS. This is the narrow rendering
|
||
hook for stream/workspace work; it doesn't capture or manage a Mac/PC stream.
|
||
|
||
## Optional separate app
|
||
|
||
You can independently install **DeoVR Video Player** (free Steam app 837380)
|
||
if you prefer its VR180/360 features. Earlier tests on SteamOS 0.3.0,
|
||
build `20260922.6101926` (2026-09-25), verified its OpenVR initialization and
|
||
8K H.265 streamed VR180 decoding under Proton. Frame Control's media features
|
||
neither install nor launch it, and do not depend on it. Its behaviour and
|
||
file-naming conventions are not evidence about our player.
|