diff --git a/docs/img/media-sbs-proof.png b/docs/img/media-sbs-proof.png new file mode 100644 index 0000000..d04ad1c Binary files /dev/null and b/docs/img/media-sbs-proof.png differ diff --git a/docs/img/media-splat-proof.png b/docs/img/media-splat-proof.png new file mode 100644 index 0000000..e74a0cf Binary files /dev/null and b/docs/img/media-splat-proof.png differ diff --git a/docs/open-questions.md b/docs/open-questions.md index 13b4c45..37ecaee 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -102,10 +102,11 @@ Still open: 4, 6, 7, 12–15, 16 (off-LAN and after a reboot), 17–21. 20. **F-Droid 2.0** (Compose 1.12): does it run? If so, the catalogue can install it instead of 1.17.2. -21. **DeoVR local files:** does DeoVR's file browser show `Videos → VR` - (the symlink from `push-vr-video.sh`) or `Z:\home\steamos\Videos\VR`, and do - the colour-coded test clips play in 3D (red left eye, cyan right) for both - H.264 and H.265? Does the DLNA browser find a server on the Mac? +21. **Owned media player:** worn-headset comfort, audio quality/lip sync, long + movies and 4K/8K decoding remain to check. Native spatial-photo container + extraction, large/immersive splats and existing-panel theatre docking need + further implementation. Remote eye isolation, short hardware decode and + owned-overlay cleanup are verified; see [vr-video.md](vr-video.md). ## Verified 2026-09-27 diff --git a/docs/panels.md b/docs/panels.md index b08720b..a4e3026 100644 --- a/docs/panels.md +++ b/docs/panels.md @@ -123,3 +123,12 @@ a limit on the number of floating panels. keyboard) or virtual desktops arrange windows within the 1280×800 rectangle. - **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a PC's desktop in SteamVR. They don't run on the Frame's standalone Linux. + +## Frame Control's media theatre + +[The owned media player](vr-video.md) can show its video or stereo image on a +larger, head-relative screen with its own dark surround. **Verified remotely +2026-09-28**, SteamOS 0.4.1 / BUILD_ID 20260925.6191901: screen, eye isolation, +surround and cleanup. It does not alter panel docking or global settings. +Its `Overlay` RGBA rendering hook is available to stream producers; applying +SteamVR theatre docking to existing Mac/PC panels is still unverified here. diff --git a/docs/scripts.md b/docs/scripts.md index ad11ebd..d9333ed 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -99,7 +99,7 @@ controls to place each panel. See [docs/panels.md](panels.md). | `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) | | `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) | | `scripts/compat-db-backup.sh` | Mac | Maintainer-only: back up the shared compatibility database locally and to Google Drive (**verified**) | -| `scripts/push-vr-video.sh` | Mac → Frame | Upload VR180/360 videos to `~/Videos/VR`, linked into DeoVR's Proton prefix; `--launch` starts DeoVR (**verified**: upload and link; in-headset playback of local files not yet checked). See [docs/vr-video.md](vr-video.md) | +| `scripts/push-vr-video.sh` | Computer → Frame | Upload movies, stereo PNG/JPEG or small `.splat` files to `~/Videos/FrameControl`; `--launch` starts our OpenVR player, `--theatre` adds a larger screen and dark surround. No separate viewer. **Verified remotely**: decode, stereo output and cleanup; worn-headset checks remain. See [docs/vr-video.md](vr-video.md) | | `scripts/push.sh` | Mac → Frame | `rsync` files to `~/Downloads` (or a given path) on the Frame (**verified**) | | `scripts/serve-bootstrap.sh` | Mac | Fallback: serve `bootstrap-on-frame.sh` with your public key embedded | | `scripts/bootstrap-on-frame.sh` | Frame | Fallback: install the key and enable `sshd` | diff --git a/docs/streaming.md b/docs/streaming.md index 707544e..93fc63f 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -40,8 +40,10 @@ flat 2D desktop streaming into a window on the Frame's Linux desktop. | Immersed / Virtual Desktop | Vendor apps | Immersed has a Mac agent but no known Frame client. Virtual Desktop's developer said he'd "try" to port it ([NewsBreak](https://www.newsbreak.com/news/4892834783961-virtual-desktop-dev-says-he-ll-try-to-bring-the-app-to-steam-frame)). | Not available as of 2026-09-25. Check again later. | | WiVRn / ALVR | VR streaming from a Linux or Windows PC | Irrelevant for a Mac host (no SteamVR/OpenXR runtime on macOS) | N/A | -For **VR video files** (180°/360° stereo), don't stream the Mac's screen. Play -them on the Frame in DeoVR instead: see [vr-video.md](vr-video.md). +For **local movies and stereo photos**, Frame Control's own OpenVR player +runs on the Frame; see [vr-video.md](vr-video.md). It currently renders a flat +stereo screen. VR180/360 projection is not implemented; the same page records +DeoVR only as an optional, independently installed alternative. ### Pre-seeding the Remmina profile (no typing in the headset) diff --git a/docs/testing.md b/docs/testing.md index b06577d..d8f7020 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -148,3 +148,12 @@ For example, on 2026-09-27 the smoke test found that Steam's `create-shortcut` refuses ids with a hyphen (`missing/invalid arguments`), which the fake had accepted. The fake now refuses them the same way, and Frame Control makes ids Steam accepts. + +## Owned media player + +`tests/test_media.py` covers layout evidence and overrides, OU eye ordering, +hardware-decoder command construction, malformed splats, stereo parallax and +fake-Frame library/process ownership. `tests/e2e/test_media.py` checks the real +HTTP/SSH upload and library listing without pretending the fake renders VR. +Real decode timings and captured stereo output are recorded in +[vr-video.md](vr-video.md). Generated media only; no external player required. diff --git a/docs/vr-video.md b/docs/vr-video.md index 19c4085..8da4563 100644 --- a/docs/vr-video.md +++ b/docs/vr-video.md @@ -1,81 +1,147 @@ -# Watching VR video (180°/360°) on the Frame +# Movies, stereo photos and splats -The confidence labels are the same as in [ssh.md](ssh.md). Everything here -was checked on SteamOS 0.3.0, build 20260922.6101926. +Frame Control has its own Frame-side OpenVR player. It uses SteamOS's ffmpeg +and V4L2 hardware decoder, Python and SteamVR. **No separate player or viewer +is required.** Chromium and immersive WebXR are not in this playback path. -## The short version +## Use it -1. Install **DeoVR Video Player** from Steam (free, app 837380) and start it once - in the headset. That creates its Proton prefix. -2. On the Mac: `scripts/push-vr-video.sh --launch ~/Movies/beach_180_LR.mp4` -3. In the headset, open DeoVR's local file browser → **Videos → VR** and - pick the file. +In **Tools → Media in the headset**, send a file, choose its layout and press +**Play**. **Theatre** gives it a larger screen and an 85% black surround. +**Stop** removes both. Refresh reads the library and the player's state. +The screen follows your head; it isn't a saved world-space panel. -## Why DeoVR - -The Frame's Chromium can't play VR video in 3D. It has no immersive WebXR -([how-the-frame-works.md](how-the-frame-works.md)). DeoVR's Windows build -runs under Proton ARM64 + FEX as a real SteamVR app. **Verified 2026-09-25:** -it found the Steam Frame headset and controller over OpenVR, and decoded -7680×3840 and 8192×4096 H.265 VR180 side-by-side streams through AVPro's -hardware Media Foundation path, mapped onto a 180° dome or fisheye mesh. - -Known quirks (verified): - -- The first launch takes about 45 s while it compiles shaders. -- Grid thumbnails stay blank. Unity's own video player, which DeoVR uses for - previews, fails with `0xc00d36bb` under Proton. Full playback uses AVPro - and isn't affected. -- The in-app store and web content are separate from local files. You don't - need an account to play your own files. - -## Getting files onto the headset - -`scripts/push-vr-video.sh` copies files with `rsync --partial`, so an -interrupted upload resumes. They go to `~/Videos/VR` on the Frame (`/home` -has about 860 GB free). The script also links that folder into DeoVR's prefix -as `C:\users\steamuser\Videos\VR`. It's reachable at -`Z:\home\steamos\Videos\VR` as well. **Verified** that the upload and link -work. **Not yet checked** whether DeoVR's file browser lands there -(open question 21). - -Speed: a test upload over Wi-Fi ran at about 3–5 MB/s (verified 2026-09-25, -one sample). At that rate an 8K file of several GB takes tens of minutes, so -start big uploads before you put the headset on. - -## Naming files so they play correctly - -DeoVR guesses the projection from the file name. Its binary contains the tags -`_180`, `_360`, `_fisheye`, `_fisheye190`, `_mkx200`, `_vrca220` and `_rf52` -(verified). For stereo layout, the common DeoVR convention is `_LR`/`_SBS` -(side by side) and `_TB` (top/bottom) (inferred). If a video looks wrong -(doubled, warped, or flat), change the projection and stereo mode in DeoVR's -player menu. - -Examples: `trip_180_LR.mp4`, `concert_360_TB.mp4`, `hike_fisheye190_LR.mp4`. - -Codecs: H.265 at 8K played (verified, streamed). Local H.264 and H.265 -files haven't been played yet. The test clips below cover that. - -## Test clips - -The script doesn't include these. To check a setup, make two 20 s clips: -3840×1920, 180° side by side, with the left eye tinted red and the right eye -cyan. In the headset each eye should see only its own colour. A single mixed -colour means the stereo split is wrong. +The command-line route uses the same player: ```sh -ffmpeg -f lavfi -i testsrc2=size=1920x1920:rate=30:duration=20 \ - -filter_complex "[0:v]split[a][b];[a]colorchannelmixer=rr=1:gg=0.3:bb=0.3[l];[b]colorchannelmixer=rr=0.3:gg=1:bb=1[r];[l][r]hstack" \ - -c:v libx264 -pix_fmt yuv420p -b:v 20M frame-test_180_LR_h264.mp4 -scripts/push-vr-video.sh frame-test_180_LR_h264.mp4 +scripts/push-vr-video.sh --launch --theatre ~/Movies/film_SBS.mp4 +scripts/push-vr-video.sh --launch --layout ou ~/Pictures/stereo.png +scripts/push-vr-video.sh --launch capture.splat +scripts/push-vr-video.sh --list +scripts/push-vr-video.sh --stop ``` -## Streaming from the Mac instead of copying (untested) +Uploads live in `~/Videos/FrameControl//` on the Frame. Each upload gets +its own directory, so sending another file with the same name doesn't replace +it. The UI's existing upload limit is 8 GiB. Transfers use the app's rsync/scp +path; resumable large uploads are not implemented here yet. Existing +`~/Videos/VR` files and Proton prefixes are left alone. The script no longer +launches DeoVR, links a Proton prefix, or accepts directories. -DeoVR has a DLNA browser (the binary contains `Searching for DLNA -devices...` and a UPnP ContentDirectory client). A DLNA server on the Mac -should therefore appear in DeoVR without copying anything, for example -`brew install rclone` then `rclone serve dlna ~/Movies/VR`. This is **inferred**, not -tried. 8K VR video needs roughly 50–100 Mbit/s sustained, and the Wi-Fi -sample above (about 30–40 Mbit/s) suggests copying first is the safer default. +| Media | Supported preview | +|---|---| +| Movies | H.264 / H.265, mono or left-first SBS / top-first OU; audio through the Frame's PulseAudio-compatible server | +| Stereo photos | PNG / JPEG containing both eyes, SBS or OU | +| Gaussian splats | Common 32-byte `.splat` records, 1–20,000 Gaussians; a stationary stereo preview | + +Auto layout reads delimited filename tags: `_SBS`, `_HSBS`, `_LR`, `_OU`, +`_TB`, `_HOU`, `_HTB`, `_FSBS`, `_FOU`, `_FTB`. SBS/OU without `F` means +half-resolution packing. Full packing preserves each eye's original aspect. +It also accepts FFmpeg's `stereo_mode` metadata (`left_right`, `top_bottom`, +`mono`); left/right and top/bottom metadata are treated as full packing. +Choose an explicit layout if that assumption doesn't match the file. +Unknown or conflicting tags ask for a choice, rather than silently flattening +stereo. The explicit selector always wins. Layout is not guessed from resolution. + +## What was verified + +**Verified remotely, 2026-09-28:** SteamOS 0.4.1, BUILD_ID +`20260925.6191901`, SteamVR 2.18.1. Nobody wore the headset for these checks. +All test media were generated by us. No paid content or DRM was involved. + +- Stock `ffmpeg` `h264_v4l2m2m` decoded 150 frames of our 1920×1080 H.264 + test in 0.128 s (decode only); converting all frames to RGBA took 0.242 s. +- Our ffmpeg → Python → OpenVR path submitted all 150 frames and exited 0. + The 1280×720 prototype took 4.775 s; the full 1920×1080 player took + 4.618 s. These are wall times, not in-headset frame-rate measurements; + These initial probes exposed an early finish. The final player explicitly + paces 30 fps output: all 150 frames then completed in **5.025 s**. +- The H.265 hardware decoder also completed the generated 1080p clip + (60 frames, exit 0); this is a short compatibility check, not a 4K/8K benchmark. +- Our actual player, launched through Frame Control's media API, displayed + SBS and OU photos with red only in the left-eye capture and cyan only in + the right. OpenVR's `SideBySide_Parallel` flag separates the eyes; we + rearrange OU rows ourselves. +- A generated 48-Gaussian `.splat` rendered in our own CPU renderer and appeared + in both eyes with separate perspective projections. +- Theatre's owned dark surround and screen appeared in captures. Steam's + dashboard remained available over them. Stop removed the owned overlays + and ended the dedicated user service. No global SteamVR setting was changed. +- A generated H.264/AAC clip reached the Pulse audio output and completed + all 100 video frames with exit 0. This proves the output path, not audible + quality or lip sync. + +![Frame Control SBS eye-isolation proof: red left, cyan right](img/media-sbs-proof.png) + +![Our Gaussian-splat stereo preview on the Frame](img/media-splat-proof.png) + +**Verified failed route:** GStreamer 1.24.2's `playbin` selected +`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139) +in the basic appsink probe and the OpenVR probe. We do not ship that route. +The ffmpeg path above completed instead. + +## Limits and blockers + +- **Not verified:** worn-headset comfort, sound quality/lip sync, long movies, + 4K/8K decode, HDR, controller interaction or behaviour on other OS builds. + Output is bounded to 1920×1080 packed pixels before OU rearrangement. +- **Not implemented:** pause, seeking, subtitles, playlists, right-eye-first + layouts, non-square pixel correction, VR180/360 projection and fisheye. + This preview is a flat stereo screen, not a dome player. +- **Native spatial-photo blocker:** HEIC/HEIF/AVIF/MPO stereo-container + extraction isn't implemented in our viewer. We reject these rather than + displaying one image and calling it spatial. Export both eyes to PNG/JPEG + first. This is a current implementation gap, not a claim that the Frame + cannot support these containers. +- **Large/immersive splat blocker:** the owned CPU rasterizer projects 3D + covariance into each eye and alpha-composites Gaussians, but renders a + fixed view at 320×240 per eye. It caps each Gaussian footprint at 32 pixels + and normalizes the scene to a two-metre box. Large scenes, PLY/SPZ, live + head-position parallax and navigation need a GPU scene renderer; this + version rejects files above 20,000 records. It is a stereo preview, not + an immersive walk-through. +- A single player can run at a time. It stops at EOF, on **Stop**, or after + four hours if the computer disconnects. Still images remain until Stop + or that timeout. The log is + `~/.local/share/frame-control/media/player.log` on the Frame. +- **Panel/stream theatre remains outside this media slice:** SteamVR's + `vrcmd --dock-overlay` accepts `theater`, but docking an existing panel + and its dimming behaviour were not verified here. The shared headset had + workspace and stream tests active. We did not reposition their panels. + Integration with [#22](https://github.com/saphid/frame-control/issues/22) + and [#31](https://github.com/saphid/frame-control/issues/31) can use the + owned rendering hook below; no second stream/window manager is introduced. + +## Integration hook + +`POST /api/upload` with `X-Mode: media` and `X-Filename` sends a file and +returns its `id`. `POST /api/media` accepts: + +```json +{"action":"play","id":"<32 hex characters>/film_SBS.mp4","layout":"auto","theatre":true} +``` + +Other actions are `list`, `status`, `stop`. These use the normal `X-Frame-UI` +guard. Play reports **starting**, not a claim that frames reached the headset; +read status for `playing`, `ended`, `stopped` or `error`. +The own-code files are copied to `~/.local/share/frame-control/media/` and +run in the `frame-control-media.service` systemd user unit. Stop affects only +that unit, including its decoder child. + +For a Frame-side stream producer, `frame_media_player.Overlay` exposes +`create(key, width, distance, stereo=False, aspect=1, order=1)`, +`pixels(handle, rgba, width, height)` and `close()`. Use an owned unique key, +pass interleaved RGBA bytes (left/right halves for stereo) and always close in +`finally`. `create` uses a head-relative transform; it does not move another +app's panel. The player demonstrates a separate black surround overlay. +`frame_media.stereo_pixels` converts OU to SBS. This is the narrow rendering +hook for stream/workspace work; it doesn't capture or manage a Mac/PC stream. + +## Optional separate app + +You can independently install **DeoVR Video Player** (free Steam app 837380) +if you prefer its VR180/360 features. Earlier tests on SteamOS 0.3.0, +build `20260922.6101926` (2026-09-25), verified its OpenVR initialization and +8K H.265 streamed VR180 decoding under Proton. Frame Control's media features +neither install nor launch it, and do not depend on it. Its behaviour and +file-naming conventions are not evidence about our player. diff --git a/scripts/push-vr-video.sh b/scripts/push-vr-video.sh index 93cb5ec..2833fd9 100755 --- a/scripts/push-vr-video.sh +++ b/scripts/push-vr-video.sh @@ -1,65 +1,8 @@ #!/usr/bin/env zsh -# Mac-side: upload VR videos to the Steam Frame so DeoVR can play them in 3D. -# -# Files go to ~/Videos/VR on the Frame. The script links that folder into -# DeoVR's Proton prefix as C:\users\steamuser\Videos\VR, so DeoVR's file -# browser finds it under Videos. (It's also reachable as Z:\home\steamos\Videos\VR.) -# Uploads resume if interrupted. -# -# Name files so DeoVR picks the projection: include _180 or _360 (or _fisheye190, -# _mkx200, _rf52 ...) plus the stereo layout (_LR / _SBS side by side, _TB over- -# under), e.g. "beach_180_LR.mp4". You can also change it in DeoVR's player. -# -# Usage: -# scripts/push-vr-video.sh FILE_OR_DIR... # upload -# scripts/push-vr-video.sh --launch FILE... # upload, then start DeoVR -# scripts/push-vr-video.sh --launch # just start DeoVR -# scripts/push-vr-video.sh --list # what's on the Frame +# Upload media for Frame Control's own OpenVR player (no external player). +# Usage: scripts/push-vr-video.sh [--launch] [--layout auto|mono|sbs|ou|full-sbs|full-ou] [--theatre] FILE... +# scripts/push-vr-video.sh --list | --stop +# Files go to ~/Videos/FrameControl//; --launch accepts exactly one file. +# Directories, automatic DeoVR launching and Proton-prefix links are no longer used. set -euo pipefail - -FRAME_ALIAS=${FRAME_ALIAS:-frame} -DEOVR_APPID=837380 -REMOTE_DIR="Videos/VR" -PREFIX_VIDEOS=".local/share/Steam/steamapps/compatdata/$DEOVR_APPID/pfx/drive_c/users/steamuser/Videos" - -launch=0 list=0 -while (( $# )); do - case "$1" in - -h|--help) sed -n '2,17p' "$0"; exit 0 ;; - --launch) launch=1; shift ;; - --list) list=1; shift ;; - --) shift; break ;; - -*) print -u2 "push-vr-video: unknown option $1"; exit 2 ;; - *) break ;; - esac -done -(( $# || launch || list )) || { sed -n '2,17p' "$0" >&2; exit 2; } - -for f in "$@"; do - [[ -e "$f" ]] || { print -u2 "push-vr-video: no such file: $f"; exit 2; } -done - -# Create the folder and link it into DeoVR's prefix (the prefix exists once -# DeoVR has run). Refresh a stale link, but never replace a real directory. -if (( $# || launch )); then - ssh "$FRAME_ALIAS" "mkdir -p ~/$REMOTE_DIR -p=~/$PREFIX_VIDEOS -if [ -d \"\$p\" ] && { [ -L \"\$p/VR\" ] || [ ! -e \"\$p/VR\" ]; }; then ln -sfn ~/$REMOTE_DIR \"\$p/VR\" -elif [ -d \"\$p/VR\" ]; then echo \"warning: \$p/VR is a real folder, so uploads won't show under DeoVR's Videos; browse Z:\\\\home\\\\steamos\\\\Videos\\\\VR instead\" >&2; fi -[ -d \"\$p\" ] || echo 'note: DeoVR has not run yet; use Z:\\home\\steamos\\Videos\\VR or run this again after starting it once' >&2" -fi - -if (( $# )); then - # -L: send what a symlink points at; a Mac-side link would dangle on the Frame - rsync -aL --partial --progress -- "$@" "$FRAME_ALIAS:$REMOTE_DIR/" -fi - -if (( list )); then - ssh "$FRAME_ALIAS" "cd ~/$REMOTE_DIR && ls -lhR" -fi - -if (( launch )); then - ssh "$FRAME_ALIAS" "command -v steam >/dev/null || { echo 'steam not found on the Frame' >&2; exit 1; } -steam steam://rungameid/$DEOVR_APPID /dev/null 2>&1 &" - print "DeoVR starting on the Frame. Open Local files / the file browser → Videos → VR." -fi +exec python3 "${0:A:h}/../ui/frame_media_cli.py" "$@" diff --git a/tests/e2e/test_media.py b/tests/e2e/test_media.py new file mode 100644 index 0000000..6c6c015 --- /dev/null +++ b/tests/e2e/test_media.py @@ -0,0 +1,20 @@ +"""Media transfer through the real HTTP/SSH path; the fake has no VR renderer.""" +import harness +from harness import ok, ssh + +harness.require() + + +class Media(harness.FrameTestCase): + def test_upload_and_list_without_launching_a_viewer(self): + # The transfer endpoint doesn't decode. Rendering belongs to the real + # headset smoke checks documented in docs/vr-video.md. + result = ok('POST', '/api/upload', raw=b'fake-media', headers={ + 'X-Mode': 'media', 'X-Filename': 'test_SBS.png'}) + identity = result['id'] + try: + files = ok('POST', '/api/media', {'action': 'list'})['files'] + self.assertIn(identity, [f['id'] for f in files]) + self.assertEqual(ssh('cat ~/Videos/FrameControl/'+identity), 'fake-media') + finally: + ssh('rm -rf ~/Videos/FrameControl/'+identity.split('/')[0]) diff --git a/tests/test_media.py b/tests/test_media.py new file mode 100644 index 0000000..4d441f1 --- /dev/null +++ b/tests/test_media.py @@ -0,0 +1,108 @@ +"""Owned media planning, eye isolation, decoder choice and fake-Frame ownership.""" +import json +from pathlib import Path +import struct +import sys +import tempfile +import unittest +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui')) +import frame_media as media +import frame_media_player as player +import frame_media_remote as remote +import frame_splat as splat +import server + + +class Media(unittest.TestCase): + def test_layout_evidence_and_override(self): + for name, layout in [('film_SBS.mp4', 'sbs'), ('film.OU.mkv', 'ou'), + ('film_FSBS.mp4', 'full-sbs'), ('photo_TB.png', 'ou')]: + self.assertEqual(media.plan(name)['layout'], layout) + self.assertEqual(media.plan('film_SBS_OU.mp4', 'mono')['source'], 'explicit') + self.assertEqual(media.plan('film.mkv', metadata={'stereo_mode': 'top_bottom'})['layout'], 'full-ou') + for name in ('film.mp4', 'film_SBS_OU.mp4', 'businessbs.mp4'): + with self.assertRaises(ValueError): + media.plan(name) + with self.assertRaises(ValueError): + media.plan('film.mkv', metadata={'stereo_mode': 'right_left'}) + + def test_unsupported_containers_do_not_flatten_spatial_photos(self): + for name in ('spatial.HEIC', 'stereo.mpo', 'cloud.ply', 'cloud.spz', 'app.exe'): + with self.assertRaises(ValueError): + media.plan(name, 'sbs') + + def test_ou_pixels_keep_each_eye_and_row(self): + a, b, c, d = [bytes([n])*8 for n in (1, 2, 3, 4)] + data, width, height = media.stereo_pixels(a+b+c+d, 2, 4, 'ou') + self.assertEqual((data, width, height), (a+c+b+d, 4, 2)) + with self.assertRaises(ValueError): + media.stereo_pixels(b'bad', 2, 4, 'sbs') + self.assertEqual(media.geometry(3840, 2160, 'sbs'), (1920, 1080, 2)) + self.assertEqual(media.geometry(1920, 1080, 'ou'), (1920, 1080, .5)) + + def test_decode_is_hardware_and_one_clock_for_audio(self): + for codec, decoder in [('h264', 'h264_v4l2m2m'), ('hevc', 'hevc_v4l2m2m')]: + cmd = player.decoder_command(Path('/tmp/a file.mp4'), {'codec_name': codec}, 1280, 720, True) + self.assertIn(decoder, cmd) + self.assertIn('-re', cmd) + self.assertIn('pulse', cmd) + self.assertIn('/tmp/a file.mp4', cmd) + with self.assertRaises(ValueError): + player.decoder_command(Path('x.webm'), {'codec_name': 'vp9'}, 640, 480, False) + cmd = player.decoder_command(Path('x.png'), {'codec_name': 'png'}, 640, 480, False, True) + self.assertNotIn('-re', cmd) + self.assertIn('-frames:v', cmd) + + def test_fake_frame_library_and_traversal(self): + with tempfile.TemporaryDirectory() as d, patch.object(remote, 'ROOT', Path(d)): + identity = 'a'*32+'/space and quote\'.png' + path = Path(d)/identity + path.parent.mkdir() + path.write_bytes(b'test') + self.assertEqual(remote.media_path(identity), path.resolve()) + for bad in ('../../etc/passwd', '/etc/passwd', 'a'*32+'/..', 'a'*32+'/x/y', None): + with self.assertRaises((ValueError, FileNotFoundError)): + remote.media_path(bad) + link = path.parent/'link.png' + link.symlink_to(path) + with self.assertRaises(ValueError): + remote.media_path('a'*32+'/link.png') + with patch.object(remote, 'status', return_value={'state': 'idle'}): + files = remote.run({'action': 'list'})['files'] + self.assertEqual([f['id'] for f in files], [identity]) + + def test_server_rejects_bad_actions_before_ssh(self): + with patch.object(server, 'ssh') as ssh: + for body in ({'action': 'delete'}, {'action': 'play', 'id': '../x'}, + {'action': 'play', 'id': 'a'*32+'/x', 'layout': 'invalid'}, + {'action': 'play', 'id': 'a'*32+'/x', 'theatre': 'false'}): + with self.assertRaises(server.Failure): + server.media(body) + ssh.assert_not_called() + + def test_fake_frame_stop_only_owns_our_unit(self): + with patch.object(remote.subprocess, 'run') as run, patch.object(remote, 'status', return_value={}): + remote.run({'action': 'stop'}) + self.assertEqual(run.call_args.args[0], ['systemctl', '--user', 'stop', 'frame-control-media.service']) + + def test_splat_invalid_records_and_stereo_parallax(self): + with tempfile.TemporaryDirectory() as d: + path = Path(d)/'small.splat' + path.write_bytes(struct.pack('<6f8B', 0, 0, 0, .001, .001, .001, + 255, 0, 0, 255, 255, 128, 128, 128)) + data, w, h = splat.render(path, 64, 48) + self.assertEqual((len(data), w, h), (128*48*4, 128, 48)) + def centroid(eye): + weights = [(x, data[(y*w+x+eye*64)*4]) for y in range(h) for x in range(64)] + return sum(x*v for x,v in weights)/sum(v for x,v in weights) + self.assertGreater(centroid(0), centroid(1)) + for bad in (b'', b'bad', struct.pack('<6f8B', float('nan'), 0, 0, 1, 1, 1, *([128]*8))): + path.write_bytes(bad) + with self.assertRaises(ValueError): + splat.read(path) + + +if __name__ == '__main__': + unittest.main() diff --git a/ui/frame_media.py b/ui/frame_media.py new file mode 100644 index 0000000..857ee91 --- /dev/null +++ b/ui/frame_media.py @@ -0,0 +1,67 @@ +"""Media planning shared by Frame Control and its own Frame-side player. + +No viewer dependencies. Filename hints are suggestions, never guesses from +resolution. Explicit layout wins; conflicting hints require a choice. +""" +import re +from pathlib import Path + +LAYOUTS = ('auto', 'mono', 'sbs', 'ou', 'full-sbs', 'full-ou') +VIDEO = {'.mp4', '.mkv', '.mov', '.webm', '.m4v'} +PHOTO = {'.png', '.jpg', '.jpeg'} + + +def plan(name, layout='auto', metadata=None): + if layout not in LAYOUTS: + raise ValueError('Choose auto, mono, sbs, ou, full-sbs or full-ou') + suffix = Path(name).suffix.lower() + if suffix in {'.heic', '.heif', '.avif', '.mpo'}: + raise ValueError('Native spatial-photo containers are not supported yet; export both eyes as SBS or OU PNG/JPEG') + if suffix == '.splat': + return {'kind': 'splat', 'layout': 'sbs', 'source': 'renderer'} + if suffix not in VIDEO | PHOTO: + raise ValueError('Use MP4/MKV/MOV/WebM video, PNG/JPEG stereo photos, or a .splat file') + source = 'explicit' + if layout == 'auto': + tokens = set(re.split(r'[^a-z0-9]+', Path(name).stem.lower())) + hints = set() + for value, tags in [('full-sbs', {'fsbs'}), ('full-ou', {'fou', 'ftb'}), + ('sbs', {'sbs', 'hsbs', 'lr'}), ('ou', {'ou', 'hou', 'tb', 'htb'})]: + if tokens & tags: + hints.add(value) + if len(hints) > 1: + raise ValueError('Conflicting stereo filename tags; choose the layout explicitly') + layout = next(iter(hints), None) + source = 'filename' + if not layout: + # Matroska StereoMode/FFmpeg stereo_mode: only known left-first modes. + mode = (metadata or {}).get('stereo_mode') + layout = {'left_right': 'full-sbs', 'top_bottom': 'full-ou', 'mono': 'mono'}.get(mode) + source = 'metadata' + if mode and layout is None: + raise ValueError('Unsupported stereo metadata; choose the eye order/layout explicitly') + if not layout: + raise ValueError('No stereo layout found; choose mono, SBS or OU (left/top eye first)') + return {'kind': 'video' if suffix in VIDEO else 'photo', 'layout': layout, 'source': source} + + +def geometry(width, height, layout): + """Bound transfer to 1920x1080; return packed dimensions and texel aspect.""" + if not 0 < width <= 32768 or not 0 < height <= 32768: + raise ValueError('Invalid media dimensions') + if layout not in LAYOUTS[1:]: + raise ValueError('Resolve the layout before playback') + scale = min(1, 1920 / width, 1080 / height) + w, h = max(2, int(width * scale) // 2 * 2), max(2, int(height * scale) // 2 * 2) + return w, h, {'mono': 1, 'sbs': 2, 'ou': .5, 'full-sbs': 1, 'full-ou': 1}[layout] + + +def stereo_pixels(data, width, height, layout): + """Normalize top/bottom to OpenVR's left/right texture; preserve eye order.""" + if len(data) != width * height * 4: + raise ValueError('Incomplete RGBA frame') + if layout not in ('ou', 'full-ou'): + return data, width, height + stride, half = width * 4, height // 2 + return b''.join(data[y*stride:(y+1)*stride] + + data[(y+half)*stride:(y+half+1)*stride] for y in range(half)), width*2, half diff --git a/ui/frame_media_cli.py b/ui/frame_media_cli.py new file mode 100644 index 0000000..0d1abb1 --- /dev/null +++ b/ui/frame_media_cli.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Send local media to Frame Control's own OpenVR player.""" +import argparse +import json +from pathlib import Path +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import frame_media +import server + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument('files', nargs='*', type=Path) + ap.add_argument('--launch', action='store_true', help='play the one file being sent') + ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto') + ap.add_argument('--theatre', action='store_true', help='bigger screen and dark surround') + ap.add_argument('--list', action='store_true') + ap.add_argument('--stop', action='store_true') + args = ap.parse_args() + if args.launch and len(args.files) != 1: + ap.error('--launch needs exactly one file') + if not args.files and not (args.list or args.stop): + ap.error('choose files, --list or --stop') + for path in args.files: + if not path.is_file(): + ap.error('not a file: %s' % path) + frame_media.plan(path.name, 'mono') + if args.stop: + print(json.dumps(server.media({'action': 'stop'}))) + for path in args.files: + result = server.push_media(path.resolve()) + print(json.dumps(result)) + if args.launch: + print(json.dumps(server.media({'action': 'play', 'id': result['id'], + 'layout': args.layout, 'theatre': args.theatre}))) + if args.list: + print(json.dumps(server.media({'action': 'list'}))) + + +if __name__ == '__main__': + try: + main() + except (ValueError, server.Failure) as e: + sys.exit(str(e)) diff --git a/ui/frame_media_player.py b/ui/frame_media_player.py new file mode 100644 index 0000000..a6d2bdc --- /dev/null +++ b/ui/frame_media_player.py @@ -0,0 +1,213 @@ +#!/usr/bin/env python3 +"""Frame Control's local-media OpenVR player. Runs on the Frame, no third-party app. + +SteamOS ffmpeg does hardware video decoding, scaling and audio output. OpenVR +owns only our screen and optional black surround. Exiting destroys both. +""" +import argparse +import ctypes as C +import json +import os +from pathlib import Path +import signal +import subprocess +import time + +import frame_media +import frame_splat + +LIB = '/opt/steamvr/bin/linuxarm64/libopenvr_api.so' +H = C.c_uint64 +# Slots from Valve's openvr_capi.h, IVROverlay_028. Fail closed on another ABI. +SLOTS = { + 'CreateOverlay': (1, [C.c_char_p, C.c_char_p, C.POINTER(H)]), + 'DestroyOverlay': (3, [H]), + 'SetOverlayFlag': (11, [H, C.c_int, C.c_bool]), + 'SetOverlayAlpha': (16, [H, C.c_float]), + 'SetOverlayTexelAspect': (18, [H, C.c_float]), + 'SetOverlaySortOrder': (20, [H, C.c_uint32]), + 'SetOverlayWidthInMeters': (22, [H, C.c_float]), + 'SetOverlayTransformTrackedDeviceRelative': (35, [H, C.c_uint32, C.c_void_p]), + 'ShowOverlay': (43, [H]), + 'SetOverlayRaw': (62, [H, C.c_void_p, C.c_uint32, C.c_uint32, C.c_uint32]), +} + + +class Overlay: + def __init__(self): + self.handles = [] + self.vr = C.CDLL(LIB) + self.vr.VR_InitInternal2.argtypes = [C.POINTER(C.c_int), C.c_int, C.c_char_p] + self.vr.VR_GetGenericInterface.argtypes = [C.c_char_p, C.POINTER(C.c_int)] + self.vr.VR_GetGenericInterface.restype = C.c_void_p + err = C.c_int() + self.vr.VR_InitInternal2(C.byref(err), 2, None) + if err.value: + raise RuntimeError('SteamVR init failed: %s' % err.value) + ptr = self.vr.VR_GetGenericInterface(b'FnTable:IVROverlay_028', C.byref(err)) + if not ptr or err.value: + self.vr.VR_ShutdownInternal() + raise RuntimeError('SteamVR needs IVROverlay_028: %s' % err.value) + self.table = C.cast(ptr, C.POINTER(C.c_void_p)) + + def call(self, name, *values): + slot, args = SLOTS[name] + rc = C.CFUNCTYPE(C.c_int, *args)(self.table[slot])(*values) + if rc: + raise RuntimeError('OpenVR %s failed: %s' % (name, rc)) + + def create(self, key, width, distance, stereo=False, aspect=1, order=1): + handle = H() + self.call('CreateOverlay', key.encode(), b'Frame Control media', C.byref(handle)) + self.handles.append(handle) + self.call('SetOverlayWidthInMeters', handle, width) + self.call('SetOverlaySortOrder', handle, order) + self.call('SetOverlayTexelAspect', handle, aspect) + if stereo: + self.call('SetOverlayFlag', handle, 1024, True) # SideBySide_Parallel + matrix = (C.c_float * 12)(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, -distance) + self.call('SetOverlayTransformTrackedDeviceRelative', handle, 0, matrix) + return handle + + def pixels(self, handle, data, width, height): + buf = C.create_string_buffer(data) + self.call('SetOverlayRaw', handle, buf, width, height, 4) + self.call('ShowOverlay', handle) + + def close(self): + try: + for h in reversed(self.handles): + self.call('DestroyOverlay', h) + finally: + self.vr.VR_ShutdownInternal() + + +def probe(path): + result = subprocess.run(['ffprobe', '-v', 'error', '-show_streams', '-of', 'json', str(path)], + capture_output=True, text=True, timeout=30) + if result.returncode: + raise ValueError(result.stderr[-2000:] or 'Cannot read media') + streams = json.loads(result.stdout)['streams'] + video = next((s for s in streams if s['codec_type'] == 'video'), None) + if not video: + raise ValueError('No image or video stream') + return video, any(s['codec_type'] == 'audio' for s in streams) + + +def decoder_command(path, info, width, height, audio, photo=False): + cmd = ['ffmpeg', '-nostdin', '-hide_banner', '-loglevel', 'error'] + if not photo: + cmd += ['-re', '-readrate_initial_burst', '0'] + codec = {'h264': 'h264_v4l2m2m', 'hevc': 'hevc_v4l2m2m'}.get(info['codec_name']) + if not codec: + raise ValueError('Hardware playback currently supports H.264 and H.265 only') + cmd += ['-c:v', codec] + cmd += ['-i', str(path), '-map', '0:v:0', '-vf', 'scale=%s:%s' % (width, height), + '-pix_fmt', 'rgba'] + if photo: + cmd += ['-frames:v', '1'] + else: + cmd += ['-r', '30'] + cmd += ['-f', 'rawvideo', 'pipe:1'] + if audio and not photo: + cmd += ['-map', '0:a:0', '-f', 'pulse', 'Frame Control Media'] + return cmd + + +def write_status(path, **values): + tmp = path.with_suffix('.tmp') + tmp.write_text(json.dumps(values)) + tmp.replace(path) + + +def play(args): + path = Path(args.file).resolve(strict=True) + status = Path(args.status) + splat = path.suffix.lower() == '.splat' + if splat: + data, width, height = frame_splat.render(path) + plan = frame_media.plan(path.name) + aspect, photo, command = 1, True, None + else: + info, audio = probe(path) + plan = frame_media.plan(path.name, args.layout, info.get('tags')) + width, height, aspect = frame_media.geometry(info['width'], info['height'], plan['layout']) + photo = plan['kind'] == 'photo' + command = decoder_command(path, info, width, height, audio, photo) + vr, proc, frames, started = None, None, 0, time.monotonic() + # systemd sends SIGTERM to the whole unit, including ffmpeg. Python unwinds + # ownership; no unrelated Steam/SteamVR process or setting is touched. + def stop(signum, frame): + raise InterruptedError('Stopped') + signal.signal(signal.SIGTERM, stop) + signal.signal(signal.SIGINT, stop) + try: + vr = Overlay() + if args.theatre: + surround = vr.create('framecontrol.media.surround', 40, 4, order=0) + vr.call('SetOverlayAlpha', surround, .85) + vr.pixels(surround, b'\x00\x00\x00\xff', 1, 1) + screen = vr.create('framecontrol.media.screen', 3 if args.theatre else 1.6, 2, + plan['layout'] != 'mono', aspect) + if splat: + vr.pixels(screen, data, width, height) + write_status(status, state='playing', file=path.name, frames=1, **plan) + while True: + time.sleep(1) + proc = subprocess.Popen(command, stdout=subprocess.PIPE) + video_start = time.monotonic() + while True: + data = proc.stdout.read(width * height * 4) + if not data: + break + data, outw, outh = frame_media.stereo_pixels(data, width, height, plan['layout']) + if not photo: + time.sleep(max(0, video_start + frames/30 - time.monotonic())) + vr.pixels(screen, data, outw, outh) + frames += 1 + if frames == 1 or frames % 30 == 0: + write_status(status, state='playing', file=path.name, frames=frames, + seconds=time.monotonic()-started, **plan) + if not photo: + time.sleep(max(0, video_start + frames/30 - time.monotonic())) + rc = proc.wait(timeout=10) + if rc: + raise RuntimeError('ffmpeg exited %s; see media log' % rc) + if not frames: + raise RuntimeError('Decoder produced no frames') + if photo: + while True: + time.sleep(1) + write_status(status, state='ended', frames=frames, seconds=time.monotonic()-started) + except InterruptedError: + write_status(status, state='stopped', frames=frames) + finally: + if proc: + if proc.poll() is None: + proc.terminate() + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + proc.stdout.close() + if vr: + vr.close() + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument('file') + ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto') + ap.add_argument('--theatre', action='store_true') + ap.add_argument('--status', required=True) + args = ap.parse_args() + try: + play(args) + except Exception as e: + write_status(Path(args.status), state='error', error=str(e)) + raise + + +if __name__ == '__main__': + main() diff --git a/ui/frame_media_remote.py b/ui/frame_media_remote.py new file mode 100644 index 0000000..acd7b90 --- /dev/null +++ b/ui/frame_media_remote.py @@ -0,0 +1,103 @@ +"""Frame-side library and process ownership for Frame Control media. + +Only the dedicated systemd user unit is controlled. No SteamVR settings change. +""" +import argparse +import json +from pathlib import Path +import re +import subprocess +import sys + +import frame_media +from frame_media_player import probe + +ROOT = Path.home() / 'Videos' / 'FrameControl' +RUNTIME = Path.home() / '.local' / 'share' / 'frame-control' / 'media' +UNIT = 'frame-control-media.service' +STATUS = RUNTIME / 'status.json' + + +def media_path(identity): + if not isinstance(identity, str) or '\\' in identity or '\x00' in identity: + raise ValueError('Invalid media id') + parts = Path(identity).parts + if len(parts) != 2 or not re.fullmatch('[0-9a-f]{32}', parts[0]) or parts[1].startswith('.'): + raise ValueError('Invalid media id') + candidate = ROOT / identity + if candidate.is_symlink() or candidate.parent.is_symlink(): + raise ValueError('Media links are not supported') + path = candidate.resolve(strict=True) + if not path.is_file() or ROOT.resolve() not in path.parents: + raise ValueError('Media file is outside the library') + return path + + +def active(): + return subprocess.run(['systemctl', '--user', 'is-active', '--quiet', UNIT]).returncode == 0 + + +def status(): + running = active() + try: + state = json.loads(STATUS.read_text()) + except (OSError, ValueError): + state = {'state': 'idle'} + if not running and state.get('state') in ('playing', 'starting', 'paused'): + state = {'state': 'stopped', 'message': 'Player exited; check the media log if this was unexpected'} + return dict(state, running=running) + + +def run(body): + action = body.get('action') + if action == 'list': + files = [] + if ROOT.exists(): + for folder in sorted(ROOT.iterdir()): + if not re.fullmatch('[0-9a-f]{32}', folder.name) or not folder.is_dir() or folder.is_symlink(): + continue + for path in sorted(folder.iterdir()): + if path.is_file() and not path.is_symlink() and not path.name.startswith('.'): + files.append({'id': folder.name+'/'+path.name, 'name': path.name, 'bytes': path.stat().st_size}) + return {'files': files, 'player': status()} + if action == 'status': + return status() + if action == 'stop': + subprocess.run(['systemctl', '--user', 'stop', UNIT], check=True, timeout=15) + return {'message': 'Media player stopped', **status()} + if action != 'play': + raise ValueError('Media action must be list, status, play or stop') + path = media_path(body.get('id')) + if type(body.get('theatre', False)) is not bool: + raise ValueError('theatre must be true or false') + info = {} if path.suffix.lower() == '.splat' else probe(path)[0] + plan = frame_media.plan(path.name, body.get('layout', 'auto'), info.get('tags')) + if active(): + raise ValueError('Stop the current media before starting another file') + # systemd owns the process group and refuses a concurrent start of this name. + # The runtime cap also cleans up if the controlling computer disconnects. + subprocess.run(['systemctl', '--user', 'reset-failed', UNIT], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, timeout=10) + STATUS.write_text(json.dumps({'state': 'starting', 'file': path.name})) + command = ['systemd-run', '--user', '--quiet', '--collect', '--unit='+UNIT, + '--property=RuntimeMaxSec=14400', '--property=TimeoutStopSec=8', + '--property=StandardOutput=append:'+str(RUNTIME/'player.log'), + '--property=StandardError=append:'+str(RUNTIME/'player.log'), + 'python3', str(RUNTIME/'frame_media_player.py'), str(path), + '--layout', plan['layout'], '--status', str(STATUS)] + if body.get('theatre'): + command.append('--theatre') + subprocess.run(command, check=True, capture_output=True, text=True, timeout=15) + return {'message': 'Starting Frame Control media', 'plan': plan} + + +def main(): + try: + print(json.dumps(run(json.load(sys.stdin)))) + except Exception as e: + print(json.dumps({'error': str(e)})) + sys.exit(1) + + +if __name__ == '__main__': + main() diff --git a/ui/frame_splat.py b/ui/frame_splat.py new file mode 100644 index 0000000..e3b149d --- /dev/null +++ b/ui/frame_splat.py @@ -0,0 +1,83 @@ +"""Small, bounded CPU Gaussian-splat preview renderer (Frame Control-owned). + +Reads the common 32-byte .splat record: position/scale float32 triplets, +RGBA bytes, then normalized quaternion bytes (wxyz). Two perspective cameras, +projected 3D covariance, back-to-front alpha compositing. This is a stationary +stereo preview, not a six-degree-of-freedom scene or a large-scene renderer. +""" +import math +from pathlib import Path +import struct + +MAX_SPLATS = 20000 +RECORD = struct.Struct('<6f8B') + + +def read(path): + size = Path(path).stat().st_size + if not size or size % RECORD.size or size > MAX_SPLATS * RECORD.size: + raise ValueError('Use a 32-byte .splat file with 1–20,000 Gaussians; PLY/SPZ and larger scenes are not supported yet') + values = [] + with open(path, 'rb') as stream: + for row in RECORD.iter_unpack(stream.read(MAX_SPLATS * RECORD.size + 1)): + xyz, scales = row[:3], row[3:6] + if not all(math.isfinite(v) and abs(v) <= 1e6 for v in row[:6]) or min(scales) <= 0: + raise ValueError('Invalid splat position or scale') + q = [(v - 128) / 128 for v in row[10:14]] + length = math.sqrt(sum(v*v for v in q)) + if length < .01: + raise ValueError('Invalid splat quaternion') + w, x, y, z = [v / length for v in q] + rotation = ((1-2*(y*y+z*z), 2*(x*y-z*w), 2*(x*z+y*w)), + (2*(x*y+z*w), 1-2*(x*x+z*z), 2*(y*z-x*w)), + (2*(x*z-y*w), 2*(y*z+x*w), 1-2*(x*x+y*y))) + cov = [[sum(rotation[i][k]*rotation[j][k]*scales[k]**2 for k in range(3)) + for j in range(3)] for i in range(3)] + values.append((xyz, cov, row[6:10])) + return values + + +def render(path, width=320, height=240): + values = read(path) + lo = [min(p[0][i] for p in values) for i in range(3)] + hi = [max(p[0][i] for p in values) for i in range(3)] + center = [(a+b)/2 for a, b in zip(lo, hi)] + radius = max(max(b-a for a, b in zip(lo, hi))/2, .01) + # Normalize captures to a two-metre box. Source units are not assumed metres. + normalized = [([(xyz[i]-center[i])/radius for i in range(3)], + [[v/radius**2 for v in row] for row in cov], color) + for xyz, cov, color in values] + normalized.sort(key=lambda p: p[0][2]) # camera is at z=3; farthest first + focal = width * .8 + eyes = [] + for eye in (-.032, .032): + pixels = bytearray(b'\x00\x00\x00\xff' * (width*height)) + for (x, y, z), cov, color in normalized: + x -= eye + depth = 3-z + px, py = width/2+focal*x/depth, height/2-focal*y/depth + jac = ((focal/depth, 0, focal*x/depth**2), + (0, -focal/depth, -focal*y/depth**2)) + screen = [[sum(jac[i][a]*cov[a][b]*jac[j][b] for a in range(3) for b in range(3)) + for j in range(2)] for i in range(2)] + a, b, c = screen[0][0]+.3, screen[0][1], screen[1][1]+.3 + det = a*c-b*b + if det <= 0 or not math.isfinite(det): + raise ValueError('Splat covariance is not renderable') + # A footprint cap bounds work on malformed or oversized Gaussians. + rx, ry = min(32, math.ceil(3*math.sqrt(a))), min(32, math.ceil(3*math.sqrt(c))) + for sy in range(max(0, int(py)-ry), min(height, int(py)+ry+1)): + dy = sy+.5-py + for sx in range(max(0, int(px)-rx), min(width, int(px)+rx+1)): + dx = sx+.5-px + power = (c*dx*dx-2*b*dx*dy+a*dy*dy)/det + if power > 9: + continue + alpha = color[3]/255 * math.exp(-.5*power) + offset = (sy*width+sx)*4 + for k in range(3): + pixels[offset+k] = round(color[k]*alpha+pixels[offset+k]*(1-alpha)) + eyes.append(pixels) + stride = width*4 + return b''.join(eyes[0][y*stride:(y+1)*stride]+eyes[1][y*stride:(y+1)*stride] + for y in range(height)), width*2, height diff --git a/ui/index.html b/ui/index.html index 8ac1265..f782b8a 100644 --- a/ui/index.html +++ b/ui/index.html @@ -87,6 +87,7 @@ .wait::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: var(--dim); flex: none; } /* ---- drop anywhere ---- */ + #media select { min-width: 0; max-width: 100%; flex: 1; } .dropzone { position: fixed; inset: 0; z-index: 40; display: grid; place-items: center; pointer-events: none; background: rgba(14,20,27,.82); backdrop-filter: blur(3px); } .dropzone > div { padding: 42px 60px; border: 2px dashed var(--blue); border-radius: 8px; text-align: center; @@ -611,6 +612,31 @@
Clipboard needs the desktop panel open in the headset.
+
+

Media in the headset

Preview
+

Play your movies and stereo photos with Frame Control's own player.

+
+
+ +
+
+
+
+
+
+
+

Choose a file to begin.

+
H.264/H.265 video; SBS/OU PNG or JPEG photos. Screens follow your head. + Small .splat files show a stationary stereo preview (up to 20,000 Gaussians). + Native HEIC/MPO photos, large splat scenes, seeking and subtitles aren't supported yet. + Stop works from here; playback also ends after four hours.
+
+

Linux apps

Flatpak
Loading…
@@ -1427,6 +1453,42 @@ function upload(file, mode) { xhr.send(file); }); } +async function refreshMedia(selectId) { + const data = await api("/api/media", { action: "list" }); + const selected = selectId || $("mediaFile").value; + $("mediaFile").replaceChildren(); + if (!data.files.length) $("mediaFile").add(new Option("Send media to begin", "")); + for (const file of data.files) $("mediaFile").add(new Option(file.name, file.id)); + if (data.files.some(f => f.id === selected)) $("mediaFile").value = selected; + const p = data.player; + $("mediaState").textContent = p.error || `${p.state}${p.file ? ": " + p.file : ""}`; + return data; +} +$("mediaChoose").onclick = () => $("mediaInput").click(); +$("mediaInput").onchange = () => act("Send media", async () => { + const file = $("mediaInput").files[0]; + if (!file) return; + const result = await upload(file, "media"); + await refreshMedia(result.id); + $("mediaInput").value = ""; + return result; +}, $("mediaChoose")); +$("mediaRefresh").onclick = () => act("Refresh media", () => refreshMedia(), $("mediaRefresh")); +$("mediaPlay").onclick = () => act("Play media", async () => { + const id = $("mediaFile").value; + if (!id) throw new Error("Send or select a media file first"); + const result = await api("/api/media", { action: "play", id, + layout: $("mediaLayout").value, theatre: $("mediaTheatre").checked }); + $("mediaState").textContent = "Starting…"; + setTimeout(() => refreshMedia().catch(e => { $("mediaState").textContent = e.message; }), 1500); + return result; +}, $("mediaPlay")); +$("mediaStop").onclick = () => act("Stop media", async () => { + const result = await api("/api/media", { action: "stop" }); + await refreshMedia(); + return result; +}, $("mediaStop")); + let apkLookup = 0; async function checkApkAlternatives(apk) { const lookup = ++apkLookup; diff --git a/ui/server.py b/ui/server.py index dd86ff6..59353e5 100755 --- a/ui/server.py +++ b/ui/server.py @@ -40,6 +40,7 @@ import frame_android # noqa: E402 import frame_apk_versions # noqa: E402 import frame_catalog # noqa: E402 import frame_host # noqa: E402 +import frame_media # noqa: E402 import frame_store # noqa: E402 import frame_titles # noqa: E402 import frame_webinstall # noqa: E402 @@ -1227,7 +1228,70 @@ def _sweep_one(prefix, d): pass -POST = {"/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard, +# ---- Our Frame-side media player ----------------------------------------- + +_MEDIA_LOCK = threading.Lock() + + +def media(body): + action = body.get("action") + if action not in ("list", "status", "play", "stop"): + raise Failure("Media action must be list, status, play or stop", 400) + if action == "play": + identity = body.get("id") + if not isinstance(identity, str) or not re.fullmatch(r"[0-9a-f]{32}/[^/\\\x00]+", identity): + raise Failure("Invalid media id", 400) + if body.get("layout", "auto") not in frame_media.LAYOUTS: + raise Failure("Invalid media layout", 400) + if type(body.get("theatre", False)) is not bool: + raise Failure("theatre must be true or false", 400) + with _MEDIA_LOCK: + # Ship only our small stdlib modules, atomically, to the user account. + sources = {name: (HERE / name).read_text() for name in ( + "frame_media.py", "frame_media_player.py", "frame_media_remote.py", "frame_splat.py")} + installer = """import json, os, pathlib, sys, tempfile +root = pathlib.Path.home()/'.local/share/frame-control/media' +root.mkdir(parents=True, exist_ok=True) +for name, source in json.load(sys.stdin).items(): + path = root/name + fd, temp = tempfile.mkstemp(dir=root, prefix=name+'.') + with os.fdopen(fd, 'w') as f: + f.write(source) + os.replace(temp, path) +""" + ssh("python3 -c " + shlex.quote(installer), stdin=json.dumps(sources)) + try: + out = ssh("python3 ~/.local/share/frame-control/media/frame_media_remote.py", + stdin=json.dumps(body), timeout=45) + except Failure as e: + for line in reversed(getattr(e, "stdout", "").splitlines()): + try: + detail = json.loads(line).get("error") + except (ValueError, AttributeError): + continue + if detail: + raise Failure(detail) from None + raise + return json.loads(out) + + +def push_media(path): + # Validate the format, but leave layout selection until playback (ffprobe + # can then read metadata on the Frame, where it is installed). + frame_media.plan(Path(path).name, "mono") + token = secrets.token_hex(16) + dest = "Videos/FrameControl/" + token + "/" + ssh("mkdir -p ~/" + dest) + try: + push_file(path, dest) + except Exception: + ssh("rm -rf ~/" + dest) + raise + return {"message": "Media sent. Choose its layout and press Play.", + "id": token + "/" + Path(path).name} + + +POST = {"/api/media": media, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard, "/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots, "/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start, "/api/webinstall/cancel": webinstall_cancel} @@ -1490,6 +1554,8 @@ class Handler(BaseHTTPRequestHandler): raise Failure("upload interrupted", 400) f.write(chunk) remaining -= len(chunk) + if mode == "media": + return push_media(dest) if mode == "apkinfo": # Read an APK for a report without installing it. try: