mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 03:00:18 +02:00
Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 saphidandClaude Opus 5.5
|
604e8da99e | ||
|
saphidandClaude Opus 5.5
|
8137740a4d | ||
|
saphidandClaude Opus 5.5
|
1cd56740a6 | ||
|
saphidandClaude Opus 5.5
|
094c12c6d8 | ||
|
Alex SouthwellandClaude Opus 5.5
|
e702adbd77 | ||
|
Alex Southwell
|
ef56025ad1 | ||
|
Alex Southwell
|
697452e5a9 | ||
|
saphid
|
ae7f733b90 | ||
|
saphidandClaude Opus 5.5
|
b00db2484d | ||
|
saphid
|
b0d6039eec | ||
|
saphidandClaude Opus 5.5
|
86b8ab1d82 | ||
|
saphidandClaude Opus 5.5
|
020e3386e1 | ||
|
saphidandClaude Opus 5.5
|
f81f70ed5d | ||
|
saphid
|
4d4e45f622 | ||
|
saphid
|
6ecd0cea39 | ||
|
saphid
|
c335cd5130 | ||
|
saphid
|
0d448ab510 |
No files matched your search
@@ -35,8 +35,8 @@ See what the headset sees, install games and Android apps, move files and text a
|
||||
<tr>
|
||||
<td width="50%" valign="top">
|
||||
|
||||
**👓 Headset view**<br>
|
||||
Live video of what the lenses show (about 30 fps), or a still of both eyes. Zoom, pan, full screen, save as PNG.
|
||||
**👓 Headset view and Desktop**<br>
|
||||
Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.
|
||||
|
||||
</td>
|
||||
<td width="50%" valign="top">
|
||||
|
||||
@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
|
||||
| Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
|
||||
| **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
|
||||
| **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
|
||||
| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
|
||||
| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
|
||||
| **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
|
||||
| Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
|
||||
| `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 409 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 901 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
@@ -102,10 +102,11 @@ Still open: 4, 6, 7, 12–15, 16 (off-LAN and after a reboot), 17–21.
|
||||
20. **F-Droid 2.0** (Compose 1.12): does it run? If so, the catalogue can
|
||||
install it instead of 1.17.2.
|
||||
|
||||
21. **DeoVR local files:** does DeoVR's file browser show `Videos → VR`
|
||||
(the symlink from `push-vr-video.sh`) or `Z:\home\steamos\Videos\VR`, and do
|
||||
the colour-coded test clips play in 3D (red left eye, cyan right) for both
|
||||
H.264 and H.265? Does the DLNA browser find a server on the Mac?
|
||||
21. **Owned media player:** worn-headset comfort, audio quality/lip sync, long
|
||||
movies and 4K/8K decoding remain to check. Native spatial-photo container
|
||||
extraction, large/immersive splats and existing-panel theatre docking need
|
||||
further implementation. Remote eye isolation, short hardware decode and
|
||||
owned-overlay cleanup are verified; see [vr-video.md](vr-video.md).
|
||||
|
||||
## Verified 2026-09-27
|
||||
|
||||
|
||||
@@ -123,3 +123,12 @@ a limit on the number of floating panels.
|
||||
keyboard) or virtual desktops arrange windows within the 1280×800 rectangle.
|
||||
- **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a
|
||||
PC's desktop in SteamVR. They don't run on the Frame's standalone Linux.
|
||||
|
||||
## Frame Control's media theatre
|
||||
|
||||
[The owned media player](vr-video.md) can show its video or stereo image on a
|
||||
larger, head-relative screen with its own dark surround. **Verified remotely
|
||||
2026-09-28**, SteamOS 0.4.1 / BUILD_ID 20260925.6191901: screen, eye isolation,
|
||||
surround and cleanup. It does not alter panel docking or global settings.
|
||||
Its `Overlay` RGBA rendering hook is available to stream producers; applying
|
||||
SteamVR theatre docking to existing Mac/PC panels is still unverified here.
|
||||
+1
-1
@@ -99,7 +99,7 @@ controls to place each panel. See [docs/panels.md](panels.md).
|
||||
| `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) |
|
||||
| `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) |
|
||||
| `scripts/compat-db-backup.sh` | Mac | Maintainer-only: back up the shared compatibility database locally and to Google Drive (**verified**) |
|
||||
| `scripts/push-vr-video.sh` | Mac → Frame | Upload VR180/360 videos to `~/Videos/VR`, linked into DeoVR's Proton prefix; `--launch` starts DeoVR (**verified**: upload and link; in-headset playback of local files not yet checked). See [docs/vr-video.md](vr-video.md) |
|
||||
| `scripts/push-vr-video.sh` | Computer → Frame | Upload movies, stereo PNG/JPEG or small `.splat` files to `~/Videos/FrameControl`; `--launch` starts our OpenVR player, `--theatre` adds a larger screen and dark surround. No separate viewer. **Verified remotely**: decode, stereo output and cleanup; worn-headset checks remain. See [docs/vr-video.md](vr-video.md) |
|
||||
| `scripts/push.sh` | Mac → Frame | `rsync` files to `~/Downloads` (or a given path) on the Frame (**verified**) |
|
||||
| `scripts/serve-bootstrap.sh` | Mac | Fallback: serve `bootstrap-on-frame.sh` with your public key embedded |
|
||||
| `scripts/bootstrap-on-frame.sh` | Frame | Fallback: install the key and enable `sshd` |
|
||||
|
||||
+66
-4
@@ -8,6 +8,7 @@ This covers three directions, plus input:
|
||||
- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md),
|
||||
including Valve's streaming and USB support. No Linux host tested yet.
|
||||
- **Input**: type and point in the Frame from the Mac or iPhone.
|
||||
- **Live view and Control**: watch a panel flat and tap on it to use it.
|
||||
|
||||
The confidence labels are the same as in [ssh.md](ssh.md).
|
||||
|
||||
@@ -45,8 +46,10 @@ flat 2D desktop streaming into a window on the Frame's Linux desktop.
|
||||
| Immersed / Virtual Desktop | Vendor apps | Immersed has a Mac agent but no known Frame client. Virtual Desktop's developer said he'd "try" to port it ([NewsBreak](https://www.newsbreak.com/news/4892834783961-virtual-desktop-dev-says-he-ll-try-to-bring-the-app-to-steam-frame)). | Not available as of 2026-09-25. Check again later. |
|
||||
| WiVRn / ALVR | VR streaming from a Linux or Windows PC | Irrelevant for a Mac host (no SteamVR/OpenXR runtime on macOS) | N/A |
|
||||
|
||||
For **VR video files** (180°/360° stereo), don't stream the Mac's screen. Play
|
||||
them on the Frame in DeoVR instead: see [vr-video.md](vr-video.md).
|
||||
For **local movies and stereo photos**, Frame Control's own OpenVR player
|
||||
runs on the Frame; see [vr-video.md](vr-video.md). It currently renders a flat
|
||||
stereo screen. VR180/360 projection is not implemented; the same page records
|
||||
DeoVR only as an optional, independently installed alternative.
|
||||
|
||||
### Pre-seeding the Remmina profile (no typing in the headset)
|
||||
|
||||
@@ -182,13 +185,72 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
|
||||
and ignores injected pointer motion; `:1` holds apps such as Chromium and
|
||||
takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
|
||||
There's also a `gamescope-0-ei` (libei) socket.
|
||||
- **Not yet tested:** typing and clicking as seen in the headset, and whether
|
||||
it reaches the KDE desktop panel (Plasma is its own session).
|
||||
- Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
|
||||
panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
|
||||
Control's own panels, Steam's UI) and, since XTest positions are clamped to
|
||||
`:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
|
||||
the live view (below) has neither limit.
|
||||
- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
|
||||
own session).
|
||||
- **Known limit:** keys and clicks typed while the link is reconnecting wait
|
||||
and are sent once it's back, but anything sent in the moment the Wi-Fi
|
||||
drops, before SSH notices, can be lost. Confirming every event would add a
|
||||
round trip to each pointer move.
|
||||
|
||||
## Live view and Control: watch a panel and tap on it
|
||||
|
||||
**Built: Home → Desktop / Headset view → Control.** The live view has two
|
||||
sources:
|
||||
|
||||
- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
|
||||
moves with the wearer's head, so Control makes the view a trackpad: drag to
|
||||
move the pointer, tap to click, press and hold to right-click, two fingers to
|
||||
scroll. With a mouse, moving over the view moves the pointer.
|
||||
- **Desktop**: the app panel in use in the headset, from its own window, so it
|
||||
stays still however the wearer looks around. Control makes taps and clicks
|
||||
land exactly where you put them. Dragging is a mouse drag, press and hold is a
|
||||
right-click, two fingers scroll, and on a computer the mouse, wheel and
|
||||
keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
|
||||
other panel, view only.
|
||||
|
||||
Below the view, a text field and key buttons type on the Frame from a phone.
|
||||
|
||||
How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
|
||||
|
||||
- **Input goes through gamescope's own injection.** gamescope serves an EIS
|
||||
socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
|
||||
it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
|
||||
talks to it with `ctypes`: nothing to install. gamescope offers one device,
|
||||
"Gamescope Virtual Input", with relative and absolute pointer, buttons,
|
||||
scroll and keyboard (Linux key codes; no text capability, so the text field
|
||||
types printable ASCII on a US layout). Its absolute region is unbounded; the
|
||||
pointer uses the focused panel's display coordinates, and gamescope fits each
|
||||
window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
|
||||
positions at two thirds scale. Taps on a 1280×720 page landed on the exact
|
||||
pixel.
|
||||
- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
|
||||
root), on either X display. In the OpenVR backend focus moves only on SteamVR
|
||||
overlay events (the controller's laser entering or clicking a panel), or to a
|
||||
new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
|
||||
with `gamescopectl focus_info`, which writes to the journal). Neither
|
||||
`GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
|
||||
gamescope command does. So Control follows the wearer: whatever they last
|
||||
used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
|
||||
gets a connector of its own and doesn't hold focus.
|
||||
- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
|
||||
apart).
|
||||
- **Known limit:** if focus moves to another panel in the middle of a drag, the
|
||||
release goes to the panel that has focus then. Whether gamescope hands it to
|
||||
the window that got the press isn't known yet. When the session ends, the
|
||||
helper lets go of every button and key it still holds.
|
||||
- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
|
||||
-window_id <window> -i :<display>` works on gamescope's redirected windows,
|
||||
while grabbing the root gives black. It streams as H.264 like the headset view
|
||||
(about 30 fps at 720p).
|
||||
- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
|
||||
text box in the panel and the text field typed into it; a trackpad move went
|
||||
exactly (+40, +25).
|
||||
|
||||
Our own `uinput` keyboard and mouse would also work (`steamos` is in the
|
||||
`input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
|
||||
remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.
|
||||
|
||||
@@ -149,6 +149,20 @@ refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
|
||||
accepted. The fake now refuses them the same way, and Frame Control makes ids
|
||||
Steam accepts.
|
||||
|
||||
## Owned media player
|
||||
|
||||
`tests/test_media.py` covers layout evidence and overrides, OU eye ordering,
|
||||
hardware-decoder command construction, malformed splats, stereo parallax and
|
||||
fake-Frame library/process ownership. `tests/e2e/test_media_transfer.py` checks the real
|
||||
HTTP/SSH upload and library listing without pretending the fake renders VR.
|
||||
Real decode timings and captured stereo output are recorded in
|
||||
[vr-video.md](vr-video.md). Generated media only; no external player required.
|
||||
|
||||
**Verified 2026-09-28**, real Frame BUILD_ID 20260925.6191901: `~/.local`
|
||||
and `~/.local/share` are `steamos:steamos`, mode 0755. The fake supervisor
|
||||
sets those parent owners too; previously its root-created Steam manifests
|
||||
left the parents root-owned and incorrectly prevented user runtime installs.
|
||||
|
||||
## Agent interfaces
|
||||
|
||||
`tests/test_agent.py` exercises MCP stdio, exact-action human approvals and the
|
||||
|
||||
+167
-72
@@ -1,81 +1,176 @@
|
||||
# Watching VR video (180°/360°) on the Frame
|
||||
# Movies, stereo photos and splats
|
||||
|
||||
The confidence labels are the same as in [ssh.md](ssh.md). Everything here
|
||||
was checked on SteamOS 0.3.0, build 20260922.6101926.
|
||||
Frame Control has its own Frame-side OpenVR player. It uses SteamOS's ffmpeg
|
||||
and V4L2 hardware decoder, Python and SteamVR. **No separate player or viewer
|
||||
is required.** Chromium and immersive WebXR are not in this playback path.
|
||||
|
||||
## The short version
|
||||
## Use it
|
||||
|
||||
1. Install **DeoVR Video Player** from Steam (free, app 837380) and start it once
|
||||
in the headset. That creates its Proton prefix.
|
||||
2. On the Mac: `scripts/push-vr-video.sh --launch ~/Movies/beach_180_LR.mp4`
|
||||
3. In the headset, open DeoVR's local file browser → **Videos → VR** and
|
||||
pick the file.
|
||||
In **Tools → Media in the headset**, send a file, choose its layout and press
|
||||
**Play**. **Theatre** gives it a larger screen and an 85% black surround.
|
||||
**Stop** removes both. Refresh reads the library and the player's state.
|
||||
The screen follows your head; it isn't a saved world-space panel.
|
||||
|
||||
## Why DeoVR
|
||||
|
||||
The Frame's Chromium can't play VR video in 3D. It has no immersive WebXR
|
||||
([how-the-frame-works.md](how-the-frame-works.md)). DeoVR's Windows build
|
||||
runs under Proton ARM64 + FEX as a real SteamVR app. **Verified 2026-09-25:**
|
||||
it found the Steam Frame headset and controller over OpenVR, and decoded
|
||||
7680×3840 and 8192×4096 H.265 VR180 side-by-side streams through AVPro's
|
||||
hardware Media Foundation path, mapped onto a 180° dome or fisheye mesh.
|
||||
|
||||
Known quirks (verified):
|
||||
|
||||
- The first launch takes about 45 s while it compiles shaders.
|
||||
- Grid thumbnails stay blank. Unity's own video player, which DeoVR uses for
|
||||
previews, fails with `0xc00d36bb` under Proton. Full playback uses AVPro
|
||||
and isn't affected.
|
||||
- The in-app store and web content are separate from local files. You don't
|
||||
need an account to play your own files.
|
||||
|
||||
## Getting files onto the headset
|
||||
|
||||
`scripts/push-vr-video.sh` copies files with `rsync --partial`, so an
|
||||
interrupted upload resumes. They go to `~/Videos/VR` on the Frame (`/home`
|
||||
has about 860 GB free). The script also links that folder into DeoVR's prefix
|
||||
as `C:\users\steamuser\Videos\VR`. It's reachable at
|
||||
`Z:\home\steamos\Videos\VR` as well. **Verified** that the upload and link
|
||||
work. **Not yet checked** whether DeoVR's file browser lands there
|
||||
(open question 21).
|
||||
|
||||
Speed: a test upload over Wi-Fi ran at about 3–5 MB/s (verified 2026-09-25,
|
||||
one sample). At that rate an 8K file of several GB takes tens of minutes, so
|
||||
start big uploads before you put the headset on.
|
||||
|
||||
## Naming files so they play correctly
|
||||
|
||||
DeoVR guesses the projection from the file name. Its binary contains the tags
|
||||
`_180`, `_360`, `_fisheye`, `_fisheye190`, `_mkx200`, `_vrca220` and `_rf52`
|
||||
(verified). For stereo layout, the common DeoVR convention is `_LR`/`_SBS`
|
||||
(side by side) and `_TB` (top/bottom) (inferred). If a video looks wrong
|
||||
(doubled, warped, or flat), change the projection and stereo mode in DeoVR's
|
||||
player menu.
|
||||
|
||||
Examples: `trip_180_LR.mp4`, `concert_360_TB.mp4`, `hike_fisheye190_LR.mp4`.
|
||||
|
||||
Codecs: H.265 at 8K played (verified, streamed). Local H.264 and H.265
|
||||
files haven't been played yet. The test clips below cover that.
|
||||
|
||||
## Test clips
|
||||
|
||||
The script doesn't include these. To check a setup, make two 20 s clips:
|
||||
3840×1920, 180° side by side, with the left eye tinted red and the right eye
|
||||
cyan. In the headset each eye should see only its own colour. A single mixed
|
||||
colour means the stereo split is wrong.
|
||||
The command-line route uses the same player:
|
||||
|
||||
```sh
|
||||
ffmpeg -f lavfi -i testsrc2=size=1920x1920:rate=30:duration=20 \
|
||||
-filter_complex "[0:v]split[a][b];[a]colorchannelmixer=rr=1:gg=0.3:bb=0.3[l];[b]colorchannelmixer=rr=0.3:gg=1:bb=1[r];[l][r]hstack" \
|
||||
-c:v libx264 -pix_fmt yuv420p -b:v 20M frame-test_180_LR_h264.mp4
|
||||
scripts/push-vr-video.sh frame-test_180_LR_h264.mp4
|
||||
scripts/push-vr-video.sh --launch --theatre ~/Movies/film_SBS.mp4
|
||||
scripts/push-vr-video.sh --launch --layout ou ~/Pictures/stereo.png
|
||||
scripts/push-vr-video.sh --launch capture.splat
|
||||
scripts/push-vr-video.sh --list
|
||||
scripts/push-vr-video.sh --stop
|
||||
```
|
||||
|
||||
## Streaming from the Mac instead of copying (untested)
|
||||
Uploads live in `~/Videos/FrameControl/<id>/` on the Frame. Each upload gets
|
||||
its own directory, so sending another file with the same name doesn't replace
|
||||
it. The UI's existing upload limit is 8 GiB. Transfers use the app's rsync/scp
|
||||
path; resumable large uploads are not implemented here yet. Existing
|
||||
`~/Videos/VR` files and Proton prefixes are left alone. The script no longer
|
||||
launches DeoVR, links a Proton prefix, or accepts directories.
|
||||
|
||||
DeoVR has a DLNA browser (the binary contains `Searching for DLNA
|
||||
devices...` and a UPnP ContentDirectory client). A DLNA server on the Mac
|
||||
should therefore appear in DeoVR without copying anything, for example
|
||||
`brew install rclone` then `rclone serve dlna ~/Movies/VR`. This is **inferred**, not
|
||||
tried. 8K VR video needs roughly 50–100 Mbit/s sustained, and the Wi-Fi
|
||||
sample above (about 30–40 Mbit/s) suggests copying first is the safer default.
|
||||
| Media | Supported preview |
|
||||
|---|---|
|
||||
| Movies | H.264 / H.265, mono or left-first SBS / top-first OU; audio through the Frame's PulseAudio-compatible server |
|
||||
| Stereo photos | PNG / JPEG containing both eyes, SBS or OU |
|
||||
| Gaussian splats | Common 32-byte `.splat` records, 1–20,000 Gaussians; a stationary stereo preview |
|
||||
|
||||
Auto layout reads delimited filename tags: `_SBS`, `_HSBS`, `_LR`, `_OU`,
|
||||
`_TB`, `_HOU`, `_HTB`, `_FSBS`, `_FOU`, `_FTB`. SBS/OU without `F` means
|
||||
half-resolution packing. Full packing preserves each eye's original aspect.
|
||||
It also accepts FFmpeg's `stereo_mode` metadata (`left_right`, `top_bottom`,
|
||||
`mono`); left/right and top/bottom metadata are treated as full packing.
|
||||
Choose an explicit layout if that assumption doesn't match the file.
|
||||
Unknown or conflicting tags ask for a choice, rather than silently flattening
|
||||
stereo. The explicit selector always wins. Layout is not guessed from resolution.
|
||||
|
||||
## What was verified
|
||||
|
||||
**Verified remotely, 2026-09-28:** SteamOS 0.4.1, BUILD_ID
|
||||
`20260925.6191901`, SteamVR 2.18.1. Nobody wore the headset for these checks.
|
||||
All test media were generated by us. No paid content or DRM was involved.
|
||||
|
||||
- Stock `ffmpeg` `h264_v4l2m2m` decoded 150 frames of our 1920×1080 H.264
|
||||
test in 0.128 s (decode only); converting all frames to RGBA took 0.242 s.
|
||||
- Our ffmpeg → Python → OpenVR path submitted all 150 frames and exited 0.
|
||||
The 1280×720 prototype took 4.775 s; the full 1920×1080 player took
|
||||
4.618 s. These are wall times, not in-headset frame-rate measurements.
|
||||
Finishing a 5 s clip early showed those probes weren't paced, so the final
|
||||
player paces output at 30 fps: all 150 frames then completed in **5.025 s**.
|
||||
- The H.265 hardware decoder also completed the generated 1080p clip
|
||||
(60 frames, exit 0); this is a short compatibility check, not a 4K/8K benchmark.
|
||||
- Our actual player, launched through Frame Control's media API, displayed
|
||||
SBS and OU photos with red only in the left-eye capture and cyan only in
|
||||
the right. OpenVR's `SideBySide_Parallel` flag separates the eyes; we
|
||||
rearrange OU rows ourselves.
|
||||
- A generated 48-Gaussian `.splat` rendered in our own CPU renderer and appeared
|
||||
in both eyes with separate perspective projections.
|
||||
- Theatre's owned dark surround and screen appeared in captures. Steam's
|
||||
dashboard remained available over them. Stop removed the owned overlays
|
||||
and ended the dedicated user service. No global SteamVR setting was changed.
|
||||
- A generated H.264/AAC clip reached the Pulse audio output and completed
|
||||
all 100 video frames with exit 0. This proves the output path, not audible
|
||||
quality or lip sync.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**Verified end to end, 2026-09-29** (same build; headset unworn): uploads
|
||||
through the HTTP API, the web UI and `scripts/push-vr-video.sh`, all played by
|
||||
the owned player. Our generated test files:
|
||||
|
||||
| Case | Result |
|
||||
|---|---|
|
||||
| H.264 half-SBS 1920×1080 with AAC, theatre | 240/240 frames in 8.09 s; audio stream "Frame Control Media" in PulseAudio |
|
||||
| H.265 half-OU 1920×1080 | 180/180 frames in 6.03 s; red left eye, cyan right |
|
||||
| H.264 full-SBS 3840×1080, `stereo_mode=left_right` only | Detected from metadata; 150/150 frames in 5.03 s |
|
||||
| H.264 1280×720, explicit 2D | 150/150 frames in 5.02 s |
|
||||
| SBS PNG, OU JPEG (theatre) | Correct eye in each capture |
|
||||
| 3,000-Gaussian `.splat` | Rendered in about 5 s, then held until Stop |
|
||||
| 2D file on Auto, `_SBS_OU` file, HEIC, VP9 | Refused with the documented message |
|
||||
| Second Play while one runs | Refused: "Stop the current media…" |
|
||||
|
||||
Stop always left the unit inactive, and no player process remained.
|
||||
|
||||
**Standby (verified):** an unworn Frame turns its displays off a few seconds
|
||||
after it wakes. `SetOverlayRaw` then returns `RequestFailed` (23). The
|
||||
first run's movie died there. The player now drops frames while the headset
|
||||
is in standby, keeps the audio and its clock going, and resumes the picture
|
||||
when the headset wakes. The 8 s movie above dropped 40 frames and finished.
|
||||
Stills and the theatre surround are re-sent after waking. Five minutes
|
||||
without an accepted frame is reported as an error. Headset-view captures
|
||||
taken during standby show a flat dark frame, not our screen.
|
||||
|
||||
|
||||
|
||||
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
|
||||
`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139)
|
||||
in the basic appsink probe and the OpenVR probe. We do not ship that route.
|
||||
The ffmpeg path above completed instead.
|
||||
|
||||
## Limits and blockers
|
||||
|
||||
- **Not verified:** worn-headset comfort, sound quality/lip sync, long movies,
|
||||
4K/8K decode, HDR, controller interaction or behaviour on other OS builds.
|
||||
Output is bounded to 1920×1080 packed pixels before OU rearrangement.
|
||||
- **Not implemented:** pause, seeking, subtitles, playlists, right-eye-first
|
||||
layouts, non-square pixel correction, VR180/360 projection and fisheye.
|
||||
This preview is a flat stereo screen, not a dome player.
|
||||
- **Native spatial-photo blocker:** HEIC/HEIF/AVIF/MPO stereo-container
|
||||
extraction isn't implemented in our viewer. We reject these rather than
|
||||
displaying one image and calling it spatial. Export both eyes to PNG/JPEG
|
||||
first. This is a current implementation gap, not a claim that the Frame
|
||||
cannot support these containers.
|
||||
- **Large/immersive splat blocker:** the owned CPU rasterizer projects 3D
|
||||
covariance into each eye and alpha-composites Gaussians, but renders a
|
||||
fixed view at 320×240 per eye. It caps each Gaussian footprint at 32 pixels
|
||||
and normalizes the scene to a two-metre box. Large scenes, PLY/SPZ, live
|
||||
head-position parallax and navigation need a GPU scene renderer; this
|
||||
version rejects files above 20,000 records. It is a stereo preview, not
|
||||
an immersive walk-through.
|
||||
- A single player can run at a time. It stops at EOF, on **Stop**, or after
|
||||
four hours in any case, so longer movies are cut off. Still images remain
|
||||
until Stop or that timeout. There is no delete action yet: remove old
|
||||
uploads from `~/Videos/FrameControl/` over SSH. The log is
|
||||
`~/.local/share/frame-control/media/player.log` on the Frame.
|
||||
- **Panel/stream theatre remains outside this media slice:** SteamVR's
|
||||
`vrcmd --dock-overlay` accepts `theater`, but docking an existing panel
|
||||
and its dimming behaviour were not verified here. The shared headset had
|
||||
workspace and stream tests active. We did not reposition their panels.
|
||||
Integration with [#22](https://github.com/saphid/frame-control/issues/22)
|
||||
and [#31](https://github.com/saphid/frame-control/issues/31) can use the
|
||||
owned rendering hook below; no second stream/window manager is introduced.
|
||||
|
||||
## Integration hook
|
||||
|
||||
`POST /api/upload` with `X-Mode: media` and `X-Filename` sends a file and
|
||||
returns its `id`. `POST /api/media` accepts:
|
||||
|
||||
```json
|
||||
{"action":"play","id":"<32 hex characters>/film_SBS.mp4","layout":"auto","theatre":true}
|
||||
```
|
||||
|
||||
Other actions are `list`, `status`, `stop`. These use the normal `X-Frame-UI`
|
||||
guard. Play reports **starting**, not a claim that frames reached the headset;
|
||||
read status for `playing`, `ended`, `stopped` or `error`.
|
||||
The own-code files are copied to `~/.local/share/frame-control/media/` and
|
||||
run in the `frame-control-media.service` systemd user unit. Stop affects only
|
||||
that unit, including its decoder child.
|
||||
|
||||
For a Frame-side stream producer, `frame_media_player.Overlay` exposes
|
||||
`create(key, width, distance, stereo=False, aspect=1, order=1)`,
|
||||
`pixels(handle, rgba, width, height)` and `close()`. Use an owned unique key,
|
||||
pass interleaved RGBA bytes (left/right halves for stereo) and always close in
|
||||
`finally`. `create` uses a head-relative transform; it does not move another
|
||||
app's panel. The player demonstrates a separate black surround overlay.
|
||||
`frame_media.stereo_pixels` converts OU to SBS. This is the narrow rendering
|
||||
hook for stream/workspace work; it doesn't capture or manage a Mac/PC stream.
|
||||
|
||||
## Optional separate app
|
||||
|
||||
You can independently install **DeoVR Video Player** (free Steam app 837380)
|
||||
if you prefer its VR180/360 features. Earlier tests on SteamOS 0.3.0,
|
||||
build `20260922.6101926` (2026-09-25), verified its OpenVR initialization and
|
||||
8K H.265 streamed VR180 decoding under Proton. Frame Control's media features
|
||||
neither install nor launch it, and do not depend on it. Its behaviour and
|
||||
file-naming conventions are not evidence about our player.
|
||||
@@ -1,65 +1,8 @@
|
||||
#!/usr/bin/env zsh
|
||||
# Mac-side: upload VR videos to the Steam Frame so DeoVR can play them in 3D.
|
||||
#
|
||||
# Files go to ~/Videos/VR on the Frame. The script links that folder into
|
||||
# DeoVR's Proton prefix as C:\users\steamuser\Videos\VR, so DeoVR's file
|
||||
# browser finds it under Videos. (It's also reachable as Z:\home\steamos\Videos\VR.)
|
||||
# Uploads resume if interrupted.
|
||||
#
|
||||
# Name files so DeoVR picks the projection: include _180 or _360 (or _fisheye190,
|
||||
# _mkx200, _rf52 ...) plus the stereo layout (_LR / _SBS side by side, _TB over-
|
||||
# under), e.g. "beach_180_LR.mp4". You can also change it in DeoVR's player.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/push-vr-video.sh FILE_OR_DIR... # upload
|
||||
# scripts/push-vr-video.sh --launch FILE... # upload, then start DeoVR
|
||||
# scripts/push-vr-video.sh --launch # just start DeoVR
|
||||
# scripts/push-vr-video.sh --list # what's on the Frame
|
||||
# Upload media for Frame Control's own OpenVR player (no external player).
|
||||
# Usage: scripts/push-vr-video.sh [--launch] [--layout auto|mono|sbs|ou|full-sbs|full-ou] [--theatre] FILE...
|
||||
# scripts/push-vr-video.sh --list | --stop
|
||||
# Files go to ~/Videos/FrameControl/<id>/; --launch accepts exactly one file.
|
||||
# Directories, automatic DeoVR launching and Proton-prefix links are no longer used.
|
||||
set -euo pipefail
|
||||
|
||||
FRAME_ALIAS=${FRAME_ALIAS:-frame}
|
||||
DEOVR_APPID=837380
|
||||
REMOTE_DIR="Videos/VR"
|
||||
PREFIX_VIDEOS=".local/share/Steam/steamapps/compatdata/$DEOVR_APPID/pfx/drive_c/users/steamuser/Videos"
|
||||
|
||||
launch=0 list=0
|
||||
while (( $# )); do
|
||||
case "$1" in
|
||||
-h|--help) sed -n '2,17p' "$0"; exit 0 ;;
|
||||
--launch) launch=1; shift ;;
|
||||
--list) list=1; shift ;;
|
||||
--) shift; break ;;
|
||||
-*) print -u2 "push-vr-video: unknown option $1"; exit 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
(( $# || launch || list )) || { sed -n '2,17p' "$0" >&2; exit 2; }
|
||||
|
||||
for f in "$@"; do
|
||||
[[ -e "$f" ]] || { print -u2 "push-vr-video: no such file: $f"; exit 2; }
|
||||
done
|
||||
|
||||
# Create the folder and link it into DeoVR's prefix (the prefix exists once
|
||||
# DeoVR has run). Refresh a stale link, but never replace a real directory.
|
||||
if (( $# || launch )); then
|
||||
ssh "$FRAME_ALIAS" "mkdir -p ~/$REMOTE_DIR
|
||||
p=~/$PREFIX_VIDEOS
|
||||
if [ -d \"\$p\" ] && { [ -L \"\$p/VR\" ] || [ ! -e \"\$p/VR\" ]; }; then ln -sfn ~/$REMOTE_DIR \"\$p/VR\"
|
||||
elif [ -d \"\$p/VR\" ]; then echo \"warning: \$p/VR is a real folder, so uploads won't show under DeoVR's Videos; browse Z:\\\\home\\\\steamos\\\\Videos\\\\VR instead\" >&2; fi
|
||||
[ -d \"\$p\" ] || echo 'note: DeoVR has not run yet; use Z:\\home\\steamos\\Videos\\VR or run this again after starting it once' >&2"
|
||||
fi
|
||||
|
||||
if (( $# )); then
|
||||
# -L: send what a symlink points at; a Mac-side link would dangle on the Frame
|
||||
rsync -aL --partial --progress -- "$@" "$FRAME_ALIAS:$REMOTE_DIR/"
|
||||
fi
|
||||
|
||||
if (( list )); then
|
||||
ssh "$FRAME_ALIAS" "cd ~/$REMOTE_DIR && ls -lhR"
|
||||
fi
|
||||
|
||||
if (( launch )); then
|
||||
ssh "$FRAME_ALIAS" "command -v steam >/dev/null || { echo 'steam not found on the Frame' >&2; exit 1; }
|
||||
steam steam://rungameid/$DEOVR_APPID </dev/null >/dev/null 2>&1 &"
|
||||
print "DeoVR starting on the Frame. Open Local files / the file browser → Videos → VR."
|
||||
fi
|
||||
exec python3 "${0:A:h}/../ui/frame_media_cli.py" "$@"
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Media transfer through the real HTTP/SSH path; the fake has no VR renderer."""
|
||||
import harness
|
||||
from harness import ok, ssh
|
||||
|
||||
harness.require()
|
||||
|
||||
|
||||
class Media(harness.FrameTestCase):
|
||||
def test_upload_and_list_without_launching_a_viewer(self):
|
||||
# The transfer endpoint doesn't decode. Rendering belongs to the real
|
||||
# headset smoke checks documented in docs/vr-video.md.
|
||||
self.assertEqual(ssh('stat -c "%U:%G %a" ~/.local/share').strip(), 'steamos:steamos 755')
|
||||
result = ok('POST', '/api/upload', raw=b'fake-media', headers={
|
||||
'X-Mode': 'media', 'X-Filename': 'test_SBS.png'})
|
||||
identity = result['id']
|
||||
try:
|
||||
files = ok('POST', '/api/media', {'action': 'list'})['files']
|
||||
self.assertIn(identity, [f['id'] for f in files])
|
||||
self.assertEqual(ssh('cat ~/Videos/FrameControl/'+identity), 'fake-media')
|
||||
finally:
|
||||
ssh('rm -rf ~/Videos/FrameControl/'+identity.split('/')[0])
|
||||
@@ -231,6 +231,13 @@ def sysfs(state):
|
||||
|
||||
def runtimes(state):
|
||||
"""Installed compat tools as Steam app manifests, and the Lepton launcher itself."""
|
||||
# Verified 2026-09-28, BUILD_ID 20260925.6191901: both parents are
|
||||
# steamos:steamos 0755. The root supervisor must not leave them root-owned
|
||||
# when creating Steam's fake manifests; user-account app installs need them.
|
||||
for directory in (HOME + '/.local', HOME + '/.local/share'):
|
||||
os.makedirs(directory, mode=0o755, exist_ok=True)
|
||||
chown(directory)
|
||||
os.chmod(directory, 0o755)
|
||||
apps = fs.STEAM_ROOT + '/steamapps'
|
||||
for alias, installed in state['runtimes'].items():
|
||||
acf = f'{apps}/appmanifest_{fs.RUNTIME_APPIDS[alias]}.acf'
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
// Control on the live view (tap, drag, hold, scroll, type), run in node against the real
|
||||
// functions from ui/index.html with a fake canvas and a fake server.
|
||||
import { readFileSync } from "fs";
|
||||
const src = readFileSync(new URL("../../ui/index.html", import.meta.url), "utf8");
|
||||
const grab = name => {
|
||||
const one = src.match(new RegExp(`\\n((?:async )?function ${name}\\(.*\\}\\n)`)); // one-line function
|
||||
if (one) return one[1];
|
||||
const m = src.match(new RegExp(`(?:\\nconst ${name} = [^\\n]*\\n)|((?:async )?function ${name}\\([\\s\\S]*?\\n}\\n)`));
|
||||
if (!m) throw new Error("not found: " + name);
|
||||
return m[0];
|
||||
};
|
||||
const NAMES = ["panelKey", "ctrlAimedAt", "ctrlSameTarget", "isMoveEvent", "isRelease", "ctrlKeepable", "TAP_MOVE", "ctrlAim", "ctrlKeyEvent", "ctrlTouchCancel", "ctrlSend", "ctrlFlush", "ctrlMoveTo", "ctrlMoveBy", "ctrlSchedule",
|
||||
"ctrlFlushMoves", "ctrlButton", "ctrlClick", "ctrlRelease", "ctrlFraction", "ctrlTouchDown", "centroid",
|
||||
"ctrlTouchMove", "ctrlTouchUp", "ctrlTap", "ctrlText"];
|
||||
const code = NAMES.map(grab).join("");
|
||||
const fail = msg => { console.log("FAIL " + msg); process.exit(1); };
|
||||
const tick = (ms = 0) => new Promise(r => setTimeout(r, ms));
|
||||
|
||||
function page({ mode = "abs", rect = { left: 0, top: 0, width: 640, height: 360 }, api } = {}) {
|
||||
const target = { panel: { window: 42, display: ":1" } };
|
||||
const ctrl = { on: true, queue: [], sending: false, state: "ready", message: "", move: null, rel: [0, 0], raf: 0,
|
||||
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
|
||||
const sent = [];
|
||||
const canvas = { width: 1280, height: 720, getBoundingClientRect: () => rect };
|
||||
const env = {
|
||||
ctrl, $: () => canvas, ctrlMode: () => mode, ctrlShow: () => {}, toast: () => {},
|
||||
ctrlTarget: () => (mode !== "abs" ? { ok: true } : target.panel ? { ok: true, panel: target.panel } : { why: "gone" }),
|
||||
api: api || (async (path, body) => { sent.push(...body.events); return { state: "ready", sent: true }; }),
|
||||
requestAnimationFrame: cb => { setTimeout(cb, 0); return 1; },
|
||||
navigator: {},
|
||||
};
|
||||
const fns = new Function(...Object.keys(env), `let ctrlWarned = false;\n${code}
|
||||
return { ctrlTouchDown, ctrlTouchMove, ctrlTouchUp, ctrlTouchCancel, ctrlSend, ctrlText, ctrlButton, ctrlKeyEvent, ctrlRelease, ctrlFlushMoves };`)(...Object.values(env));
|
||||
const at = (id, x, y) => ({ pointerId: id, clientX: x, clientY: y });
|
||||
return { ctrl, sent, target, ...fns, at };
|
||||
}
|
||||
|
||||
// A tap lands where it was tapped: pointer there first, then the click.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
|
||||
await tick(10);
|
||||
const [move, down, up] = p.sent;
|
||||
if (!(move.fx === 0.5 && Math.abs(move.fy - 0.25) < 1e-9 && move.window === 42 && move.display === ":1")) fail("tap position " + JSON.stringify(move));
|
||||
if (!(down.window === 42 && down.display === ":1")) fail("a press names its panel " + JSON.stringify(down));
|
||||
if (!(down.button === "left" && down.down && up.button === "left" && up.down === false && p.sent.length === 3))
|
||||
fail("tap click " + JSON.stringify(p.sent));
|
||||
}
|
||||
// The panel is letterboxed in a taller view: taps map inside it, taps on the bars do nothing.
|
||||
{
|
||||
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } }); // 640x360 picture, 60px bars
|
||||
p.ctrlTouchDown(p.at(1, 320, 150)); p.ctrlTouchUp(p.at(1, 320, 150));
|
||||
p.ctrlTouchDown(p.at(2, 320, 10)); p.ctrlTouchUp(p.at(2, 320, 10));
|
||||
await tick(10);
|
||||
if (!(p.sent.length === 3 && Math.abs(p.sent[0].fy - 0.25) < 1e-9)) fail("letterbox " + JSON.stringify(p.sent));
|
||||
}
|
||||
// A drag is a mouse drag: down where it started, moves, up at the end.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 100, 100));
|
||||
p.ctrlTouchMove(p.at(1, 140, 100)); await tick(5);
|
||||
p.ctrlTouchMove(p.at(1, 200, 120)); await tick(5);
|
||||
p.ctrlTouchUp(p.at(1, 200, 120)); await tick(10);
|
||||
const kinds = p.sent.map(e => "fx" in e ? "move" : `${e.button}-${e.down ? "down" : "up"}`);
|
||||
if (!(kinds[0] === "move" && kinds[1] === "left-down" && kinds.at(-1) === "left-up" && kinds.includes("move", 2)))
|
||||
fail("drag " + kinds.join(","));
|
||||
if (Math.abs(p.sent[0].fx - 100 / 640) > 1e-9) fail("drag starts where the finger went down");
|
||||
if (p.ctrl.held.size) fail("drag left the button held");
|
||||
}
|
||||
// Press and hold is a right-click where the finger is.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 64, 36));
|
||||
await tick(620);
|
||||
p.ctrlTouchUp(p.at(1, 64, 36)); await tick(10);
|
||||
const buttons = p.sent.filter(e => "button" in e).map(e => `${e.button}-${e.down}`);
|
||||
if (buttons.join() !== "right-true,right-false") fail("hold " + buttons.join());
|
||||
if (!(p.sent[0].fx === 0.1)) fail("hold position " + JSON.stringify(p.sent[0]));
|
||||
}
|
||||
// Two fingers scroll, and the content follows them (fingers up scrolls down).
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 200, 200));
|
||||
p.ctrlTouchMove(p.at(1, 100, 180)); p.ctrlTouchMove(p.at(2, 200, 180));
|
||||
p.ctrlTouchUp(p.at(1, 100, 180)); p.ctrlTouchUp(p.at(2, 200, 180));
|
||||
await tick(10);
|
||||
const dy = p.sent.filter(e => e.scroll).reduce((a, e) => a + e.scroll[1], 0);
|
||||
if (!(dy === 40 && !p.sent.some(e => "button" in e))) fail("scroll " + JSON.stringify(p.sent));
|
||||
}
|
||||
// On the headset view it's a trackpad: drags move the pointer, taps click where it is.
|
||||
{
|
||||
const p = page({ mode: "rel" });
|
||||
p.ctrlTouchDown(p.at(1, 100, 100));
|
||||
p.ctrlTouchMove(p.at(1, 110, 100)); p.ctrlTouchMove(p.at(1, 120, 105));
|
||||
p.ctrlTouchUp(p.at(1, 120, 105)); await tick(10);
|
||||
p.ctrlTouchDown(p.at(1, 50, 50)); p.ctrlTouchUp(p.at(1, 50, 50)); await tick(10);
|
||||
const moved = p.sent.filter(e => "dx" in e).reduce((a, e) => [a[0] + e.dx, a[1] + e.dy], [0, 0]);
|
||||
if (!(moved[0] === 32 && moved[1] === 8)) fail("trackpad move " + JSON.stringify(moved));
|
||||
if (p.sent.some(e => "fx" in e)) fail("trackpad sent an absolute position");
|
||||
if (p.sent.filter(e => e.button === "left").length !== 2) fail("trackpad tap " + JSON.stringify(p.sent));
|
||||
}
|
||||
// Text: plain ASCII only, in chunks the server takes.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlText("héllo " + "x".repeat(600));
|
||||
await tick(10);
|
||||
const text = p.sent.map(e => e.text).join("");
|
||||
if (!(text === "hllo " + "x".repeat(600) && p.sent.every(e => e.text.length <= 500))) fail("text " + text.length);
|
||||
}
|
||||
// Not connected yet: clicks and keys wait with their position. The request fails: only releases wait.
|
||||
{
|
||||
let calls = 0;
|
||||
const p = page({ api: async () => { calls++; if (calls === 1) return { state: "starting", sent: false }; throw new Error("offline"); } });
|
||||
p.ctrlSend([{ fx: 0.1, fy: 0.1, window: 42 }, { button: "left", down: true }, { key: 30, down: true }]);
|
||||
await tick(5);
|
||||
const q = p.ctrl.queue;
|
||||
if (!(q.length === 3 && "fx" in q[0] && q[1].button === "left" && q[2].key === 30)) fail("kept while starting " + JSON.stringify(q));
|
||||
clearTimeout(p.ctrl.retry);
|
||||
p.ctrl.queue = []; p.ctrl.retryAt = 0;
|
||||
p.ctrlSend([{ button: "left", down: false }, { key: 31, down: true }, { dx: 3, dy: 1 }]);
|
||||
await tick(5);
|
||||
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].button === "left" && p.ctrl.queue[0].down === false))
|
||||
fail("release kept on failure " + JSON.stringify(p.ctrl.queue));
|
||||
if (calls !== 2) fail("retried in a tight loop: " + calls + " requests");
|
||||
clearTimeout(p.ctrl.retry);
|
||||
}
|
||||
// Turning Control off (or the view losing focus) lets go of anything held.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlButton("left", true);
|
||||
p.ctrlRelease(); await tick(10);
|
||||
if (!(p.sent.at(-1).button === "left" && p.sent.at(-1).down === false && !p.ctrl.held.size)) fail("release " + JSON.stringify(p.sent));
|
||||
}
|
||||
// Keys held on the Frame are let go too.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlKeyEvent(42, true);
|
||||
p.ctrlRelease(); await tick(10);
|
||||
if (!(p.sent.at(-1).key === 42 && p.sent.at(-1).down === false && !p.ctrl.keys.size)) fail("key release " + JSON.stringify(p.sent));
|
||||
}
|
||||
// A long queue sheds moves and old scrolls, never a release.
|
||||
{
|
||||
const p = page({ api: () => new Promise(() => {}) }); // stuck request
|
||||
p.ctrlSend([{ dx: 1, dy: 1 }]);
|
||||
p.ctrlSend([{ button: "left", down: false }]);
|
||||
for (let i = 0; i < 320; i++) p.ctrlSend([{ scroll: [0, 1] }]);
|
||||
if (!p.ctrl.queue.some(e => e.button === "left" && e.down === false)) fail("trim dropped a release");
|
||||
if (p.ctrl.queue.length > 300) fail("trim kept " + p.ctrl.queue.length);
|
||||
}
|
||||
// Broken on the Frame side: no hammering, and only releases wait.
|
||||
{
|
||||
let calls = 0;
|
||||
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
|
||||
p.ctrlSend([{ button: "left", down: true }, { button: "left", down: false }]);
|
||||
await tick(50);
|
||||
if (calls !== 1) fail("error response reposted " + calls + " times");
|
||||
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].down === false)) fail("error kept " + JSON.stringify(p.ctrl.queue));
|
||||
clearTimeout(p.ctrl.retry);
|
||||
}
|
||||
// Connecting: a tap keeps its position, so it lands where it was made.
|
||||
{
|
||||
const p = page({ api: async () => ({ state: "starting", sent: false }) });
|
||||
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
|
||||
await tick(10);
|
||||
const q = p.ctrl.queue;
|
||||
if (!("fx" in q[0] && q[1].button === "left")) fail("connecting tap " + JSON.stringify(q));
|
||||
clearTimeout(p.ctrl.retry);
|
||||
}
|
||||
// Lifting one of two scrolling fingers doesn't jump the scroll or start a drag.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 300, 200));
|
||||
p.ctrlTouchUp(p.at(1, 100, 200));
|
||||
p.ctrlTouchMove(p.at(2, 300, 199));
|
||||
p.ctrlTouchUp(p.at(2, 300, 199)); await tick(10);
|
||||
if (p.sent.length) fail("one finger left after scrolling " + JSON.stringify(p.sent));
|
||||
}
|
||||
// A cancelled touch isn't a tap.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 100, 100)); p.ctrlTouchCancel(p.at(1, 100, 100)); await tick(10);
|
||||
if (p.sent.length) fail("cancel clicked " + JSON.stringify(p.sent));
|
||||
}
|
||||
// Press and hold on the bars around the picture does nothing.
|
||||
{
|
||||
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } });
|
||||
p.ctrlTouchDown(p.at(1, 320, 10)); await tick(620); p.ctrlTouchUp(p.at(1, 320, 10)); await tick(10);
|
||||
if (p.sent.length) fail("hold on the bars " + JSON.stringify(p.sent));
|
||||
}
|
||||
// Taps only reach the panel in use, and only once its picture is the one on screen.
|
||||
{
|
||||
const fns = ["panelKey", "deskPanel", "ctrlTarget"].map(grab).join("");
|
||||
const check = (desk, live = true) => new Function("desk", "live", "ctrlMode", fns + "; return ctrlTarget();")(desk, live, () => "abs");
|
||||
const a = { display: ":1", window: 5 }, b = { display: ":0", window: 5 };
|
||||
const base = { panels: [a, b], loaded: true, pick: "", focus: ":1/5", shown: ":1/5" };
|
||||
if (!check(base).ok) fail("target: shown and in use");
|
||||
if (check({ ...base, shown: ":0/5" }).ok) fail("target: the picture is another panel with the same id");
|
||||
if (!/Capture or Live/.test(check({ ...base, shown: null }, false).why)) fail("target: stale picture message");
|
||||
if (check({ ...base, pick: ":0/5", shown: ":0/5" }).ok) fail("target: a watched panel that isn't in use");
|
||||
if (check({ ...base, focus: null }).ok) fail("target: nothing in use");
|
||||
}
|
||||
// Focus moves to another panel mid-gesture: the tap or hold does nothing.
|
||||
{
|
||||
const p = page();
|
||||
p.ctrlTouchDown(p.at(1, 100, 100));
|
||||
p.target.panel = { window: 43, display: ":1" };
|
||||
p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
|
||||
p.target.panel = { window: 42, display: ":1" };
|
||||
p.ctrlTouchDown(p.at(1, 100, 100));
|
||||
p.target.panel = null;
|
||||
await tick(620); p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
|
||||
if (p.sent.some(e => "button" in e)) fail("gesture outlived its panel " + JSON.stringify(p.sent));
|
||||
}
|
||||
// Trimming keeps a click together with its position.
|
||||
{
|
||||
const p = page({ api: () => new Promise(() => {}) });
|
||||
p.ctrlSend([{ dx: 1, dy: 0 }]); // in flight forever
|
||||
for (let i = 0; i < 100; i++) p.ctrlSend([{ scroll: [0, 1] }]);
|
||||
p.ctrlSend([{ fx: 0.5, fy: 0.5, window: 42, display: ":1" }, { button: "left", down: true }, { button: "left", down: false }]);
|
||||
for (let i = 0; i < 198; i++) p.ctrlSend([{ scroll: [0, 1] }]);
|
||||
const q = p.ctrl.queue, i = q.findIndex(e => e.button === "left" && e.down);
|
||||
if (i < 1 || !("fx" in q[i - 1])) fail("trim split a click from its position");
|
||||
}
|
||||
// Backing off holds for new input too.
|
||||
{
|
||||
let calls = 0;
|
||||
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
|
||||
p.ctrlSend([{ button: "left", down: false }]);
|
||||
await tick(5);
|
||||
for (let i = 0; i < 10; i++) { p.ctrlSend([{ key: 30, down: false }]); await tick(2); }
|
||||
if (calls !== 1) fail("new input bypassed the backoff: " + calls + " requests");
|
||||
clearTimeout(p.ctrl.retry);
|
||||
}
|
||||
// A long paste goes in several requests, so a release never waits behind all of it.
|
||||
{
|
||||
const batches = [];
|
||||
const p = page({ api: async (path, body) => { batches.push(body.events); return { state: "ready", sent: true }; } });
|
||||
p.ctrlText("y".repeat(450));
|
||||
p.ctrlButton("left", false);
|
||||
await tick(30);
|
||||
if (!batches.every(b => b.reduce((a, e) => a + (e.text?.length || 0), 0) <= 100)) fail("a batch carried too much text");
|
||||
if (batches.length < 5) fail("paste went in " + batches.length + " requests");
|
||||
}
|
||||
console.log("control gestures ok");
|
||||
@@ -0,0 +1,312 @@
|
||||
"""Owned media planning, eye isolation, decoder choice and fake-Frame ownership."""
|
||||
import argparse
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
|
||||
import frame_media as media
|
||||
import frame_media_player as player
|
||||
import frame_media_remote as remote
|
||||
import frame_splat as splat
|
||||
import server
|
||||
|
||||
|
||||
def stop_now():
|
||||
"""What systemd's SIGTERM does to the player, without signalling the test process."""
|
||||
handler = signal.getsignal(signal.SIGTERM)
|
||||
if callable(handler):
|
||||
handler(signal.SIGTERM, None)
|
||||
|
||||
|
||||
class Media(unittest.TestCase):
|
||||
def test_layout_evidence_and_override(self):
|
||||
for name, layout in [('film_SBS.mp4', 'sbs'), ('film.OU.mkv', 'ou'),
|
||||
('film_FSBS.mp4', 'full-sbs'), ('photo_TB.png', 'ou')]:
|
||||
self.assertEqual(media.plan(name)['layout'], layout)
|
||||
self.assertEqual(media.plan('film_SBS_OU.mp4', 'mono')['source'], 'explicit')
|
||||
self.assertEqual(media.plan('film.mkv', metadata={'stereo_mode': 'top_bottom'})['layout'], 'full-ou')
|
||||
for name in ('film.mp4', 'film_SBS_OU.mp4', 'businessbs.mp4'):
|
||||
with self.assertRaises(ValueError):
|
||||
media.plan(name)
|
||||
with self.assertRaises(ValueError):
|
||||
media.plan('film.mkv', metadata={'stereo_mode': 'right_left'})
|
||||
|
||||
def test_unsupported_containers_do_not_flatten_spatial_photos(self):
|
||||
for name in ('spatial.HEIC', 'stereo.mpo', 'cloud.ply', 'cloud.spz', 'app.exe'):
|
||||
with self.assertRaises(ValueError):
|
||||
media.plan(name, 'sbs')
|
||||
|
||||
def test_ou_pixels_keep_each_eye_and_row(self):
|
||||
a, b, c, d = [bytes([n])*8 for n in (1, 2, 3, 4)]
|
||||
data, width, height = media.stereo_pixels(a+b+c+d, 2, 4, 'ou')
|
||||
self.assertEqual((data, width, height), (a+c+b+d, 4, 2))
|
||||
with self.assertRaises(ValueError):
|
||||
media.stereo_pixels(b'bad', 2, 4, 'sbs')
|
||||
self.assertEqual(media.geometry(3840, 2160, 'sbs'), (1920, 1080, 2))
|
||||
self.assertEqual(media.geometry(1920, 1080, 'ou'), (1920, 1080, .5))
|
||||
|
||||
def test_decode_is_hardware_and_one_clock_for_audio(self):
|
||||
for codec, decoder in [('h264', 'h264_v4l2m2m'), ('hevc', 'hevc_v4l2m2m')]:
|
||||
cmd = player.decoder_command(Path('/tmp/a file.mp4'), {'codec_name': codec}, 1280, 720, True)
|
||||
self.assertIn(decoder, cmd)
|
||||
self.assertIn('-re', cmd)
|
||||
self.assertIn('pulse', cmd)
|
||||
self.assertIn(str(Path('/tmp/a file.mp4')), cmd)
|
||||
with self.assertRaises(ValueError):
|
||||
player.decoder_command(Path('x.webm'), {'codec_name': 'vp9'}, 640, 480, False)
|
||||
cmd = player.decoder_command(Path('x.png'), {'codec_name': 'png'}, 640, 480, False, True)
|
||||
self.assertNotIn('-re', cmd)
|
||||
self.assertIn('-frames:v', cmd)
|
||||
|
||||
def play_with(self, name, busy=0, on_pixels=None, sleeps=None, info=None,
|
||||
fail=None, layout='auto'):
|
||||
"""Run the player against a fake OpenVR; returns (status, pixels calls).
|
||||
|
||||
Handle 0 is the theatre surround and 1 the screen. `fail(handle, n)` makes
|
||||
the n-th upload busy; every status written is kept in self.writes."""
|
||||
calls = []
|
||||
self.writes = []
|
||||
write_status = player.write_status
|
||||
|
||||
def record(path, **values):
|
||||
self.writes.append(values)
|
||||
write_status(path, **values)
|
||||
|
||||
class FakeOverlay:
|
||||
created = 0
|
||||
|
||||
def create(self, *a, **k):
|
||||
# Handles in creation order: surround 0, then screen 1 (theatre).
|
||||
FakeOverlay.created += 1
|
||||
return FakeOverlay.created - 1
|
||||
|
||||
def call(self, *a):
|
||||
pass
|
||||
|
||||
def pixels(self, handle, data, w, h):
|
||||
calls.append((handle, w, h))
|
||||
if on_pixels:
|
||||
on_pixels(len(calls))
|
||||
if len(calls) <= busy or (fail and fail(handle, len(calls))):
|
||||
raise player.OverlayBusy('standby')
|
||||
|
||||
def close(self):
|
||||
# A Stop landing during cleanup must be ignored, not become an error.
|
||||
# Call the installed handler directly: a real SIGTERM kills Windows.
|
||||
stop_now()
|
||||
|
||||
frame = bytes(4*2*4)
|
||||
proc = unittest.mock.MagicMock()
|
||||
proc.stdout = io.BytesIO(frame*4)
|
||||
proc.wait.return_value = 0
|
||||
proc.poll.return_value = 0
|
||||
old = signal.getsignal(signal.SIGTERM), signal.getsignal(signal.SIGINT)
|
||||
with tempfile.TemporaryDirectory() as d, \
|
||||
patch.object(player, 'Overlay', FakeOverlay), \
|
||||
patch.object(player, 'probe', return_value=(info or {'codec_name': 'h264', 'width': 4, 'height': 2}, False)), \
|
||||
patch.object(player.subprocess, 'Popen', return_value=proc), \
|
||||
patch.object(player, 'write_status', side_effect=record), \
|
||||
patch.object(player.frame_splat, 'render', return_value=(bytes(4*4*2), 4, 2)), \
|
||||
patch.object(player.time, 'sleep', side_effect=sleeps):
|
||||
path = Path(d)/name
|
||||
path.write_bytes(b'x')
|
||||
status = Path(d)/'status.json'
|
||||
try:
|
||||
player.play(argparse.Namespace(file=str(path), layout=layout, theatre=True, status=str(status)))
|
||||
finally:
|
||||
signal.signal(signal.SIGTERM, old[0])
|
||||
signal.signal(signal.SIGINT, old[1])
|
||||
return json.loads(status.read_text()), calls
|
||||
|
||||
def test_video_survives_standby_and_stop_after_end_stays_ended(self):
|
||||
# Verified 2026-09-29: an unworn Frame enters standby within seconds and
|
||||
# SetOverlayRaw then returns RequestFailed (23) until it wakes.
|
||||
result, calls = self.play_with('clip_SBS.mp4', busy=3)
|
||||
self.assertEqual((result['state'], result['frames']), ('ended', 4))
|
||||
# Two video frames were dropped; the surround (handle 0) waited and was re-sent.
|
||||
self.assertEqual(result['dropped'], 2)
|
||||
self.assertIn((0, 1, 1), calls[3:])
|
||||
|
||||
def test_theatre_surround_failure_does_not_stop_playback(self):
|
||||
def fail_surround(n):
|
||||
if n == 1: # the surround's upload is the first pixels call
|
||||
raise RuntimeError('OpenVR SetOverlayRaw failed: 11')
|
||||
result, _ = self.play_with('clip_SBS.mp4', on_pixels=fail_surround)
|
||||
self.assertEqual((result['state'], result['frames']), ('ended', 4))
|
||||
|
||||
def test_stop_mid_video_reports_stopped(self):
|
||||
result, _ = self.play_with('clip_SBS.mp4', on_pixels=lambda n: n == 3 and stop_now())
|
||||
self.assertEqual(result['state'], 'stopped')
|
||||
|
||||
def test_video_errors_when_steamvr_never_takes_frames(self):
|
||||
with patch.object(player, 'BUSY_LIMIT', -1), \
|
||||
self.assertRaisesRegex(RuntimeError, 'stopped accepting frames'):
|
||||
self.play_with('clip_SBS.mp4', busy=99)
|
||||
|
||||
def test_still_waits_out_standby_without_a_limit(self):
|
||||
# Stills have no timeline: keep retrying (here past BUSY_LIMIT) until shown.
|
||||
ticks = iter(range(10))
|
||||
def sleep(_):
|
||||
if next(ticks) == 8:
|
||||
stop_now()
|
||||
with patch.object(player, 'BUSY_LIMIT', -1):
|
||||
result, calls = self.play_with('photo_SBS.png', busy=5, sleeps=sleep,
|
||||
info={'codec_name': 'png', 'width': 4, 'height': 2})
|
||||
self.assertEqual(result['state'], 'stopped')
|
||||
screen = [c for c in calls if c[0] == 1]
|
||||
self.assertGreater(len(screen), 1) # retried through standby
|
||||
self.assertEqual(calls[-1], (0, 1, 1)) # surround drained once the screen took a frame
|
||||
|
||||
def still_with_late_surround(self, name, **kw):
|
||||
# The screen takes its first frame while the surround is still refused
|
||||
# (its first upload, the drain right after the screen, and one retry).
|
||||
ticks = iter(range(10))
|
||||
def sleep(_):
|
||||
if next(ticks) == 5:
|
||||
stop_now()
|
||||
surround_tries = []
|
||||
def fail(handle, n):
|
||||
if handle == 0:
|
||||
surround_tries.append(n)
|
||||
return len(surround_tries) <= 3
|
||||
return False
|
||||
result, calls = self.play_with(name, sleeps=sleep, fail=fail, **kw)
|
||||
self.assertEqual(result['state'], 'stopped')
|
||||
screen = [c for c in calls if c[0] == 1]
|
||||
surround = [c for c in calls if c[0] == 0]
|
||||
self.assertEqual(len(screen), 1) # shown once, not re-sent every second
|
||||
self.assertEqual(len(surround), 4) # kept retrying after the screen, until it took
|
||||
self.assertEqual(calls[-1][0], 0)
|
||||
return calls
|
||||
|
||||
def test_photo_surround_recovers_after_screen_is_shown(self):
|
||||
self.still_with_late_surround('photo_SBS.png',
|
||||
info={'codec_name': 'png', 'width': 4, 'height': 2})
|
||||
|
||||
def test_splat_surround_recovers_after_screen_is_shown(self):
|
||||
self.still_with_late_surround('scene.splat')
|
||||
|
||||
def test_video_standby_limit_is_five_minutes_without_an_accepted_frame(self):
|
||||
self.assertEqual(player.BUSY_LIMIT, 300)
|
||||
def run(times, busy):
|
||||
# Upload n happens at times[n] seconds on a fake clock; 1 is the surround.
|
||||
clock = [1000.0]
|
||||
def on_pixels(n):
|
||||
clock[0] = 1000.0 + times.get(n, times[max(times)])
|
||||
with patch.object(player.time, 'monotonic', side_effect=lambda: clock[0]):
|
||||
return self.play_with('clip_SBS.mp4', on_pixels=on_pixels,
|
||||
fail=lambda h, n: h == 1 and n in busy)
|
||||
# Busy for 299 s, then a frame lands: no error.
|
||||
result, _ = run({1: 0, 2: 0, 3: 299, 4: 299, 5: 299}, busy={2, 3})
|
||||
self.assertEqual((result['state'], result['dropped']), ('ended', 2))
|
||||
# An accepted frame resets the timer: 600 s busy in total, never 300 s in a row.
|
||||
result, _ = run({1: 0, 2: 0, 3: 200, 4: 250, 5: 450}, busy={2, 3, 5})
|
||||
self.assertEqual((result['state'], result['dropped']), ('ended', 3))
|
||||
# 301 s in a row without an accepted frame is an error.
|
||||
with self.assertRaisesRegex(RuntimeError, 'stopped accepting frames for 300 s'):
|
||||
run({1: 0, 2: 0, 3: 301}, busy={2, 3, 4, 5})
|
||||
|
||||
def test_status_reports_where_the_layout_came_from(self):
|
||||
video = {'codec_name': 'h264', 'width': 4, 'height': 2}
|
||||
for name, layout, tags, expect in [
|
||||
('clip_SBS.mp4', 'auto', None, ('sbs', 'filename')),
|
||||
('clip.mkv', 'auto', {'stereo_mode': 'left_right'}, ('full-sbs', 'metadata')),
|
||||
('clip_OU.mp4', 'sbs', None, ('sbs', 'explicit')),
|
||||
('clip.mkv', 'mono', {'stereo_mode': 'left_right'}, ('mono', 'explicit'))]:
|
||||
with self.subTest(name=name, layout=layout):
|
||||
self.play_with(name, layout=layout, info=dict(video, tags=tags) if tags else video)
|
||||
playing = self.writes[0]
|
||||
self.assertEqual(playing['state'], 'playing')
|
||||
self.assertEqual((playing['layout'], playing['source']), expect)
|
||||
|
||||
def test_fake_frame_library_and_traversal(self):
|
||||
with tempfile.TemporaryDirectory() as d, patch.object(remote, 'ROOT', Path(d)):
|
||||
identity = 'a'*32+'/space and quote\'.png'
|
||||
path = Path(d)/identity
|
||||
path.parent.mkdir()
|
||||
path.write_bytes(b'test')
|
||||
self.assertEqual(remote.media_path(identity), path.resolve())
|
||||
for bad in ('../../etc/passwd', '/etc/passwd', 'a'*32+'/..', 'a'*32+'/x/y', None):
|
||||
with self.assertRaises((ValueError, FileNotFoundError)):
|
||||
remote.media_path(bad)
|
||||
link = path.parent/'link.png'
|
||||
link.symlink_to(path)
|
||||
with self.assertRaises(ValueError):
|
||||
remote.media_path('a'*32+'/link.png')
|
||||
with patch.object(remote, 'status', return_value={'state': 'idle'}):
|
||||
files = remote.run({'action': 'list'})['files']
|
||||
self.assertEqual([f['id'] for f in files], [identity])
|
||||
|
||||
def test_server_rejects_bad_actions_before_ssh(self):
|
||||
with patch.object(server, 'ssh') as ssh:
|
||||
for body in ({'action': 'delete'}, {'action': 'play', 'id': '../x'},
|
||||
{'action': 'play', 'id': 'a'*32+'/x', 'layout': 'invalid'},
|
||||
{'action': 'play', 'id': 'a'*32+'/x', 'theatre': 'false'}):
|
||||
with self.assertRaises(server.Failure):
|
||||
server.media(body)
|
||||
ssh.assert_not_called()
|
||||
|
||||
def test_fake_frame_stop_only_owns_our_unit(self):
|
||||
for rc in (0, 5): # 5: already collected ("not loaded"), a no-op
|
||||
with patch.object(remote.subprocess, 'run') as run, patch.object(remote, 'status', return_value={'state': 'ended'}):
|
||||
run.return_value.returncode = rc
|
||||
self.assertEqual(remote.run({'action': 'stop'})['state'], 'ended')
|
||||
self.assertEqual(run.call_args.args[0], ['systemctl', '--user', 'stop', 'frame-control-media.service'])
|
||||
with patch.object(remote.subprocess, 'run') as run, patch.object(remote, 'status', return_value={}):
|
||||
run.return_value.returncode, run.return_value.stderr = 1, 'Access denied'
|
||||
with self.assertRaisesRegex(RuntimeError, 'Access denied'):
|
||||
remote.run({'action': 'stop'})
|
||||
|
||||
def test_start_failure_reports_systemd_error(self):
|
||||
with tempfile.TemporaryDirectory() as d, patch.object(remote, 'ROOT', Path(d)), \
|
||||
patch.object(remote, 'STATUS', Path(d)/'status.json'), \
|
||||
patch.object(remote, 'active', return_value=False), \
|
||||
patch.object(remote.subprocess, 'run') as run:
|
||||
identity = 'b'*32+'/still_SBS.png'
|
||||
(Path(d)/identity).parent.mkdir()
|
||||
(Path(d)/identity).write_bytes(b'x')
|
||||
run.return_value.returncode, run.return_value.stderr = 1, 'Unit already exists'
|
||||
with patch.object(remote, 'probe', return_value=({}, False)), \
|
||||
self.assertRaisesRegex(RuntimeError, 'Unit already exists'):
|
||||
remote.run({'action': 'play', 'id': identity})
|
||||
self.assertEqual(run.call_args.args[0][0], 'systemd-run') # reset-failed's result is ignored
|
||||
|
||||
def test_upload_rejects_unplayable_names_and_keeps_copy_error(self):
|
||||
with patch.object(server, 'ssh') as ssh, patch.object(server, 'push_file') as push:
|
||||
# On Windows a backslash is a separator, so such a name can't reach here.
|
||||
for name in ('.hidden.mp4',) + (('a\\b_SBS.mp4',) if os.sep == '/' else ()):
|
||||
with self.assertRaises(server.Failure):
|
||||
server.push_media(Path('/tmp')/name)
|
||||
ssh.assert_not_called()
|
||||
push.side_effect = server.Failure('copy failed')
|
||||
ssh.side_effect = [None, server.Failure('link down')]
|
||||
with self.assertRaisesRegex(server.Failure, 'copy failed'):
|
||||
server.push_media(Path('/tmp/film_SBS.mp4'))
|
||||
|
||||
def test_splat_invalid_records_and_stereo_parallax(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d)/'small.splat'
|
||||
path.write_bytes(struct.pack('<6f8B', 0, 0, 0, .001, .001, .001,
|
||||
255, 0, 0, 255, 255, 128, 128, 128))
|
||||
data, w, h = splat.render(path, 64, 48)
|
||||
self.assertEqual((len(data), w, h), (128*48*4, 128, 48))
|
||||
def centroid(eye):
|
||||
weights = [(x, data[(y*w+x+eye*64)*4]) for y in range(h) for x in range(64)]
|
||||
return sum(x*v for x,v in weights)/sum(v for x,v in weights)
|
||||
self.assertGreater(centroid(0), centroid(1))
|
||||
for bad in (b'', b'bad', struct.pack('<6f8B', float('nan'), 0, 0, 1, 1, 1, *([128]*8))):
|
||||
path.write_bytes(bad)
|
||||
with self.assertRaises(ValueError):
|
||||
splat.read(path)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,283 @@
|
||||
"""Touch: the live view's Control (ui/frame_touch.py on the Frame, and the server's checks).
|
||||
|
||||
Run: python3 -m unittest discover -s tests
|
||||
"""
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT / "ui"))
|
||||
|
||||
|
||||
class Mapping(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
import frame_touch
|
||||
cls.t = frame_touch
|
||||
|
||||
def test_panel_bigger_than_its_display_is_scaled(self):
|
||||
# Verified 2026-09-29: a 1920x1080 window on :1 (1280x720) took Accept at
|
||||
# window (1828, 1020) as pointer (1219, 680).
|
||||
panel = {"root": [1280, 720], "width": 1920, "height": 1080}
|
||||
x, y = self.t.to_root(panel, 1828 / 1920, 1020 / 1080)
|
||||
self.assertAlmostEqual(x, 1218.7, delta=1)
|
||||
self.assertAlmostEqual(y, 679.4, delta=1)
|
||||
|
||||
def test_same_size_is_one_to_one_and_clamped(self):
|
||||
panel = {"root": [1280, 720], "width": 1280, "height": 720}
|
||||
self.assertEqual(self.t.to_root(panel, 0, 0), (0, 0))
|
||||
self.assertEqual(self.t.to_root(panel, 1, 1), (1279, 719))
|
||||
self.assertEqual(self.t.to_root(panel, -3, 7), (0, 719))
|
||||
|
||||
def test_other_shapes_are_letterboxed(self):
|
||||
panel = {"root": [1280, 720], "width": 800, "height": 800} # square: bars left and right
|
||||
x, y = self.t.to_root(panel, 0, 0.5)
|
||||
self.assertAlmostEqual(x, 280, delta=0.5)
|
||||
self.assertAlmostEqual(y, 359.5, delta=0.5)
|
||||
|
||||
def test_same_id_on_both_displays_is_told_apart_by_pid(self):
|
||||
t = self.t
|
||||
saved = t.displays, t.window_info, t.window_pid
|
||||
self.addCleanup(lambda: (setattr(t, "displays", saved[0]), setattr(t, "window_info", saved[1]),
|
||||
setattr(t, "window_pid", saved[2])))
|
||||
t.displays = lambda: [":0", ":1"]
|
||||
t.window_info = lambda d, w: {"name": f"on {d}", "width": 1280 if w != "root" else 1920, "height": 720}
|
||||
t.window_pid = lambda d, w: {":0": 111, ":1": 222}[d]
|
||||
self.assertEqual(t.locate(5, 222)["display"], ":1")
|
||||
self.assertEqual(t.locate(5, 111)["display"], ":0")
|
||||
self.assertEqual(t.locate(5, None)["display"], ":0")
|
||||
|
||||
def test_focus_display_is_decoded(self):
|
||||
t = self.t
|
||||
saved = t.xprop_root
|
||||
self.addCleanup(lambda: setattr(t, "xprop_root", saved))
|
||||
for values, want in (([12602, 0, 58], ":1"), ([12346], ":0"), ([], None), ([0x41], None)):
|
||||
t.xprop_root = lambda d, n, v=values: v
|
||||
self.assertEqual(t.focus_display(), want)
|
||||
|
||||
def test_ascii_table_covers_printable_characters(self):
|
||||
for code in range(0x20, 0x7F):
|
||||
self.assertIn(chr(code), self.t.ASCII, chr(code))
|
||||
self.assertEqual(self.t.ASCII["a"], (30, False))
|
||||
self.assertEqual(self.t.ASCII["A"], (30, True))
|
||||
self.assertEqual(self.t.ASCII["?"], (53, True))
|
||||
self.assertEqual(self.t.ASCII["1"], (2, False))
|
||||
self.assertEqual(self.t.ASCII["0"], (11, False))
|
||||
|
||||
def test_events_parsing(self):
|
||||
self.assertEqual(self.t.events(b'{"dx": 1}'), [{"dx": 1}])
|
||||
self.assertEqual(self.t.events(b'[{"dx": 1}, 5]'), [{"dx": 1}])
|
||||
self.assertEqual(self.t.events(b"nope"), [])
|
||||
|
||||
|
||||
class FakeGamescope:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def __getattr__(self, name):
|
||||
return lambda *a: self.calls.append((name, *a))
|
||||
|
||||
|
||||
class Apply(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
import frame_touch
|
||||
cls.t = frame_touch
|
||||
|
||||
def setUp(self):
|
||||
self.focused = {"window": 7, "display": ":1", "root": [1280, 720], "width": 1280, "height": 720, "name": "x"}
|
||||
saved = self.t.focus, self.t.say, self.t.focus_now
|
||||
self.t.STALE[0] = False
|
||||
self.said = []
|
||||
self.t.focus = lambda: dict(self.focused)
|
||||
self.t.focus_now = lambda: (self.focused["window"], self.focused["display"])
|
||||
self.t.say = lambda state, **more: self.said.append((state, more))
|
||||
self.addCleanup(lambda: (setattr(self.t, "focus", saved[0]), setattr(self.t, "say", saved[1]),
|
||||
setattr(self.t, "focus_now", saved[2])))
|
||||
|
||||
def test_tap_moves_then_clicks_in_order(self):
|
||||
gs, panel = FakeGamescope(), None
|
||||
for e in ({"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, {"button": "left", "down": True, "window": 7, "display": ":1"},
|
||||
{"button": "left", "down": False}):
|
||||
panel = self.t.apply(gs, e, panel)
|
||||
self.assertEqual([c[0] for c in gs.calls], ["move_to", "button", "button"])
|
||||
self.assertEqual(gs.calls[1][1:], ("left", True))
|
||||
|
||||
def test_a_tap_meant_for_another_panel_goes_nowhere(self):
|
||||
# Focus moved on: the position and the press are dropped; the release still goes.
|
||||
gs, panel = FakeGamescope(), None
|
||||
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"button": "left", "down": True, "window": 99, "display": ":1"},
|
||||
{"key": 30, "down": True, "window": 7, "display": ":0"}, {"button": "left", "down": False, "window": 99, "display": ":1"}):
|
||||
panel = self.t.apply(gs, e, panel)
|
||||
self.assertEqual(gs.calls, [("button", "left", False)])
|
||||
self.assertEqual(self.said[0], ("ready", {"focus": 7, "display": ":1", "stale": True}))
|
||||
|
||||
def test_presses_read_focus_afresh(self):
|
||||
# A move may use a recent reading; a press checks again, so a panel that just
|
||||
# took focus doesn't get a click meant for another.
|
||||
gs, calls = FakeGamescope(), []
|
||||
self.t.focus = lambda: calls.append(1) or dict(self.focused)
|
||||
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, None)
|
||||
panel = self.t.apply(gs, {"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}, panel)
|
||||
self.assertEqual(len(calls), 1)
|
||||
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
|
||||
self.assertEqual(len(calls), 1) # same panel still: the quick check was enough
|
||||
self.focused["window"] = 8
|
||||
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
|
||||
self.assertEqual(len(calls), 2)
|
||||
self.assertEqual([c[0] for c in gs.calls], ["move_to", "move_to", "button"])
|
||||
|
||||
def test_stale_is_said_once_and_cleared(self):
|
||||
gs, panel = FakeGamescope(), None
|
||||
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"},
|
||||
{"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}):
|
||||
panel = self.t.apply(gs, e, panel)
|
||||
self.assertEqual(self.said, [("ready", {"focus": 7, "display": ":1", "stale": True}),
|
||||
("ready", {"focus": 7, "display": ":1"})])
|
||||
|
||||
def test_stale_clears_on_a_trackpad_move_but_not_on_a_stale_release(self):
|
||||
gs = FakeGamescope()
|
||||
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, None)
|
||||
panel = self.t.apply(gs, {"button": "left", "down": False, "window": 99, "display": ":1"}, panel)
|
||||
self.assertTrue(self.t.STALE[0]) # that release was still aimed at the old panel
|
||||
self.t.apply(gs, {"dx": 3, "dy": 0}, panel)
|
||||
self.assertFalse(self.t.STALE[0])
|
||||
self.assertEqual(self.said[-1][1].get("stale"), None)
|
||||
|
||||
def test_same_window_id_on_the_other_display_is_another_panel(self):
|
||||
gs = FakeGamescope()
|
||||
self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)
|
||||
self.assertEqual(gs.calls, [])
|
||||
|
||||
def test_relative_scroll_keys_text(self):
|
||||
gs = FakeGamescope()
|
||||
for e in ({"dx": 5, "dy": -3}, {"scroll": [0, 120]}, {"key": 30, "down": True}, {"text": "hi"},
|
||||
{"key": True}, {"button": "sideways"}):
|
||||
self.t.apply(gs, e, None)
|
||||
self.assertEqual([c[0] for c in gs.calls], ["move_by", "scroll", "key", "text"])
|
||||
|
||||
def test_everything_held_is_let_go(self):
|
||||
class Held(self.t.Gamescope):
|
||||
def __init__(self):
|
||||
self.held, self.keys, self.log = set(), set(), []
|
||||
|
||||
def frame(self):
|
||||
pass
|
||||
|
||||
g = Held()
|
||||
g.L = type("L", (), {"ei_device_button_button": lambda *a: g.log.append(("button",) + a[2:]),
|
||||
"ei_device_keyboard_key": lambda *a: g.log.append(("key",) + a[2:])})()
|
||||
g.device = object()
|
||||
g.button("left", True)
|
||||
g.key(42, True)
|
||||
g.release_all()
|
||||
self.assertEqual(g.log[-2:], [("button", 0x110, False), ("key", 42, False)])
|
||||
self.assertEqual((g.held, g.keys), (set(), set()))
|
||||
|
||||
def test_resuming_after_a_pause_lets_go_of_everything(self):
|
||||
t = self.t
|
||||
log, queue = [], [1]
|
||||
|
||||
class L:
|
||||
def __getattr__(self, name):
|
||||
if name == "ei_get_event":
|
||||
return lambda ei: queue.pop(0) if queue else None
|
||||
return {"ei_event_get_type": lambda ev: t.EV_DEVICE_RESUMED, "ei_event_get_device": lambda ev: "dev",
|
||||
"ei_device_has_capability": lambda d, c: True,
|
||||
"ei_device_button_button": lambda d, c, down: log.append(("button", c, down)),
|
||||
"ei_device_keyboard_key": lambda d, c, down: log.append(("key", c, down))}.get(name, lambda *a: 0)
|
||||
|
||||
g = t.Gamescope.__new__(t.Gamescope)
|
||||
g.L, g.ei, g.fd, g.device, g.sequence, g.alive = L(), None, None, None, 0, True
|
||||
g.held, g.keys = {0x110}, {42}
|
||||
g.frame = lambda: None
|
||||
old = t.select.select
|
||||
t.select.select = lambda *a: ([], [], [])
|
||||
try:
|
||||
g.pump()
|
||||
finally:
|
||||
t.select.select = old
|
||||
self.assertEqual(sorted(log), [("button", 0x110, False), ("key", 42, False)])
|
||||
self.assertEqual(g.device, "dev")
|
||||
|
||||
def test_text_uses_shift_for_capitals(self):
|
||||
class Keys(self.t.Gamescope):
|
||||
def __init__(self):
|
||||
self.pressed = []
|
||||
|
||||
def key(self, code, down):
|
||||
self.pressed.append((code, down))
|
||||
k = Keys()
|
||||
k.text("Hié") # the é has no key on a US layout and is left out
|
||||
self.assertEqual(k.pressed, [(42, True), (35, True), (35, False), (42, False), (23, True), (23, False)])
|
||||
|
||||
|
||||
class ServerChecks(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
import server
|
||||
cls.s = server
|
||||
|
||||
def test_touch_event_keeps_known_fields(self):
|
||||
ev = self.s.touch_event
|
||||
self.assertEqual(ev({"fx": 0.5, "fy": 2, "window": 5, "display": ":1"}), {"fx": 0.5, "fy": 1.0, "window": 5, "display": ":1"})
|
||||
self.assertEqual(ev({"button": "left", "window": 5, "display": ":0"}), {"button": "left", "down": True, "window": 5, "display": ":0"})
|
||||
self.assertEqual(ev({"button": "right"}), {"button": "right", "down": True})
|
||||
self.assertEqual(ev({"key": 30, "down": False}), {"key": 30, "down": False})
|
||||
self.assertEqual(ev({"scroll": [0, 1e9]}), {"scroll": [0.0, 5000.0]})
|
||||
self.assertEqual(ev({"dx": 3, "other": 1}), {"dx": 3.0})
|
||||
|
||||
def test_touch_event_rejects_bad_ones(self):
|
||||
for bad in (None, {}, {"fx": 0.5, "fy": 0.5}, {"fx": "1", "fy": 0, "window": 1, "display": ":1"}, {"button": "side"},
|
||||
{"fx": 0.5, "fy": 0.5, "window": 1}, {"fx": 0.5, "fy": 0.5, "window": 1, "display": ":1;x"},
|
||||
{"window": 1, "display": ":1"},
|
||||
{"key": 0}, {"key": 999}, {"key": True}, {"scroll": [1]}, {"text": ""}, {"text": "x" * 501},
|
||||
{"fx": 0.1, "fy": 0.1, "window": True, "display": ":1"}):
|
||||
with self.assertRaises(self.s.Failure, msg=repr(bad)):
|
||||
self.s.touch_event(bad)
|
||||
|
||||
def test_panel_stream_is_the_window_and_checked(self):
|
||||
cmd = self.s.stream_command("src=panel&window=10485777&display=:1&h=720&fps=30")
|
||||
self.assertIn("DISPLAY=:1 ffmpeg", cmd)
|
||||
self.assertIn("-window_id 10485777 -i :1", cmd)
|
||||
for bad in ("src=panel&window=1;rm&display=:1", "src=panel&window=1&display=:1;x", "src=panel&display=:1"):
|
||||
with self.assertRaises(self.s.Failure):
|
||||
self.s.stream_command(bad + "&h=720&fps=30")
|
||||
|
||||
def test_touch_agent_runs_the_helper_with_nothing_to_copy(self):
|
||||
agent = self.s.TouchAgent()
|
||||
self.assertEqual(agent.deliver(lambda m: None), "")
|
||||
cmd = agent.command()
|
||||
self.assertTrue(cmd.startswith("python3 -u -c '"))
|
||||
self.assertIn("frame_touch", cmd)
|
||||
|
||||
def test_batch_limit(self):
|
||||
with self.assertRaises(self.s.Failure):
|
||||
self.s.remote_touch({"events": [{"dx": 1}] * (self.s.INPUT_BATCH_LIMIT + 1)})
|
||||
|
||||
|
||||
@unittest.skipUnless(shutil.which("node"), "needs node")
|
||||
class PageGestures(unittest.TestCase):
|
||||
"""Control's gestures and queue (tests/page/ctrl_gestures.mjs runs the real functions from index.html)."""
|
||||
|
||||
def test_gestures(self):
|
||||
r = subprocess.run(["node", str(ROOT / "tests" / "page" / "ctrl_gestures.mjs")], capture_output=True, text=True,
|
||||
timeout=60)
|
||||
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
|
||||
|
||||
|
||||
class Script(unittest.TestCase):
|
||||
def test_helper_compiles_on_the_frames_python(self):
|
||||
# The Frame runs it with its own python3 (3.13 on SteamOS 0.4.1); stdlib and ctypes only.
|
||||
src = (ROOT / "ui/frame_touch.py").read_text()
|
||||
compile(src, "frame_touch.py", "exec")
|
||||
for mod in ("import ctypes", "import json", "import select"):
|
||||
self.assertIn(mod, src)
|
||||
self.assertNotIn("import requests", src)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Media planning shared by Frame Control and its own Frame-side player.
|
||||
|
||||
No viewer dependencies. Filename hints are suggestions, never guesses from
|
||||
resolution. Explicit layout wins; conflicting hints require a choice.
|
||||
"""
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
LAYOUTS = ('auto', 'mono', 'sbs', 'ou', 'full-sbs', 'full-ou')
|
||||
VIDEO = {'.mp4', '.mkv', '.mov', '.webm', '.m4v'}
|
||||
PHOTO = {'.png', '.jpg', '.jpeg'}
|
||||
|
||||
|
||||
def plan(name, layout='auto', metadata=None):
|
||||
if layout not in LAYOUTS:
|
||||
raise ValueError('Choose auto, mono, sbs, ou, full-sbs or full-ou')
|
||||
suffix = Path(name).suffix.lower()
|
||||
if suffix in {'.heic', '.heif', '.avif', '.mpo'}:
|
||||
raise ValueError('Native spatial-photo containers are not supported yet; export both eyes as SBS or OU PNG/JPEG')
|
||||
if suffix == '.splat':
|
||||
return {'kind': 'splat', 'layout': 'sbs', 'source': 'renderer'}
|
||||
if suffix not in VIDEO | PHOTO:
|
||||
raise ValueError('Use MP4/MKV/MOV/WebM video, PNG/JPEG stereo photos, or a .splat file')
|
||||
source = 'explicit'
|
||||
if layout == 'auto':
|
||||
tokens = set(re.split(r'[^a-z0-9]+', Path(name).stem.lower()))
|
||||
hints = set()
|
||||
for value, tags in [('full-sbs', {'fsbs'}), ('full-ou', {'fou', 'ftb'}),
|
||||
('sbs', {'sbs', 'hsbs', 'lr'}), ('ou', {'ou', 'hou', 'tb', 'htb'})]:
|
||||
if tokens & tags:
|
||||
hints.add(value)
|
||||
if len(hints) > 1:
|
||||
raise ValueError('Conflicting stereo filename tags; choose the layout explicitly')
|
||||
layout = next(iter(hints), None)
|
||||
source = 'filename'
|
||||
if not layout:
|
||||
# Matroska StereoMode/FFmpeg stereo_mode: only known left-first modes.
|
||||
mode = (metadata or {}).get('stereo_mode')
|
||||
layout = {'left_right': 'full-sbs', 'top_bottom': 'full-ou', 'mono': 'mono'}.get(mode)
|
||||
source = 'metadata'
|
||||
if mode and layout is None:
|
||||
raise ValueError('Unsupported stereo metadata; choose the eye order/layout explicitly')
|
||||
if not layout:
|
||||
raise ValueError('No stereo layout found; choose mono, SBS or OU (left/top eye first)')
|
||||
return {'kind': 'video' if suffix in VIDEO else 'photo', 'layout': layout, 'source': source}
|
||||
|
||||
|
||||
def geometry(width, height, layout):
|
||||
"""Bound transfer to 1920x1080; return packed dimensions and texel aspect."""
|
||||
if not 0 < width <= 32768 or not 0 < height <= 32768:
|
||||
raise ValueError('Invalid media dimensions')
|
||||
if layout not in LAYOUTS[1:]:
|
||||
raise ValueError('Resolve the layout before playback')
|
||||
scale = min(1, 1920 / width, 1080 / height)
|
||||
w, h = max(2, int(width * scale) // 2 * 2), max(2, int(height * scale) // 2 * 2)
|
||||
return w, h, {'mono': 1, 'sbs': 2, 'ou': .5, 'full-sbs': 1, 'full-ou': 1}[layout]
|
||||
|
||||
|
||||
def stereo_pixels(data, width, height, layout):
|
||||
"""Normalize top/bottom to OpenVR's left/right texture; preserve eye order."""
|
||||
if len(data) != width * height * 4:
|
||||
raise ValueError('Incomplete RGBA frame')
|
||||
if layout not in ('ou', 'full-ou'):
|
||||
return data, width, height
|
||||
stride, half = width * 4, height // 2
|
||||
return b''.join(data[y*stride:(y+1)*stride] +
|
||||
data[(y+half)*stride:(y+half+1)*stride] for y in range(half)), width*2, half
|
||||
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Send local media to Frame Control's own OpenVR player."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import frame_media
|
||||
import server
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument('files', nargs='*', type=Path)
|
||||
ap.add_argument('--launch', action='store_true', help='play the one file being sent')
|
||||
ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto')
|
||||
ap.add_argument('--theatre', action='store_true', help='bigger screen and dark surround')
|
||||
ap.add_argument('--list', action='store_true')
|
||||
ap.add_argument('--stop', action='store_true')
|
||||
args = ap.parse_args()
|
||||
if args.launch and len(args.files) != 1:
|
||||
ap.error('--launch needs exactly one file')
|
||||
if not args.files and not (args.list or args.stop):
|
||||
ap.error('choose files, --list or --stop')
|
||||
for path in args.files:
|
||||
if not path.is_file():
|
||||
ap.error('not a file: %s' % path)
|
||||
frame_media.plan(path.name, 'mono')
|
||||
if args.stop:
|
||||
print(json.dumps(server.media({'action': 'stop'})))
|
||||
for path in args.files:
|
||||
result = server.push_media(path.resolve())
|
||||
print(json.dumps(result))
|
||||
if args.launch:
|
||||
print(json.dumps(server.media({'action': 'play', 'id': result['id'],
|
||||
'layout': args.layout, 'theatre': args.theatre})))
|
||||
if args.list:
|
||||
print(json.dumps(server.media({'action': 'list'})))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
except (ValueError, server.Failure) as e:
|
||||
sys.exit(str(e))
|
||||
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Frame Control's local-media OpenVR player. Runs on the Frame, no third-party app.
|
||||
|
||||
SteamOS ffmpeg does hardware video decoding, scaling and audio output. OpenVR
|
||||
owns only our screen and optional black surround. Exiting destroys both.
|
||||
"""
|
||||
import argparse
|
||||
import ctypes as C
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import frame_media
|
||||
import frame_splat
|
||||
|
||||
LIB = '/opt/steamvr/bin/linuxarm64/libopenvr_api.so'
|
||||
H = C.c_uint64
|
||||
# Slots from Valve's openvr_capi.h, IVROverlay_028. Fail closed on another ABI.
|
||||
SLOTS = {
|
||||
'CreateOverlay': (1, [C.c_char_p, C.c_char_p, C.POINTER(H)]),
|
||||
'DestroyOverlay': (3, [H]),
|
||||
'SetOverlayFlag': (11, [H, C.c_int, C.c_bool]),
|
||||
'SetOverlayAlpha': (16, [H, C.c_float]),
|
||||
'SetOverlayTexelAspect': (18, [H, C.c_float]),
|
||||
'SetOverlaySortOrder': (20, [H, C.c_uint32]),
|
||||
'SetOverlayWidthInMeters': (22, [H, C.c_float]),
|
||||
'SetOverlayTransformTrackedDeviceRelative': (35, [H, C.c_uint32, C.c_void_p]),
|
||||
'ShowOverlay': (43, [H]),
|
||||
'SetOverlayRaw': (62, [H, C.c_void_p, C.c_uint32, C.c_uint32, C.c_uint32]),
|
||||
}
|
||||
|
||||
|
||||
class OverlayBusy(RuntimeError):
|
||||
"""SetOverlayRaw's RequestFailed (23): SteamVR isn't taking frames, e.g. the
|
||||
unworn headset is in standby (verified 2026-09-29). Transient, not fatal."""
|
||||
|
||||
|
||||
# A screen that can't take a frame for this long is broken, not asleep.
|
||||
BUSY_LIMIT = 300
|
||||
|
||||
|
||||
class Overlay:
|
||||
def __init__(self):
|
||||
self.handles = []
|
||||
self.vr = C.CDLL(LIB)
|
||||
self.vr.VR_InitInternal2.argtypes = [C.POINTER(C.c_int), C.c_int, C.c_char_p]
|
||||
self.vr.VR_GetGenericInterface.argtypes = [C.c_char_p, C.POINTER(C.c_int)]
|
||||
self.vr.VR_GetGenericInterface.restype = C.c_void_p
|
||||
err = C.c_int()
|
||||
self.vr.VR_InitInternal2(C.byref(err), 2, None)
|
||||
if err.value:
|
||||
raise RuntimeError('SteamVR init failed: %s' % err.value)
|
||||
ptr = self.vr.VR_GetGenericInterface(b'FnTable:IVROverlay_028', C.byref(err))
|
||||
if not ptr or err.value:
|
||||
self.vr.VR_ShutdownInternal()
|
||||
raise RuntimeError('SteamVR needs IVROverlay_028: %s' % err.value)
|
||||
self.table = C.cast(ptr, C.POINTER(C.c_void_p))
|
||||
|
||||
def call(self, name, *values):
|
||||
slot, args = SLOTS[name]
|
||||
rc = C.CFUNCTYPE(C.c_int, *args)(self.table[slot])(*values)
|
||||
if rc == 23 and name == 'SetOverlayRaw':
|
||||
raise OverlayBusy('SteamVR is not accepting frames (standby?)')
|
||||
if rc:
|
||||
raise RuntimeError('OpenVR %s failed: %s' % (name, rc))
|
||||
|
||||
def create(self, key, width, distance, stereo=False, aspect=1, order=1):
|
||||
handle = H()
|
||||
self.call('CreateOverlay', key.encode(), b'Frame Control media', C.byref(handle))
|
||||
self.handles.append(handle)
|
||||
self.call('SetOverlayWidthInMeters', handle, width)
|
||||
self.call('SetOverlaySortOrder', handle, order)
|
||||
self.call('SetOverlayTexelAspect', handle, aspect)
|
||||
if stereo:
|
||||
self.call('SetOverlayFlag', handle, 1024, True) # SideBySide_Parallel
|
||||
matrix = (C.c_float * 12)(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, -distance)
|
||||
self.call('SetOverlayTransformTrackedDeviceRelative', handle, 0, matrix)
|
||||
return handle
|
||||
|
||||
def pixels(self, handle, data, width, height):
|
||||
buf = C.create_string_buffer(data)
|
||||
self.call('SetOverlayRaw', handle, buf, width, height, 4)
|
||||
self.call('ShowOverlay', handle)
|
||||
|
||||
def close(self):
|
||||
# Best effort: SteamVR removes a disconnected client's overlays anyway,
|
||||
# and a teardown error must not overwrite a finished playback's status.
|
||||
try:
|
||||
for h in reversed(self.handles):
|
||||
try:
|
||||
self.call('DestroyOverlay', h)
|
||||
except Exception: # signals are already ignored here, so Stop isn't lost
|
||||
pass
|
||||
finally:
|
||||
self.vr.VR_ShutdownInternal()
|
||||
|
||||
|
||||
def probe(path):
|
||||
result = subprocess.run(['ffprobe', '-v', 'error', '-show_streams', '-of', 'json', str(path)],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
if result.returncode:
|
||||
raise ValueError(result.stderr[-2000:] or 'Cannot read media')
|
||||
streams = json.loads(result.stdout)['streams']
|
||||
video = next((s for s in streams if s['codec_type'] == 'video'), None)
|
||||
if not video:
|
||||
raise ValueError('No image or video stream')
|
||||
return video, any(s['codec_type'] == 'audio' for s in streams)
|
||||
|
||||
|
||||
def decoder_command(path, info, width, height, audio, photo=False):
|
||||
cmd = ['ffmpeg', '-nostdin', '-hide_banner', '-loglevel', 'error']
|
||||
if not photo:
|
||||
cmd += ['-re', '-readrate_initial_burst', '0']
|
||||
codec = {'h264': 'h264_v4l2m2m', 'hevc': 'hevc_v4l2m2m'}.get(info['codec_name'])
|
||||
if not codec:
|
||||
raise ValueError('Hardware playback currently supports H.264 and H.265 only')
|
||||
cmd += ['-c:v', codec]
|
||||
cmd += ['-i', str(path), '-map', '0:v:0', '-vf', 'scale=%s:%s' % (width, height),
|
||||
'-pix_fmt', 'rgba']
|
||||
if photo:
|
||||
cmd += ['-frames:v', '1']
|
||||
else:
|
||||
cmd += ['-r', '30']
|
||||
cmd += ['-f', 'rawvideo', 'pipe:1']
|
||||
if audio and not photo:
|
||||
cmd += ['-map', '0:a:0', '-f', 'pulse', 'Frame Control Media']
|
||||
return cmd
|
||||
|
||||
|
||||
def ignore_signals():
|
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
||||
|
||||
|
||||
def write_status(path, **values):
|
||||
tmp = path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(values))
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def play(args):
|
||||
path = Path(args.file).resolve(strict=True)
|
||||
status = Path(args.status)
|
||||
splat = path.suffix.lower() == '.splat'
|
||||
if splat:
|
||||
data, width, height = frame_splat.render(path)
|
||||
plan = frame_media.plan(path.name)
|
||||
aspect, photo, command = 1, True, None
|
||||
else:
|
||||
info, audio = probe(path)
|
||||
plan = frame_media.plan(path.name, args.layout, info.get('tags'))
|
||||
width, height, aspect = frame_media.geometry(info['width'], info['height'], plan['layout'])
|
||||
photo = plan['kind'] == 'photo'
|
||||
command = decoder_command(path, info, width, height, audio, photo)
|
||||
vr, proc, frames, started = None, None, 0, time.monotonic()
|
||||
dropped, busy_since, pending = 0, None, []
|
||||
|
||||
def show(handle, data, w, h, video=False):
|
||||
"""Submit a frame. During standby return False; a video frame is dropped."""
|
||||
nonlocal dropped, busy_since
|
||||
try:
|
||||
vr.pixels(handle, data, w, h)
|
||||
except OverlayBusy:
|
||||
if not video:
|
||||
return False # stills and the surround just wait; nothing is lost
|
||||
dropped += 1
|
||||
busy_since = busy_since or time.monotonic()
|
||||
if time.monotonic() - busy_since > BUSY_LIMIT:
|
||||
raise RuntimeError('SteamVR stopped accepting frames for %d s' % BUSY_LIMIT)
|
||||
return False
|
||||
if video:
|
||||
busy_since = None
|
||||
drain()
|
||||
return True
|
||||
|
||||
def drain():
|
||||
"""Re-send anything that arrived during standby (e.g. the theatre surround)."""
|
||||
while pending:
|
||||
item = pending.pop(0)
|
||||
try:
|
||||
vr.pixels(*item)
|
||||
except OverlayBusy:
|
||||
pending.insert(0, item)
|
||||
return
|
||||
except RuntimeError:
|
||||
pass # the surround is cosmetic; never end playback over it
|
||||
|
||||
def hold(handle, data, w, h):
|
||||
"""Keep a still (photo or splat) up until Stop, retrying through standby."""
|
||||
shown = False
|
||||
while True:
|
||||
if shown:
|
||||
drain()
|
||||
else:
|
||||
shown = show(handle, data, w, h)
|
||||
time.sleep(1)
|
||||
|
||||
# systemd sends SIGTERM to the whole unit, including ffmpeg. Python unwinds
|
||||
# ownership; no unrelated Steam/SteamVR process or setting is touched.
|
||||
def stop(signum, frame):
|
||||
raise InterruptedError('Stopped')
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
try:
|
||||
vr = Overlay()
|
||||
if args.theatre:
|
||||
try:
|
||||
surround = vr.create('framecontrol.media.surround', 40, 4, order=0)
|
||||
vr.call('SetOverlayAlpha', surround, .85)
|
||||
if not show(surround, b'\x00\x00\x00\xff', 1, 1):
|
||||
pending.append((surround, b'\x00\x00\x00\xff', 1, 1))
|
||||
except RuntimeError as e:
|
||||
# Cosmetic: play without the dark surround rather than not at all.
|
||||
print('Theatre surround unavailable: %s' % e, flush=True)
|
||||
screen = vr.create('framecontrol.media.screen', 3 if args.theatre else 1.6, 2,
|
||||
plan['layout'] != 'mono', aspect)
|
||||
if splat:
|
||||
write_status(status, state='playing', file=path.name, frames=1, **plan)
|
||||
hold(screen, data, width, height)
|
||||
proc = subprocess.Popen(command, stdout=subprocess.PIPE)
|
||||
video_start = time.monotonic()
|
||||
while True:
|
||||
data = proc.stdout.read(width * height * 4)
|
||||
if not data:
|
||||
break
|
||||
data, outw, outh = frame_media.stereo_pixels(data, width, height, plan['layout'])
|
||||
if not photo:
|
||||
time.sleep(max(0, video_start + frames/30 - time.monotonic()))
|
||||
if photo:
|
||||
still = data, outw, outh
|
||||
else:
|
||||
show(screen, data, outw, outh, video=True)
|
||||
frames += 1
|
||||
if frames == 1 or frames % 30 == 0:
|
||||
write_status(status, state='playing', file=path.name, frames=frames,
|
||||
dropped=dropped, seconds=time.monotonic()-started, **plan)
|
||||
if not photo:
|
||||
time.sleep(max(0, video_start + frames/30 - time.monotonic()))
|
||||
rc = proc.wait(timeout=10)
|
||||
if rc:
|
||||
raise RuntimeError('ffmpeg exited %s; see media log' % rc)
|
||||
if not frames:
|
||||
raise RuntimeError('Decoder produced no frames')
|
||||
if photo:
|
||||
hold(screen, *still)
|
||||
# From here on a Stop can't change the outcome; don't let it turn
|
||||
# 'ended' into an error while we write status and clean up.
|
||||
ignore_signals()
|
||||
write_status(status, state='ended', frames=frames, dropped=dropped,
|
||||
seconds=time.monotonic()-started)
|
||||
except InterruptedError:
|
||||
ignore_signals()
|
||||
write_status(status, state='stopped', frames=frames, dropped=dropped)
|
||||
finally:
|
||||
ignore_signals()
|
||||
if proc:
|
||||
if proc.poll() is None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
proc.stdout.close()
|
||||
if vr:
|
||||
vr.close()
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument('file')
|
||||
ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto')
|
||||
ap.add_argument('--theatre', action='store_true')
|
||||
ap.add_argument('--status', required=True)
|
||||
args = ap.parse_args()
|
||||
try:
|
||||
play(args)
|
||||
except Exception as e:
|
||||
write_status(Path(args.status), state='error', error=str(e))
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Frame-side library and process ownership for Frame Control media.
|
||||
|
||||
Only the dedicated systemd user unit is controlled. No SteamVR settings change.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import frame_media
|
||||
from frame_media_player import probe
|
||||
|
||||
ROOT = Path.home() / 'Videos' / 'FrameControl'
|
||||
RUNTIME = Path.home() / '.local' / 'share' / 'frame-control' / 'media'
|
||||
UNIT = 'frame-control-media.service'
|
||||
STATUS = RUNTIME / 'status.json'
|
||||
|
||||
|
||||
def media_path(identity):
|
||||
if not isinstance(identity, str) or '\\' in identity or '\x00' in identity:
|
||||
raise ValueError('Invalid media id')
|
||||
parts = Path(identity).parts
|
||||
if len(parts) != 2 or not re.fullmatch('[0-9a-f]{32}', parts[0]) or parts[1].startswith('.'):
|
||||
raise ValueError('Invalid media id')
|
||||
candidate = ROOT / identity
|
||||
if candidate.is_symlink() or candidate.parent.is_symlink():
|
||||
raise ValueError('Media links are not supported')
|
||||
path = candidate.resolve(strict=True)
|
||||
if not path.is_file() or ROOT.resolve() not in path.parents:
|
||||
raise ValueError('Media file is outside the library')
|
||||
return path
|
||||
|
||||
|
||||
def active():
|
||||
return subprocess.run(['systemctl', '--user', 'is-active', '--quiet', UNIT]).returncode == 0
|
||||
|
||||
|
||||
def status():
|
||||
running = active()
|
||||
try:
|
||||
state = json.loads(STATUS.read_text())
|
||||
except (OSError, ValueError):
|
||||
state = {'state': 'idle'}
|
||||
if not running and state.get('state') in ('playing', 'starting', 'paused'):
|
||||
state = {'state': 'stopped', 'message': 'Player exited; check the media log if this was unexpected'}
|
||||
return dict(state, running=running)
|
||||
|
||||
|
||||
def run(body):
|
||||
action = body.get('action')
|
||||
if action == 'list':
|
||||
files = []
|
||||
if ROOT.exists():
|
||||
for folder in sorted(ROOT.iterdir()):
|
||||
if not re.fullmatch('[0-9a-f]{32}', folder.name) or not folder.is_dir() or folder.is_symlink():
|
||||
continue
|
||||
for path in sorted(folder.iterdir()):
|
||||
if path.is_file() and not path.is_symlink() and not path.name.startswith('.'):
|
||||
files.append({'id': folder.name+'/'+path.name, 'name': path.name, 'bytes': path.stat().st_size})
|
||||
return {'files': files, 'player': status()}
|
||||
if action == 'status':
|
||||
return status()
|
||||
if action == 'stop':
|
||||
# --collect unloads the unit after it exits; systemctl then exits 5
|
||||
# ("not loaded", verified on the Frame). That's a finished player, not an error.
|
||||
stopped = subprocess.run(['systemctl', '--user', 'stop', UNIT], capture_output=True, text=True, timeout=15)
|
||||
if stopped.returncode not in (0, 5):
|
||||
raise RuntimeError('Could not stop the media player: ' + (stopped.stderr.strip() or 'exit %s' % stopped.returncode))
|
||||
return {'message': 'Media player stopped', **status()}
|
||||
if action != 'play':
|
||||
raise ValueError('Media action must be list, status, play or stop')
|
||||
path = media_path(body.get('id'))
|
||||
if type(body.get('theatre', False)) is not bool:
|
||||
raise ValueError('theatre must be true or false')
|
||||
info = {} if path.suffix.lower() == '.splat' else probe(path)[0]
|
||||
plan = frame_media.plan(path.name, body.get('layout', 'auto'), info.get('tags'))
|
||||
if active():
|
||||
raise ValueError('Stop the current media before starting another file')
|
||||
# systemd owns the process group and refuses a concurrent start of this name.
|
||||
# The runtime cap also cleans up if the controlling computer disconnects.
|
||||
subprocess.run(['systemctl', '--user', 'reset-failed', UNIT], stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL, timeout=10)
|
||||
STATUS.write_text(json.dumps({'state': 'starting', 'file': path.name}))
|
||||
command = ['systemd-run', '--user', '--quiet', '--collect', '--unit='+UNIT,
|
||||
'--property=RuntimeMaxSec=14400', '--property=TimeoutStopSec=8',
|
||||
'--property=StandardOutput=append:'+str(RUNTIME/'player.log'),
|
||||
'--property=StandardError=append:'+str(RUNTIME/'player.log'),
|
||||
'python3', str(RUNTIME/'frame_media_player.py'), str(path),
|
||||
'--layout', body.get('layout', 'auto'), '--status', str(STATUS)]
|
||||
if body.get('theatre'):
|
||||
command.append('--theatre')
|
||||
started = subprocess.run(command, capture_output=True, text=True, timeout=15)
|
||||
if started.returncode:
|
||||
raise RuntimeError('Could not start the media player: ' + (started.stderr.strip() or 'systemd-run exited %s' % started.returncode))
|
||||
return {'message': 'Starting Frame Control media', 'plan': plan}
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
print(json.dumps(run(json.load(sys.stdin))))
|
||||
except Exception as e:
|
||||
print(json.dumps({'error': str(e)}))
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Small, bounded CPU Gaussian-splat preview renderer (Frame Control-owned).
|
||||
|
||||
Reads the common 32-byte .splat record: position/scale float32 triplets,
|
||||
RGBA bytes, then normalized quaternion bytes (wxyz). Two perspective cameras,
|
||||
projected 3D covariance, back-to-front alpha compositing. This is a stationary
|
||||
stereo preview, not a six-degree-of-freedom scene or a large-scene renderer.
|
||||
"""
|
||||
import math
|
||||
from pathlib import Path
|
||||
import struct
|
||||
|
||||
MAX_SPLATS = 20000
|
||||
RECORD = struct.Struct('<6f8B')
|
||||
|
||||
|
||||
def read(path):
|
||||
size = Path(path).stat().st_size
|
||||
if not size or size % RECORD.size or size > MAX_SPLATS * RECORD.size:
|
||||
raise ValueError('Use a 32-byte .splat file with 1–20,000 Gaussians; PLY/SPZ and larger scenes are not supported yet')
|
||||
values = []
|
||||
with open(path, 'rb') as stream:
|
||||
for row in RECORD.iter_unpack(stream.read(MAX_SPLATS * RECORD.size + 1)):
|
||||
xyz, scales = row[:3], row[3:6]
|
||||
if not all(math.isfinite(v) and abs(v) <= 1e6 for v in row[:6]) or min(scales) <= 0:
|
||||
raise ValueError('Invalid splat position or scale')
|
||||
q = [(v - 128) / 128 for v in row[10:14]]
|
||||
length = math.sqrt(sum(v*v for v in q))
|
||||
if length < .01:
|
||||
raise ValueError('Invalid splat quaternion')
|
||||
w, x, y, z = [v / length for v in q]
|
||||
rotation = ((1-2*(y*y+z*z), 2*(x*y-z*w), 2*(x*z+y*w)),
|
||||
(2*(x*y+z*w), 1-2*(x*x+z*z), 2*(y*z-x*w)),
|
||||
(2*(x*z-y*w), 2*(y*z+x*w), 1-2*(x*x+y*y)))
|
||||
cov = [[sum(rotation[i][k]*rotation[j][k]*scales[k]**2 for k in range(3))
|
||||
for j in range(3)] for i in range(3)]
|
||||
values.append((xyz, cov, row[6:10]))
|
||||
return values
|
||||
|
||||
|
||||
def render(path, width=320, height=240):
|
||||
values = read(path)
|
||||
lo = [min(p[0][i] for p in values) for i in range(3)]
|
||||
hi = [max(p[0][i] for p in values) for i in range(3)]
|
||||
center = [(a+b)/2 for a, b in zip(lo, hi)]
|
||||
radius = max(max(b-a for a, b in zip(lo, hi))/2, .01)
|
||||
# Normalize captures to a two-metre box. Source units are not assumed metres.
|
||||
normalized = [([(xyz[i]-center[i])/radius for i in range(3)],
|
||||
[[v/radius**2 for v in row] for row in cov], color)
|
||||
for xyz, cov, color in values]
|
||||
normalized.sort(key=lambda p: p[0][2]) # camera is at z=3; farthest first
|
||||
focal = width * .8
|
||||
eyes = []
|
||||
for eye in (-.032, .032):
|
||||
pixels = bytearray(b'\x00\x00\x00\xff' * (width*height))
|
||||
for (x, y, z), cov, color in normalized:
|
||||
x -= eye
|
||||
depth = 3-z
|
||||
px, py = width/2+focal*x/depth, height/2-focal*y/depth
|
||||
jac = ((focal/depth, 0, focal*x/depth**2),
|
||||
(0, -focal/depth, -focal*y/depth**2))
|
||||
screen = [[sum(jac[i][a]*cov[a][b]*jac[j][b] for a in range(3) for b in range(3))
|
||||
for j in range(2)] for i in range(2)]
|
||||
a, b, c = screen[0][0]+.3, screen[0][1], screen[1][1]+.3
|
||||
det = a*c-b*b
|
||||
if det <= 0 or not math.isfinite(det):
|
||||
raise ValueError('Splat covariance is not renderable')
|
||||
# A footprint cap bounds work on malformed or oversized Gaussians.
|
||||
rx, ry = min(32, math.ceil(3*math.sqrt(a))), min(32, math.ceil(3*math.sqrt(c)))
|
||||
for sy in range(max(0, int(py)-ry), min(height, int(py)+ry+1)):
|
||||
dy = sy+.5-py
|
||||
for sx in range(max(0, int(px)-rx), min(width, int(px)+rx+1)):
|
||||
dx = sx+.5-px
|
||||
power = (c*dx*dx-2*b*dx*dy+a*dy*dy)/det
|
||||
if power > 9:
|
||||
continue
|
||||
alpha = color[3]/255 * math.exp(-.5*power)
|
||||
offset = (sy*width+sx)*4
|
||||
for k in range(3):
|
||||
pixels[offset+k] = round(color[k]*alpha+pixels[offset+k]*(1-alpha))
|
||||
eyes.append(pixels)
|
||||
stride = width*4
|
||||
return b''.join(eyes[0][y*stride:(y+1)*stride]+eyes[1][y*stride:(y+1)*stride]
|
||||
for y in range(height)), width*2, height
|
||||
@@ -0,0 +1,446 @@
|
||||
"""Touch and direct input for the Steam Frame's panels. Frame Control's server runs this ON the Frame.
|
||||
|
||||
gamescope, the Frame's compositor, serves Valve's own input injection: an EIS
|
||||
socket (libei's server side), which Steam uses to feed it Remote Play input.
|
||||
This connects to it with libei, which is on the SteamOS image, and points,
|
||||
clicks, scrolls and types into the panel that has focus in the headset: the
|
||||
one the wearer last used. No install, and it reaches every panel, on either
|
||||
of gamescope's X displays (see docs/streaming.md).
|
||||
|
||||
python3 frame_touch.py focus print the focused panel as JSON
|
||||
python3 frame_touch.py panels print every app panel as JSON, and which has focus
|
||||
python3 frame_touch.py read events on stdin, one JSON object (or list) per line:
|
||||
{"fx": 0.5, "fy": 0.2, "window": 123, "display": ":1"}
|
||||
pointer to that fraction of that panel; any event can
|
||||
name its panel, and goes nowhere if another has focus
|
||||
{"dx": 4, "dy": -2} pointer by that much
|
||||
{"button": "left", "down": true} left, right or middle; "down" false releases
|
||||
{"scroll": [0, 120]} by pixels; positive y scrolls down
|
||||
{"key": 30, "down": true} a Linux (evdev) key code, as the page maps KeyboardEvent.code
|
||||
{"text": "hello"} printable ASCII, typed on a US layout
|
||||
|
||||
Status goes to stdout, one JSON object per line: {"state": "ready" | "error", ...}.
|
||||
Standard library only (ctypes for libei), like the rest of what runs on the Frame.
|
||||
"""
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
SOCKET = "/run/user/{uid}/gamescope-0-ei" # filled in on the Frame (Windows has no getuid; the tests import this)
|
||||
BUTTONS = {"left": 0x110, "right": 0x111, "middle": 0x112} # BTN_LEFT, BTN_RIGHT, BTN_MIDDLE
|
||||
SHIFT = 42 # KEY_LEFTSHIFT
|
||||
# Printable ASCII on a US layout: character -> (evdev key code, shifted).
|
||||
ROWS = [("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
|
||||
("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)]
|
||||
ASCII = {" ": (57, False), "\n": (28, False), "\t": (15, False)}
|
||||
for plain, shifted, first in ROWS:
|
||||
for i, (a, b) in enumerate(zip(plain, shifted)):
|
||||
ASCII[a], ASCII[b] = (first + i, False), (first + i, True)
|
||||
|
||||
# libei's event types and device capabilities (libei.h, libei 1.4).
|
||||
EV_CONNECT, EV_DISCONNECT, EV_SEAT_ADDED, EV_DEVICE_ADDED, EV_DEVICE_REMOVED = 1, 2, 3, 5, 6
|
||||
EV_DEVICE_PAUSED, EV_DEVICE_RESUMED = 7, 8
|
||||
CAP_POINTER, CAP_ABSOLUTE, CAP_KEYBOARD, CAP_SCROLL, CAP_BUTTON = 1, 2, 4, 16, 32
|
||||
|
||||
|
||||
def say(state, **more):
|
||||
print(json.dumps({"state": state, **more}), flush=True)
|
||||
|
||||
|
||||
# ---- which panel has focus -----------------------------------------------------
|
||||
|
||||
def xprop_root(display, name):
|
||||
try:
|
||||
out = subprocess.run(["xprop", "-root", name], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return []
|
||||
values = out.split("=", 1)[1] if "=" in out else ""
|
||||
return [int(v) for v in values.replace(",", " ").split() if v.isdigit()]
|
||||
|
||||
|
||||
def window_info(display, window):
|
||||
"""Name and geometry of a window on one X display, or None if it isn't there."""
|
||||
try:
|
||||
which = ["-root"] if window == "root" else ["-id", str(window)]
|
||||
out = subprocess.run(["xwininfo", *which], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
if "IsViewable" not in out:
|
||||
return None
|
||||
info = {}
|
||||
for line in out.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("xwininfo: Window id:"):
|
||||
info["name"] = line.split('"', 1)[1].rsplit('"', 1)[0] if '"' in line else ""
|
||||
for key, field in (("Absolute upper-left X:", "x"), ("Absolute upper-left Y:", "y"),
|
||||
("Width:", "width"), ("Height:", "height")):
|
||||
if line.startswith(key):
|
||||
info[field] = int(line.split(":", 1)[1])
|
||||
return info if "width" in info else None
|
||||
|
||||
|
||||
def displays():
|
||||
return sorted(f":{n[1:]}" for n in os.listdir("/tmp/.X11-unix") if n[1:].isdigit())
|
||||
|
||||
|
||||
def window_pid(display, window):
|
||||
try:
|
||||
out = subprocess.run(["xprop", "-id", str(window), "_NET_WM_PID"], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
value = out.rsplit("=", 1)[-1].strip() if "=" in out else ""
|
||||
return int(value) if value.isdigit() else None
|
||||
|
||||
|
||||
def locate(window, pid):
|
||||
"""The display a focusable window is on, with its name and geometry.
|
||||
|
||||
Window ids are per X server, so :0 and :1 can both have one; the pid gamescope
|
||||
lists with it (GAMESCOPE_FOCUSABLE_WINDOWS) tells them apart.
|
||||
"""
|
||||
found = []
|
||||
for display in displays():
|
||||
info = window_info(display, window)
|
||||
if info:
|
||||
found.append((display, info))
|
||||
if len(found) > 1 and pid:
|
||||
found = [f for f in found if window_pid(f[0], window) == pid] or found
|
||||
if not found:
|
||||
return None
|
||||
display, info = found[0]
|
||||
root = window_info(display, "root") or {}
|
||||
return {"window": window, "display": display, **info,
|
||||
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
|
||||
|
||||
|
||||
def focusable():
|
||||
"""gamescope's focusable windows as (window, app id, pid)."""
|
||||
t = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
|
||||
return [tuple(t[i:i + 3]) for i in range(0, len(t) - 2, 3)]
|
||||
|
||||
|
||||
def focus_display():
|
||||
"""The display of the focused window, from GAMESCOPE_FOCUS_DISPLAY on :0's root.
|
||||
|
||||
gamescope writes the name (":1") as 32-bit items, so its first four bytes land,
|
||||
little-endian, in the first value: 12602 is 0x313A, ":1" (steamcompmgr.cpp;
|
||||
seen 2026-09-29).
|
||||
"""
|
||||
values = xprop_root(":0", "GAMESCOPE_FOCUS_DISPLAY")
|
||||
if not values:
|
||||
return None
|
||||
name = (values[0] & 0xFFFFFFFF).to_bytes(4, "little").split(b"\0", 1)[0].decode("ascii", "replace")
|
||||
return name if name[:1] == ":" and name[1:].isdigit() else None
|
||||
|
||||
|
||||
def focus_now():
|
||||
"""Just which window and display have focus: two property reads, for checking a press."""
|
||||
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
|
||||
return (window or None, focus_display() if window else None)
|
||||
|
||||
|
||||
def focus():
|
||||
"""The panel that has focus in the headset: window, display, name and sizes (gamescope
|
||||
publishes the window and its display on :0's root)."""
|
||||
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
|
||||
if not window:
|
||||
return {"window": None}
|
||||
app, pid = next(((a, p) for w, a, p in focusable() if w == window), (None, None))
|
||||
display = focus_display()
|
||||
info = window_info(display, window) if display else None
|
||||
if info:
|
||||
root = window_info(display, "root") or {}
|
||||
panel = {"window": window, "display": display, **info,
|
||||
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
|
||||
else:
|
||||
panel = locate(window, pid) # no display published: tell them apart by pid
|
||||
return {**panel, "app": app} if panel else {"window": None}
|
||||
|
||||
|
||||
def panels():
|
||||
"""Every app panel (gamescope's focusable windows), for watching one that hasn't focus."""
|
||||
now = focus()
|
||||
found = []
|
||||
for window, app, pid in focusable():
|
||||
panel = locate(window, pid)
|
||||
if panel and panel["width"] > 1 and panel["height"] > 1 and \
|
||||
not any(f["window"] == window and f["display"] == panel["display"] for f in found):
|
||||
panel.pop("root", None)
|
||||
found.append({**panel, "app": app, "focused": (window, panel["display"]) ==
|
||||
(now.get("window"), now.get("display"))})
|
||||
return {"focus": now.get("window"), "focus_display": now.get("display"), "panels": found}
|
||||
|
||||
|
||||
def to_root(panel, fx, fy):
|
||||
"""A point given as a fraction of the panel, in the root coordinates gamescope's pointer uses.
|
||||
|
||||
gamescope fits each panel's window to its display, so a 1920x1080 window on a
|
||||
1280x720 display takes pointer positions at two thirds scale (verified 2026-09-29).
|
||||
"""
|
||||
rw, rh = panel["root"]
|
||||
w, h = panel["width"], panel["height"]
|
||||
s = min(rw / w, rh / h)
|
||||
ox, oy = (rw - w * s) / 2, (rh - h * s) / 2
|
||||
fx, fy = min(max(fx, 0.0), 1.0), min(max(fy, 0.0), 1.0)
|
||||
return ox + fx * (w * s - 1), oy + fy * (h * s - 1)
|
||||
|
||||
|
||||
# ---- gamescope's input socket ----------------------------------------------------
|
||||
|
||||
def libei():
|
||||
L = ctypes.CDLL("libei.so.1")
|
||||
vp, c = ctypes.c_void_p, ctypes
|
||||
sig = {
|
||||
"ei_new_sender": (vp, [vp]), "ei_configure_name": (None, [vp, c.c_char_p]),
|
||||
"ei_setup_backend_socket": (c.c_int, [vp, c.c_char_p]), "ei_get_fd": (c.c_int, [vp]),
|
||||
"ei_dispatch": (None, [vp]), "ei_get_event": (vp, [vp]), "ei_event_get_type": (c.c_int, [vp]),
|
||||
"ei_event_unref": (vp, [vp]), "ei_event_get_seat": (vp, [vp]), "ei_event_get_device": (vp, [vp]),
|
||||
"ei_device_has_capability": (c.c_bool, [vp, c.c_int]), "ei_now": (c.c_uint64, [vp]),
|
||||
"ei_device_start_emulating": (None, [vp, c.c_uint32]), "ei_device_stop_emulating": (None, [vp]),
|
||||
"ei_device_frame": (None, [vp, c.c_uint64]),
|
||||
"ei_device_pointer_motion": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_pointer_motion_absolute": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_button_button": (None, [vp, c.c_uint32, c.c_bool]),
|
||||
"ei_device_scroll_delta": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_keyboard_key": (None, [vp, c.c_uint32, c.c_bool]),
|
||||
"ei_unref": (vp, [vp]),
|
||||
}
|
||||
for name, (res, args) in sig.items():
|
||||
f = getattr(L, name)
|
||||
f.restype, f.argtypes = res, args
|
||||
return L
|
||||
|
||||
|
||||
class Gamescope:
|
||||
"""One connection to gamescope's EIS socket and its virtual input device."""
|
||||
|
||||
def __init__(self):
|
||||
self.L = L = libei()
|
||||
self.ei = L.ei_new_sender(None)
|
||||
L.ei_configure_name(self.ei, b"Frame Control")
|
||||
if L.ei_setup_backend_socket(self.ei, SOCKET.format(uid=os.getuid()).encode()) != 0:
|
||||
raise RuntimeError("Couldn't reach gamescope's input socket. Is the headset on?")
|
||||
self.fd = L.ei_get_fd(self.ei)
|
||||
self.device, self.sequence, self.held, self.keys, self.alive = None, 0, set(), set(), True
|
||||
|
||||
def pump(self, wait=0.0):
|
||||
"""Handle gamescope's events; False once it has disconnected."""
|
||||
select.select([self.fd], [], [], wait)
|
||||
self.L.ei_dispatch(self.ei)
|
||||
alive = True
|
||||
while True:
|
||||
ev = self.L.ei_get_event(self.ei)
|
||||
if not ev:
|
||||
return alive
|
||||
kind = self.L.ei_event_get_type(ev)
|
||||
if kind == EV_SEAT_ADDED:
|
||||
seat = self.L.ei_event_get_seat(ev)
|
||||
# Variadic, ending in 0 (NULL): ask for everything we send.
|
||||
self.L.ei_seat_bind_capabilities(ctypes.c_void_p(seat), *map(ctypes.c_int, (
|
||||
CAP_POINTER, CAP_ABSOLUTE, CAP_BUTTON, CAP_SCROLL, CAP_KEYBOARD, 0)))
|
||||
elif kind == EV_DEVICE_RESUMED:
|
||||
device = self.L.ei_event_get_device(ev)
|
||||
if self.L.ei_device_has_capability(device, CAP_ABSOLUTE):
|
||||
self.sequence += 1
|
||||
self.L.ei_device_start_emulating(device, self.sequence)
|
||||
self.device = device
|
||||
# Releases that arrived while it was paused were dropped: let go of
|
||||
# everything now, so the headset and this agent agree nothing is held.
|
||||
if self.held or self.keys:
|
||||
self.release_all()
|
||||
elif kind in (EV_DEVICE_PAUSED, EV_DEVICE_REMOVED):
|
||||
if self.L.ei_event_get_device(ev) == self.device:
|
||||
self.device = None
|
||||
elif kind == EV_DISCONNECT:
|
||||
self.device, alive, self.alive = None, False, False
|
||||
self.L.ei_event_unref(ev)
|
||||
|
||||
def wait_ready(self, timeout=5):
|
||||
end = time.time() + timeout
|
||||
while self.device is None and time.time() < end:
|
||||
if not self.pump(0.1):
|
||||
break
|
||||
if self.device is None:
|
||||
raise RuntimeError("gamescope closed its input socket" if not self.alive
|
||||
else "gamescope didn't offer an input device")
|
||||
|
||||
def frame(self):
|
||||
self.L.ei_device_frame(self.device, self.L.ei_now(self.ei))
|
||||
self.L.ei_dispatch(self.ei)
|
||||
|
||||
def move_to(self, x, y):
|
||||
self.L.ei_device_pointer_motion_absolute(self.device, x, y)
|
||||
self.frame()
|
||||
|
||||
def move_by(self, dx, dy):
|
||||
self.L.ei_device_pointer_motion(self.device, dx, dy)
|
||||
self.frame()
|
||||
|
||||
def button(self, name, down):
|
||||
code = BUTTONS[name]
|
||||
if down == (code in self.held):
|
||||
return # already in that state
|
||||
self.L.ei_device_button_button(self.device, code, down)
|
||||
self.frame()
|
||||
(self.held.add if down else self.held.discard)(code)
|
||||
|
||||
def scroll(self, dx, dy):
|
||||
self.L.ei_device_scroll_delta(self.device, dx, dy)
|
||||
self.frame()
|
||||
|
||||
def key(self, code, down):
|
||||
self.L.ei_device_keyboard_key(self.device, code, down)
|
||||
self.frame()
|
||||
(self.keys.add if down else self.keys.discard)(code)
|
||||
# Paced: a burst of keys can reach the app out of order (seen 2026-09-29).
|
||||
time.sleep(0.008)
|
||||
|
||||
def text(self, text):
|
||||
for ch in text:
|
||||
if ch not in ASCII:
|
||||
continue
|
||||
code, shifted = ASCII[ch]
|
||||
if shifted:
|
||||
self.key(SHIFT, True)
|
||||
self.key(code, True)
|
||||
self.key(code, False)
|
||||
if shifted:
|
||||
self.key(SHIFT, False)
|
||||
|
||||
def release_all(self):
|
||||
"""Let go of every button and key still down, so nothing stays held in the headset."""
|
||||
for code in list(self.held):
|
||||
name = next(n for n, c in BUTTONS.items() if c == code)
|
||||
self.button(name, False)
|
||||
for code in list(self.keys):
|
||||
self.key(code, False)
|
||||
|
||||
|
||||
# ---- events from the server --------------------------------------------------------
|
||||
|
||||
def events(line):
|
||||
try:
|
||||
data = json.loads(line)
|
||||
except ValueError:
|
||||
return []
|
||||
return [e for e in (data if isinstance(data, list) else [data]) if isinstance(e, dict)]
|
||||
|
||||
|
||||
def number(value, limit=100000.0):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise ValueError("not a number")
|
||||
return max(-limit, min(limit, float(value)))
|
||||
|
||||
|
||||
STALE = [False] # whether the last status said a tap went nowhere
|
||||
|
||||
|
||||
def aimed_elsewhere(event, panel):
|
||||
"""Whether an event names a panel that isn't the one with focus now."""
|
||||
if "window" not in event:
|
||||
return False
|
||||
return (panel.get("window"), panel.get("display")) != (event.get("window"), event.get("display"))
|
||||
|
||||
|
||||
def apply(gs, event, panel):
|
||||
"""Send one event; returns the focused panel it checked against (looked up at most once a second).
|
||||
|
||||
Positions and presses name the panel they were meant for. If focus has moved to
|
||||
another panel since, they go nowhere, so a tap can't land on the wrong one;
|
||||
releases always go, so nothing stays held.
|
||||
"""
|
||||
# Moves may use a focus reading up to a second old; anything that acts (a press, key,
|
||||
# text or scroll) reads it afresh, so it can't land on a panel that took focus since.
|
||||
acts = any(k in event for k in ("button", "key", "text", "scroll")) and event.get("down") is not False
|
||||
if "window" in event:
|
||||
if panel and acts and focus_now() == (panel.get("window"), panel.get("display")):
|
||||
pass # still the same panel (its geometry is re-read on the usual one-second schedule)
|
||||
elif acts or not panel or time.time() - panel.get("_at", 0) > 1 or aimed_elsewhere(event, panel):
|
||||
panel = {**focus(), "_at": time.time()}
|
||||
stale = aimed_elsewhere(event, panel) if "window" in event else False
|
||||
if stale and not (event.get("down") is False and ("button" in event or "key" in event)):
|
||||
say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True) # the page re-syncs
|
||||
STALE[0] = True
|
||||
return panel
|
||||
if STALE[0] and not stale:
|
||||
# Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
|
||||
STALE[0] = False
|
||||
say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
|
||||
if "fx" in event and panel and panel.get("window"):
|
||||
gs.move_to(*to_root(panel, number(event["fx"], 1), number(event["fy"], 1)))
|
||||
if "dx" in event or "dy" in event:
|
||||
gs.move_by(number(event.get("dx", 0), 2000), number(event.get("dy", 0), 2000))
|
||||
if event.get("button") in BUTTONS:
|
||||
gs.button(event["button"], event.get("down") is not False)
|
||||
if isinstance(event.get("scroll"), list) and len(event["scroll"]) == 2:
|
||||
gs.scroll(number(event["scroll"][0], 5000), number(event["scroll"][1], 5000))
|
||||
if isinstance(event.get("key"), int) and not isinstance(event["key"], bool) and 0 < event["key"] < 768:
|
||||
gs.key(event["key"], event.get("down") is not False)
|
||||
if isinstance(event.get("text"), str):
|
||||
gs.text(event["text"][:500])
|
||||
return panel
|
||||
|
||||
|
||||
def main():
|
||||
if sys.argv[1:] == ["focus"]:
|
||||
print(json.dumps(focus()))
|
||||
return 0
|
||||
if sys.argv[1:] == ["panels"]:
|
||||
print(json.dumps(panels()))
|
||||
return 0
|
||||
try:
|
||||
gs = Gamescope()
|
||||
gs.wait_ready()
|
||||
except (OSError, RuntimeError) as e:
|
||||
say("error", message=str(e))
|
||||
return 1
|
||||
say("ready", focus=focus().get("window"))
|
||||
stdin, pending, panel = sys.stdin.fileno(), b"", None
|
||||
try:
|
||||
while True:
|
||||
ready, _, _ = select.select([stdin, gs.fd], [], [], 30)
|
||||
if gs.fd in ready and not gs.pump():
|
||||
say("error", message="gamescope closed its input socket")
|
||||
return 1
|
||||
if stdin not in ready:
|
||||
continue
|
||||
chunk = os.read(stdin, 65536)
|
||||
if not chunk:
|
||||
return 0 # the server went away
|
||||
*lines, pending = (pending + chunk).split(b"\n")
|
||||
for line in lines:
|
||||
waited = False
|
||||
for event in events(line):
|
||||
if gs.device is None and waited:
|
||||
continue # still paused: don't wait again for each event of this batch
|
||||
if gs.device is None:
|
||||
waited = True
|
||||
# Paused (gamescope can pause the device): wait a moment; drop this
|
||||
# event if it doesn't come back. Only a disconnect ends the session.
|
||||
try:
|
||||
gs.wait_ready(2)
|
||||
except RuntimeError:
|
||||
if not gs.alive:
|
||||
raise
|
||||
continue
|
||||
try:
|
||||
panel = apply(gs, event, panel)
|
||||
except (ValueError, KeyError, TypeError, OSError):
|
||||
continue # the server checks events; skip anything odd
|
||||
except RuntimeError as e:
|
||||
say("error", message=str(e))
|
||||
return 1
|
||||
finally:
|
||||
if gs.device is not None:
|
||||
gs.release_all() # never leave a button held down in the headset
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+570
-23
@@ -87,6 +87,7 @@
|
||||
.wait::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: var(--dim); flex: none; }
|
||||
|
||||
/* ---- drop anywhere ---- */
|
||||
#media select { min-width: 0; max-width: 100%; flex: 1; }
|
||||
.dropzone { position: fixed; inset: 0; z-index: 40; display: grid; place-items: center; pointer-events: none;
|
||||
background: rgba(14,20,27,.82); backdrop-filter: blur(3px); }
|
||||
.dropzone > div { padding: 42px 60px; border: 2px dashed var(--blue); border-radius: 8px; text-align: center;
|
||||
@@ -146,6 +147,18 @@
|
||||
.viewer .badge { position: absolute; left: 12px; top: 12px; font-size: 11px; letter-spacing: 1.2px; text-transform: uppercase;
|
||||
background: rgba(0,0,0,.6); color: var(--text); padding: 4px 9px; border-radius: 2px; }
|
||||
.viewer .badge.live { left: auto; right: 12px; background: var(--bad); color: #fff; font-weight: 700; }
|
||||
.view-actions { display: flex; gap: 10px; }
|
||||
.ctrl-status { margin-top: 10px; font-size: 13px; color: var(--blue); }
|
||||
.ctrl-status::before { content: "● "; }
|
||||
.ctrl-status.warn { color: var(--warn); }
|
||||
/* Control: the view takes taps, clicks, drags and keys, and none of them zoom or pan it. */
|
||||
.viewer.control { outline: 1px solid var(--blue); }
|
||||
.viewer.control canvas { cursor: crosshair; touch-action: none; }
|
||||
.viewer.control.rel canvas { cursor: move; }
|
||||
.viewer.immersive { position: fixed; z-index: 40; inset: env(safe-area-inset-top) env(safe-area-inset-right) 0 env(safe-area-inset-left);
|
||||
max-height: none; aspect-ratio: auto; border-radius: 0; }
|
||||
.ctrl-keys { margin-top: 10px; }
|
||||
#panelPick { max-width: 240px; height: 30px; }
|
||||
.viewer .stamp { position: absolute; right: 12px; bottom: 12px; font-size: 11px; color: #c7d5e0;
|
||||
background: rgba(0,0,0,.6); padding: 3px 8px; border-radius: 2px; }
|
||||
.viewer .asleep { position: absolute; inset: auto 0 0 0; padding: 14px 18px; text-align: center; color: var(--text);
|
||||
@@ -389,6 +402,9 @@
|
||||
.grid-top, .grid-3, .and-grid, .and-col { gap: 14px; }
|
||||
.toolbar { gap: 8px; }
|
||||
.toolbar .spacer { display: none; }
|
||||
.view-actions { flex: 1 1 100%; display: grid; grid-template-columns: repeat(4, 1fr); gap: 6px; }
|
||||
.view-actions button { justify-content: center; padding: 0 4px; }
|
||||
#panelPick { flex: 1 1 100%; max-width: none; }
|
||||
.shelf-head { flex-wrap: wrap; row-gap: 8px; }
|
||||
.shelf { grid-template-columns: repeat(auto-fill, minmax(100px, 1fr)); gap: 10px; }
|
||||
.cat-grid { grid-template-columns: 1fr; }
|
||||
@@ -468,16 +484,20 @@
|
||||
<div class="toolbar">
|
||||
<div class="seg" aria-label="What to capture">
|
||||
<button data-view="headset" class="on" title="What the headset shows, via the SteamVR compositor">Headset view</button>
|
||||
<button data-view="flat" title="gamescope's 2D layer: the desktop panel and Steam's flat UI">Desktop panel</button>
|
||||
<button data-view="flat" title="The app panel in use, flat: it stays still however the wearer looks around">Desktop</button>
|
||||
</div>
|
||||
<select id="panelPick" class="small" hidden aria-label="Which panel to show"></select>
|
||||
<div class="seg" id="eyeSeg" aria-label="Crop">
|
||||
<button data-eye="left" class="on" title="One eye, like pointing a camera into one lens">Left eye</button>
|
||||
<button data-eye="both" title="Both eyes side by side">Both eyes</button>
|
||||
</div>
|
||||
<div class="spacer"></div>
|
||||
<button class="action" id="shotBtn">Capture</button>
|
||||
<button id="liveBtn" title="Keep updating: video of the headset view, or repeated captures of the desktop panel">Live</button>
|
||||
<button id="saveBtn" disabled>Save</button>
|
||||
<span class="view-actions">
|
||||
<button class="action" id="shotBtn">Capture</button>
|
||||
<button id="liveBtn" title="Keep updating, as video">Live</button>
|
||||
<button id="ctrlBtn" title="Control the Frame by tapping or clicking on the view (C)">Control</button>
|
||||
<button id="saveBtn" disabled>Save</button>
|
||||
</span>
|
||||
</div>
|
||||
<div class="viewer" id="viewer">
|
||||
<div class="empty" id="viewerEmpty">
|
||||
@@ -499,6 +519,25 @@
|
||||
<div class="asleep" id="asleep" hidden>The headset is showing a blank frame. It's probably asleep or not being worn. Put it on and capture again.</div>
|
||||
<div class="loading"><div class="spinner"></div></div>
|
||||
</div>
|
||||
<div class="ctrl-status" id="ctrlBadge" role="status" hidden></div>
|
||||
<div class="pad-keys ctrl-keys" id="ctrlKeys" hidden aria-label="Keys and clicks for the Frame">
|
||||
<input id="ctrlType" type="text" placeholder="Type here to type on the Frame" autocomplete="off" autocapitalize="off"
|
||||
autocorrect="off" spellcheck="false" enterkeyhint="send">
|
||||
<span class="seg pad-clicks">
|
||||
<button data-ctrl-click="left" title="Left click where the pointer is">Click</button>
|
||||
<button data-ctrl-click="right" title="Right click where the pointer is">Right-click</button>
|
||||
</span>
|
||||
<span class="seg pad-special">
|
||||
<button data-ctrl-key="1" title="Escape">Esc</button>
|
||||
<button data-ctrl-key="15" title="Tab">Tab</button>
|
||||
<button data-ctrl-key="105" title="Left arrow" aria-label="Left arrow">←</button>
|
||||
<button data-ctrl-key="103" title="Up arrow" aria-label="Up arrow">↑</button>
|
||||
<button data-ctrl-key="108" title="Down arrow" aria-label="Down arrow">↓</button>
|
||||
<button data-ctrl-key="106" title="Right arrow" aria-label="Right arrow">→</button>
|
||||
<button data-ctrl-key="14" title="Backspace" aria-label="Backspace">⌫</button>
|
||||
<button data-ctrl-key="28" title="Enter">Enter</button>
|
||||
</span>
|
||||
</div>
|
||||
<div class="hint" id="viewHint"></div>
|
||||
</section>
|
||||
|
||||
@@ -714,6 +753,31 @@
|
||||
<div class="hint">Clipboard needs the desktop panel open in the headset.</div>
|
||||
</section>
|
||||
|
||||
<section class="panel" id="media">
|
||||
<div class="shelf-head"><h2>Media in the headset</h2><span class="count">Preview</span></div>
|
||||
<p class="sub">Play your movies and stereo photos with Frame Control's own player.</p>
|
||||
<div class="row"><button class="small" id="mediaChoose">Send media…</button>
|
||||
<button class="small" id="mediaRefresh">Refresh</button></div>
|
||||
<input type="file" id="mediaInput" accept=".mp4,.mkv,.mov,.m4v,.webm,.png,.jpg,.jpeg,.splat" hidden>
|
||||
<div class="row" style="margin-top:12px"><label for="mediaFile">On the Frame</label>
|
||||
<select id="mediaFile"><option value="">Refresh to load your media</option></select></div>
|
||||
<div class="row" style="margin-top:8px"><label for="mediaLayout">Layout</label>
|
||||
<select id="mediaLayout">
|
||||
<option value="auto">Detect from name or metadata</option><option value="mono">2D</option>
|
||||
<option value="sbs">Half SBS · left / right</option><option value="ou">Half OU · top / bottom</option>
|
||||
<option value="full-sbs">Full SBS · left / right</option><option value="full-ou">Full OU · top / bottom</option>
|
||||
</select></div>
|
||||
<div class="row" style="margin-top:8px"><label><input type="checkbox" id="mediaTheatre" checked>
|
||||
Theatre · bigger screen, dark surround</label></div>
|
||||
<div class="row" style="margin-top:12px"><button class="action small" id="mediaPlay">Play</button>
|
||||
<button class="small" id="mediaStop">Stop</button></div>
|
||||
<p class="sub" id="mediaState" role="status" aria-live="polite">Choose a file to begin.</p>
|
||||
<div class="hint">H.264/H.265 video; SBS/OU PNG or JPEG photos. Screens follow your head.
|
||||
Small .splat files show a stationary stereo preview (up to 20,000 Gaussians).
|
||||
Native HEIC/MPO photos, large splat scenes, seeking and subtitles aren't supported yet.
|
||||
Stop works from here; playback also ends after four hours.</div>
|
||||
</section>
|
||||
|
||||
<section class="panel" id="apps">
|
||||
<div class="shelf-head"><h2>Linux apps</h2><span class="count">Flatpak</span></div>
|
||||
<div class="list" id="flatpaks"><div class="sub">Loading…</div></div>
|
||||
@@ -917,9 +981,9 @@ const QUICK = [["Remmina", "org.remmina.Remmina"], ["Moonlight", "com.moonlight_
|
||||
const CDN = "https://cdn.cloudflare.steamstatic.com/steam/apps";
|
||||
const HINTS = {
|
||||
headset: "What the lenses show, composited by SteamVR: the room, floating panels, dashboard and controllers. Live streams it as video (one eye, about 30 fps); Capture takes a still of both eyes. Captures show everything on screen, including anything private.",
|
||||
flat: "gamescope's 2D layer: the desktop panel and Steam's flat UI, without the room or VR scene.",
|
||||
flat: "The app panel in use in the headset, flat and still: it doesn't move as the wearer looks around. Pick another panel to watch it. Control lets you tap or click right where you want on it.",
|
||||
};
|
||||
const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", shot: "Screenshot" };
|
||||
const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", panel: "Desktop", shot: "Screenshot" };
|
||||
let state = null, view = "headset", eye = "left", live = false, liveTimer = null, volTimer = null;
|
||||
let lastImg = null, lastSource = null, lastShot = null, viewGen = 0;
|
||||
|
||||
@@ -1179,8 +1243,12 @@ function setView(v) {
|
||||
view = v;
|
||||
document.querySelectorAll("[data-view]").forEach(b => b.classList.toggle("on", b.dataset.view === v));
|
||||
$("eyeSeg").style.visibility = v === "headset" && !video.ctl ? "visible" : "hidden";
|
||||
$("eyeSeg").style.display = v === "flat" ? "none" : ""; // the Desktop view has no eyes to pick
|
||||
$("viewHint").textContent = HINTS[v];
|
||||
if (live && changed) { toggleLive(false); toggleLive(true); } // video for the headset, captures for the panel
|
||||
$("panelPick").hidden = v !== "flat";
|
||||
if (v === "flat") watchPanels(); else clearTimeout(desk.timer);
|
||||
if (live && changed) { toggleLive(false); toggleLive(true); }
|
||||
ctrlShow();
|
||||
}
|
||||
function setEye(e) {
|
||||
eye = e;
|
||||
@@ -1207,20 +1275,27 @@ function setZoom(z, cx, cy) {
|
||||
$("zoomIn").onclick = () => setZoom(zoom * 1.5);
|
||||
$("zoomOut").onclick = () => setZoom(zoom / 1.5);
|
||||
$("zoomFit").onclick = () => setZoom(1);
|
||||
$("fullBtn").onclick = () => document.fullscreenElement ? document.exitFullscreen() : $("viewer").requestFullscreen();
|
||||
// Full screen, or on phones (no element full screen there) the viewer filling the window.
|
||||
$("fullBtn").onclick = () => {
|
||||
if (document.fullscreenElement) return document.exitFullscreen();
|
||||
if ($("viewer").requestFullscreen && !TOUCH) return $("viewer").requestFullscreen().catch(() => $("viewer").classList.toggle("immersive"));
|
||||
$("viewer").classList.toggle("immersive");
|
||||
setZoom(1);
|
||||
};
|
||||
document.addEventListener("fullscreenchange", () => setZoom(1));
|
||||
$("viewer").addEventListener("wheel", e => {
|
||||
if ($("canvas").hidden) return;
|
||||
if ($("canvas").hidden || ctrl.on) return;
|
||||
e.preventDefault();
|
||||
const v = $("viewer").getBoundingClientRect();
|
||||
setZoom(zoom * Math.exp(-e.deltaY * 0.002), e.clientX - v.left, e.clientY - v.top);
|
||||
}, { passive: false });
|
||||
$("canvas").addEventListener("dblclick", e => {
|
||||
if (ctrl.on) return;
|
||||
const v = $("viewer").getBoundingClientRect();
|
||||
setZoom(zoom > 1 ? 1 : 2.5, e.clientX - v.left, e.clientY - v.top);
|
||||
});
|
||||
$("canvas").addEventListener("pointerdown", e => {
|
||||
if (zoom === 1) return;
|
||||
if (zoom === 1 || ctrl.on) return;
|
||||
const start = { x: e.clientX - panX, y: e.clientY - panY };
|
||||
$("viewer").classList.add("panning");
|
||||
$("canvas").setPointerCapture(e.pointerId);
|
||||
@@ -1230,7 +1305,10 @@ $("canvas").addEventListener("pointerdown", e => {
|
||||
$("canvas").addEventListener("pointerup", up, { once: true });
|
||||
});
|
||||
document.addEventListener("keydown", e => {
|
||||
if (page !== "home" || $("canvas").hidden || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
|
||||
if (page !== "home" || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
|
||||
if (ctrl.on && document.activeElement === $("viewer")) return; // keys are going to the Frame
|
||||
if (e.key === "c") return $("ctrlBtn").click();
|
||||
if ($("canvas").hidden) return;
|
||||
if (e.key === "+" || e.key === "=") setZoom(zoom * 1.5);
|
||||
else if (e.key === "-") setZoom(zoom / 1.5);
|
||||
else if (e.key === "0") setZoom(1);
|
||||
@@ -1272,8 +1350,14 @@ async function capture() {
|
||||
if (!live) $("viewer").classList.add("busy"); // no spinner flashing over a live stream
|
||||
let url = null;
|
||||
try {
|
||||
const r = await fetch(view === "headset" ? "/api/screenshot?view=headset" : "/api/screenshot",
|
||||
{ headers: {"X-Frame-UI": UI_KEY} });
|
||||
let url_ = "/api/screenshot?view=headset";
|
||||
var shotPanel = null;
|
||||
if (view === "flat") {
|
||||
shotPanel = deskPanel() || (await loadPanels(), deskPanel());
|
||||
if (!shotPanel) throw new Error("no app panel is open in the headset");
|
||||
url_ = `/api/screenshot?view=panel&window=${shotPanel.window}&display=${encodeURIComponent(shotPanel.display)}`;
|
||||
}
|
||||
const r = await fetch(url_, { headers: {"X-Frame-UI": UI_KEY} });
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
|
||||
const source = r.headers.get("X-Capture-Source") || "gamescope";
|
||||
url = URL.createObjectURL(await r.blob());
|
||||
@@ -1282,6 +1366,9 @@ async function capture() {
|
||||
if (gen !== viewGen) return true;
|
||||
lastImg = img; lastSource = source; lastShot = null;
|
||||
draw();
|
||||
desk.shown = source === "panel" ? panelKey(shotPanel) : null;
|
||||
if (source === "panel") $("srcBadge").textContent = `Desktop · ${shotPanel?.name || "panel"}`;
|
||||
ctrlShow();
|
||||
$("stamp").hidden = false; $("stamp").textContent = new Date().toLocaleTimeString();
|
||||
$("saveBtn").disabled = false;
|
||||
return true;
|
||||
@@ -1294,26 +1381,29 @@ async function capture() {
|
||||
$("viewer").classList.remove("busy");
|
||||
}
|
||||
}
|
||||
let liveFailures = 0;
|
||||
async function liveLoop() {
|
||||
if (!live) return;
|
||||
liveFailures = (await capture()) ? 0 : liveFailures + 1;
|
||||
let liveFailures = 0, liveGen = 0;
|
||||
async function liveLoop(gen = liveGen) {
|
||||
if (!live || gen !== liveGen) return;
|
||||
const ok = await capture();
|
||||
if (gen !== liveGen) return; // Live was restarted (say, as video) while this capture ran
|
||||
liveFailures = ok ? 0 : liveFailures + 1;
|
||||
if (liveFailures >= 5) {
|
||||
toggleLive(false);
|
||||
toast("Live stopped after 5 failed captures. Is the headset on?", true);
|
||||
toast(view === "flat" ? "Live stopped: couldn't capture the panel." : "Live stopped after 5 failed captures. Is the headset on?", true);
|
||||
return;
|
||||
}
|
||||
if (live) liveTimer = setTimeout(liveLoop, view === "headset" ? 300 : 700);
|
||||
if (live) liveTimer = setTimeout(() => liveLoop(gen), view === "headset" ? 300 : 700);
|
||||
}
|
||||
function toggleLive(on) {
|
||||
live = on;
|
||||
liveGen++;
|
||||
liveFailures = 0;
|
||||
$("liveBtn").classList.toggle("on", live);
|
||||
$("liveBadge").hidden = !live;
|
||||
clearTimeout(liveTimer);
|
||||
stopVideo();
|
||||
if (!live) return;
|
||||
if (view === "headset" && "VideoDecoder" in window) {
|
||||
if ("VideoDecoder" in window && (view === "headset" || deskPanel())) {
|
||||
const started = startVideo();
|
||||
const ctl = video.ctl;
|
||||
started.catch(e => {
|
||||
@@ -1358,9 +1448,10 @@ async function startVideo() {
|
||||
if (!shown++) {
|
||||
viewGen++; // a capture still in flight mustn't replace the video
|
||||
lastImg = null; lastSource = "video"; lastShot = null;
|
||||
desk.shown = panelKey(panel); ctrlShow();
|
||||
$("viewer").classList.remove("busy");
|
||||
c.hidden = false; $("viewerEmpty").hidden = true; $("zoombar").hidden = false; $("asleep").hidden = true;
|
||||
$("srcBadge").hidden = false; $("srcBadge").textContent = "Headset view · video";
|
||||
$("srcBadge").hidden = false; $("srcBadge").textContent = `${video.label} · video`;
|
||||
$("stamp").hidden = false; $("saveBtn").disabled = false;
|
||||
}
|
||||
frames++;
|
||||
@@ -1372,7 +1463,11 @@ async function startVideo() {
|
||||
},
|
||||
error: e => log("Video decoder: " + e.message, "e"),
|
||||
});
|
||||
const r = await fetch(`/api/stream?${STREAM_QUERY}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
|
||||
const panel = view === "flat" ? deskPanel() : null;
|
||||
desk.shown = null; // until this stream's first picture
|
||||
video.label = panel ? `Desktop · ${panel.name || "panel"}` : "Headset view";
|
||||
const query = panel ? `src=panel&window=${panel.window}&display=${encodeURIComponent(panel.display)}&${STREAM_QUERY}` : STREAM_QUERY;
|
||||
const r = await fetch(`/api/stream?${query}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
|
||||
const reader = r.body.getReader();
|
||||
let buf = new Uint8Array(0), scan = 0, auStart = -1;
|
||||
@@ -1417,7 +1512,7 @@ function stopVideo() {
|
||||
if (video.dec && video.dec.state !== "closed") video.dec.close();
|
||||
video.ctl = video.dec = null;
|
||||
$("viewer").classList.remove("busy");
|
||||
if (lastSource === "video") { $("srcBadge").textContent = "Headset view · video (stopped)"; $("stamp").hidden = true; }
|
||||
if (lastSource === "video") { $("srcBadge").textContent = `${video.label || "Headset view"} · video (stopped)`; $("stamp").hidden = true; }
|
||||
$("eyeSeg").style.visibility = view === "headset" ? "visible" : "hidden";
|
||||
}
|
||||
$("shotBtn").onclick = () => capture();
|
||||
@@ -1437,6 +1532,421 @@ function download(blob, name) {
|
||||
document.querySelectorAll("[data-view]").forEach(b => b.onclick = () => setView(b.dataset.view));
|
||||
document.querySelectorAll("[data-eye]").forEach(b => b.onclick = () => setEye(b.dataset.eye));
|
||||
|
||||
// ---- Desktop view: the app panel in use in the headset (or one picked to watch) ----
|
||||
// gamescope says which panel has focus: the one the wearer last used. That's the one
|
||||
// Control reaches, so the view follows it unless you pick another to watch.
|
||||
// Panels are "display/window": window ids are per X display, and the Frame has two.
|
||||
// desk.shown is the panel whose picture is on screen now; Control only acts on that one.
|
||||
const desk = { panels: [], focus: null, pick: "", timer: null, loaded: false, shown: null };
|
||||
const panelKey = p => p ? `${p.display}/${p.window}` : null;
|
||||
function deskPanel() {
|
||||
const want = desk.pick || desk.focus;
|
||||
// Nothing in use and nothing picked: show the first panel rather than nothing.
|
||||
return desk.panels.find(p => panelKey(p) === want) || (!desk.pick && !desk.focus && desk.panels[0]) || null;
|
||||
}
|
||||
async function loadPanels() {
|
||||
try {
|
||||
const before = panelKey(deskPanel());
|
||||
const r = await api("/api/panels");
|
||||
desk.panels = r.panels || []; desk.loaded = true;
|
||||
desk.focus = r.focus ? `${r.focus_display}/${r.focus}` : null;
|
||||
if (desk.pick && !desk.panels.some(p => panelKey(p) === desk.pick)) desk.pick = "";
|
||||
renderPanelPick();
|
||||
const now = panelKey(deskPanel());
|
||||
if (view === "flat" && live && now !== before) { toggleLive(false); toggleLive(true); }
|
||||
ctrlShow();
|
||||
} catch (e) {
|
||||
log("Couldn't list the headset's panels: " + e.message, "e");
|
||||
}
|
||||
}
|
||||
function watchPanels() {
|
||||
clearTimeout(desk.timer);
|
||||
if (view !== "flat") return;
|
||||
loadPanels().finally(() => { if (view === "flat") desk.timer = setTimeout(watchPanels, document.hidden ? 10000 : 3000); });
|
||||
}
|
||||
function renderPanelPick() {
|
||||
const sel = $("panelPick");
|
||||
sel.innerHTML = `<option value="">${desk.focus ? "Panel in use" : "Panel in use (none yet)"}</option>` +
|
||||
desk.panels.map(p => `<option value="${esc(panelKey(p))}">${esc(p.name || "Untitled")}${p.focused ? " (in use)" : ""}</option>`).join("");
|
||||
sel.value = desk.pick;
|
||||
}
|
||||
$("panelPick").onchange = () => {
|
||||
desk.pick = $("panelPick").value;
|
||||
ctrlShow();
|
||||
if (live) { toggleLive(false); toggleLive(true); } else if (lastImg || lastSource) capture();
|
||||
};
|
||||
|
||||
// ---- Control: tap or click on the view to use the Frame ----
|
||||
// Through gamescope's own input (frame_touch.py). On the Desktop view a tap lands
|
||||
// exactly where you tapped; on the headset view, which moves with the wearer's head,
|
||||
// the view is a trackpad for the pointer instead.
|
||||
const ctrl = { on: false, queue: [], sending: false, state: "off", message: "", move: null, rel: [0, 0], raf: 0,
|
||||
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
|
||||
const EVDEV = { Escape: 1, Minus: 12, Equal: 13, Backspace: 14, Tab: 15, BracketLeft: 26, BracketRight: 27, Enter: 28,
|
||||
ControlLeft: 29, Semicolon: 39, Quote: 40, Backquote: 41, ShiftLeft: 42, Backslash: 43, Comma: 51, Period: 52,
|
||||
Slash: 53, ShiftRight: 54, NumpadMultiply: 55, AltLeft: 56, Space: 57, CapsLock: 58, F11: 87, F12: 88,
|
||||
IntlBackslash: 86, NumpadEnter: 96, ControlRight: 97, NumpadDivide: 98, AltRight: 100, Home: 102, ArrowUp: 103,
|
||||
PageUp: 104, ArrowLeft: 105, ArrowRight: 106, End: 107, ArrowDown: 108, PageDown: 109, Insert: 110, Delete: 111,
|
||||
MetaLeft: 125, MetaRight: 126, ContextMenu: 127, NumpadSubtract: 74, NumpadAdd: 78, NumpadDecimal: 83 };
|
||||
[..."qwertyuiop"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 16 + i);
|
||||
[..."asdfghjkl"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 30 + i);
|
||||
[..."zxcvbnm"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 44 + i);
|
||||
[..."1234567890"].forEach((c, i) => EVDEV["Digit" + c] = 2 + i);
|
||||
for (let i = 1; i <= 10; i++) EVDEV["F" + i] = 58 + i;
|
||||
[[7, 71], [8, 72], [9, 73], [4, 75], [5, 76], [6, 77], [1, 79], [2, 80], [3, 81], [0, 82]].forEach(([n, k]) => EVDEV["Numpad" + n] = k);
|
||||
const CTRL_BUTTONS = ["left", "middle", "right"]; // MouseEvent.button 0, 1, 2
|
||||
|
||||
function ctrlMode() { return view === "flat" ? "abs" : "rel"; }
|
||||
// Where taps go on the Desktop view, or why they can't.
|
||||
function ctrlTarget() {
|
||||
if (ctrlMode() === "rel") return { ok: true };
|
||||
const p = deskPanel();
|
||||
if (!desk.loaded) return { why: "Finding the headset's panels…" };
|
||||
if (!p) return { why: "No app panel is open in the headset." };
|
||||
if (!desk.focus) return { why: "Nobody's using a panel in the headset yet. Taps work once one is in use." };
|
||||
if (panelKey(p) !== desk.focus) return { why: "Only the panel in use takes taps. Pick “Panel in use” to control it." };
|
||||
if (desk.shown !== desk.focus) return { why: live ? "Waiting for the picture of the panel in use…"
|
||||
: "The panel in use has changed. Press Capture or Live to see it first." };
|
||||
return { ok: true, panel: p };
|
||||
}
|
||||
function ctrlShow() {
|
||||
const v = $("viewer"), b = $("ctrlBadge");
|
||||
$("ctrlBtn").classList.toggle("on", ctrl.on);
|
||||
v.classList.toggle("control", ctrl.on);
|
||||
v.classList.toggle("rel", ctrl.on && ctrlMode() === "rel");
|
||||
$("ctrlKeys").hidden = !ctrl.on;
|
||||
b.hidden = !ctrl.on;
|
||||
if (!ctrl.on) return;
|
||||
const t = ctrlTarget();
|
||||
let text = ctrlMode() === "abs" ? `Tap or click to use ${t.panel ? t.panel.name || "the panel" : "the panel"}`
|
||||
: "Drag to move the pointer, tap to click, two fingers to scroll";
|
||||
let warn = false;
|
||||
if (!t.ok) { text = t.why; warn = true; }
|
||||
else if (ctrl.state === "error") { text = ctrl.message || "Control isn't working"; warn = true; }
|
||||
else if (ctrl.state === "starting") text = "Connecting…";
|
||||
b.textContent = text;
|
||||
b.classList.toggle("warn", warn);
|
||||
}
|
||||
function ctrlToggle(on) {
|
||||
if (!on) ctrlRelease();
|
||||
ctrl.on = on;
|
||||
if (on) {
|
||||
setZoom(1);
|
||||
ctrlPoll(0);
|
||||
if (view === "flat" && !desk.loaded) watchPanels();
|
||||
if (!TOUCH) $("viewer").focus({ preventScroll: true });
|
||||
}
|
||||
if (window.frameApp && window.frameApp.captureKeys) window.frameApp.captureKeys(on && document.activeElement === $("viewer"));
|
||||
ctrlShow();
|
||||
}
|
||||
$("ctrlBtn").onclick = () => ctrlToggle(!ctrl.on);
|
||||
// Until it's ready, check on the Frame side every half second (for up to 15 s).
|
||||
async function ctrlPoll(n) {
|
||||
if (!ctrl.on) return;
|
||||
try {
|
||||
const r = await api(n ? "/api/touch" : "/api/touch?start=1");
|
||||
ctrl.state = r.state; ctrl.message = r.message || "";
|
||||
} catch (e) { ctrl.state = "error"; ctrl.message = e.message; }
|
||||
if (!ctrl.on) return; // turned off while connecting
|
||||
ctrlShow();
|
||||
if (ctrl.state !== "ready" && ctrl.state !== "error" && n < 30) return setTimeout(() => ctrlPoll(n + 1), 500);
|
||||
if (ctrl.state === "ready") return ctrlFlush();
|
||||
if (ctrl.state !== "error") {
|
||||
ctrl.state = "error"; ctrl.message = "The Frame didn't answer. Turn Control off and on to try again.";
|
||||
ctrlShow();
|
||||
}
|
||||
}
|
||||
$("viewer").tabIndex = 0;
|
||||
$("viewer").addEventListener("focus", () => { if (ctrl.on && window.frameApp?.captureKeys) window.frameApp.captureKeys(true); });
|
||||
$("viewer").addEventListener("blur", () => { ctrlRelease(); if (window.frameApp?.captureKeys) window.frameApp.captureKeys(false); });
|
||||
document.addEventListener("visibilitychange", () => { if (document.hidden) ctrlRelease(); });
|
||||
window.addEventListener("blur", () => ctrlRelease()); // another app took the keyboard
|
||||
|
||||
const isMoveEvent = e => ("fx" in e || "dx" in e) && !("button" in e) && !("key" in e);
|
||||
const isRelease = e => ("button" in e || "key" in e) && e.down === false;
|
||||
// While it connects, keep clicks and keys, each with the last position before it, so a
|
||||
// tap still lands where it was made; lone moves are stale by then.
|
||||
function ctrlKeepable(events) {
|
||||
const out = [];
|
||||
let move = null;
|
||||
for (const e of events) {
|
||||
if (isMoveEvent(e)) { if ("fx" in e) move = e; continue; }
|
||||
if (move && "button" in e && e.down !== false) out.push(move);
|
||||
move = null;
|
||||
out.push(e);
|
||||
}
|
||||
if (move) out.push(move); // the last position: a click queued after it needs it
|
||||
return out;
|
||||
}
|
||||
function ctrlSend(events) {
|
||||
ctrl.queue.push(...events.map(e => isMoveEvent(e) ? e : ctrlAim(e)).filter(Boolean));
|
||||
// Keep order; if the Frame is slow to answer, drop pointer moves first, then the oldest
|
||||
// presses and scrolls, never a release.
|
||||
if (ctrl.queue.length > 300) {
|
||||
const q = ctrlKeepable(ctrl.queue), keep = new Set();
|
||||
q.forEach((e, i) => { if (isRelease(e) || i >= q.length - 200) keep.add(i); });
|
||||
// A kept press keeps the position before it, so it lands where it was made.
|
||||
for (const i of [...keep]) if (i > 0 && "button" in q[i] && q[i].down !== false && "fx" in q[i - 1]) keep.add(i - 1);
|
||||
ctrl.queue = q.filter((e, i) => keep.has(i));
|
||||
}
|
||||
ctrlFlush();
|
||||
}
|
||||
async function ctrlFlush() {
|
||||
if (ctrl.sending || !ctrl.queue.length) return;
|
||||
// Backing off after a failure: new input waits for the same retry.
|
||||
const wait = (ctrl.retryAt || 0) - Date.now();
|
||||
if (wait > 0) {
|
||||
clearTimeout(ctrl.retry);
|
||||
ctrl.retry = setTimeout(ctrlFlush, wait);
|
||||
return;
|
||||
}
|
||||
ctrl.sending = true;
|
||||
// Up to 200 events, and at most about 100 characters of text: the Frame types a batch
|
||||
// key by key before reading the next, and a release mustn't wait behind a long paste.
|
||||
let n = 0, chars = 0;
|
||||
while (n < ctrl.queue.length && n < 200 && (n === 0 || chars + (ctrl.queue[n].text?.length || 0) <= 100)) {
|
||||
chars += ctrl.queue[n].text?.length || 0;
|
||||
n++;
|
||||
}
|
||||
const batch = ctrl.queue.splice(0, n);
|
||||
let again = 0;
|
||||
try {
|
||||
const r = await api("/api/touch", { events: batch });
|
||||
ctrl.state = r.state; ctrl.message = r.message || "";
|
||||
if (r.stale && typeof loadPanels === "function") loadPanels(); // focus moved on: catch up now
|
||||
if (!r.sent && r.state === "error") {
|
||||
// Broken: presses would be stale by the time it's back; releases still matter.
|
||||
ctrl.queue.unshift(...batch.filter(isRelease));
|
||||
again = 2000;
|
||||
} else if (!r.sent) {
|
||||
ctrl.queue.unshift(...ctrlKeepable(batch)); // connecting: send them once it's ready
|
||||
again = 300;
|
||||
}
|
||||
} catch (e) {
|
||||
ctrl.state = "error"; ctrl.message = e.message;
|
||||
ctrl.queue.unshift(...batch.filter(isRelease));
|
||||
again = 1000; // try the releases again in a moment, not in a tight loop
|
||||
} finally {
|
||||
ctrl.sending = false;
|
||||
ctrlShow();
|
||||
clearTimeout(ctrl.retry);
|
||||
ctrl.retryAt = again ? Date.now() + again : 0;
|
||||
if (again) { if (ctrl.queue.length) ctrl.retry = setTimeout(ctrlFlush, again); }
|
||||
else if (ctrl.queue.length) ctrlFlush();
|
||||
}
|
||||
}
|
||||
// Pointer moves: at most one per animation frame.
|
||||
function ctrlMoveTo(fx, fy) {
|
||||
const p = ctrlTarget().panel;
|
||||
if (!p) return;
|
||||
ctrl.move = { fx, fy, window: p.window, display: p.display };
|
||||
ctrlSchedule();
|
||||
}
|
||||
function ctrlMoveBy(dx, dy) { ctrl.rel[0] += dx; ctrl.rel[1] += dy; ctrlSchedule(); }
|
||||
function ctrlSchedule() { if (!ctrl.raf) ctrl.raf = requestAnimationFrame(ctrlFlushMoves); }
|
||||
function ctrlFlushMoves() {
|
||||
ctrl.raf = 0;
|
||||
const out = [];
|
||||
if (ctrl.move) { out.push(ctrl.move); ctrl.move = null; }
|
||||
if (ctrl.rel[0] || ctrl.rel[1]) { out.push({ dx: Math.round(ctrl.rel[0]), dy: Math.round(ctrl.rel[1]) }); ctrl.rel = [0, 0]; }
|
||||
if (out.length) ctrlSend(out);
|
||||
}
|
||||
// On the Desktop view, presses name the panel they're for, so the Frame drops them if
|
||||
// focus has moved on (releases always go).
|
||||
// A press with no panel to aim at (the Desktop view lost its target) isn't sent at all.
|
||||
function ctrlAim(e) {
|
||||
if (ctrlMode() !== "abs" || e.down === false) return e;
|
||||
const p = ctrlTarget().panel;
|
||||
return p ? { ...e, window: p.window, display: p.display } : null;
|
||||
}
|
||||
function ctrlButton(name, down) {
|
||||
ctrlFlushMoves(); // the click lands where the pointer was sent
|
||||
if (down) ctrl.held.add(name); else ctrl.held.delete(name);
|
||||
ctrlSend([ctrlAim({ button: name, down })]);
|
||||
}
|
||||
function ctrlKeyEvent(code, down) {
|
||||
ctrlFlushMoves();
|
||||
if (down) ctrl.keys.add(code); else ctrl.keys.delete(code);
|
||||
ctrlSend([ctrlAim({ key: code, down })]);
|
||||
}
|
||||
function ctrlClick(name = "left") { ctrlButton(name, true); ctrlButton(name, false); }
|
||||
function ctrlRelease() {
|
||||
for (const b of [...ctrl.held]) ctrlButton(b, false);
|
||||
for (const k of [...ctrl.keys]) ctrlKeyEvent(k, false);
|
||||
if (ctrl.g && ctrl.g.timer) clearTimeout(ctrl.g.timer);
|
||||
ctrl.pointers.clear(); ctrl.g = null;
|
||||
}
|
||||
// Where a pointer event falls on the panel, as fractions (the canvas box includes any letterboxing).
|
||||
function ctrlFraction(e) {
|
||||
const c = $("canvas"), r = c.getBoundingClientRect();
|
||||
const scale = Math.min(r.width / c.width, r.height / c.height);
|
||||
const w = c.width * scale, h = c.height * scale;
|
||||
const x = (e.clientX - r.left - (r.width - w) / 2) / w, y = (e.clientY - r.top - (r.height - h) / 2) / h;
|
||||
return x >= 0 && x <= 1 && y >= 0 && y <= 1 ? [x, y] : null;
|
||||
}
|
||||
|
||||
// Mouse (and pen) on a computer: the view is the Frame's screen.
|
||||
function ctrlMouse(e) {
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(e);
|
||||
if (f) ctrlMoveTo(...f);
|
||||
} else if (e.type === "pointermove" && (e.movementX || e.movementY)) {
|
||||
ctrlMoveBy(e.movementX * 1.5, e.movementY * 1.5);
|
||||
}
|
||||
}
|
||||
// Touch: tap to click, press and hold to right-click, two fingers to scroll. Dragging is a
|
||||
// mouse drag on the Desktop view (sliders, selecting text) and moves the pointer on the headset view.
|
||||
const TAP_MOVE = 8, HOLD_MS = 550, TOUCH_SPEED = 1.6, SCROLL_SPEED = 2;
|
||||
function ctrlTouchDown(e) {
|
||||
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
|
||||
if (ctrl.pointers.size === 1) {
|
||||
const g = ctrl.g = { kind: "maybe", x: e.clientX, y: e.clientY, e, target: ctrlAimedAt() };
|
||||
g.timer = setTimeout(() => {
|
||||
if (ctrl.g !== g || g.kind !== "maybe" || !ctrlSameTarget(g)) return;
|
||||
g.kind = "held";
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(g.e);
|
||||
if (!f) return; // on the bars around the picture: nothing to press
|
||||
ctrlMoveTo(...f);
|
||||
}
|
||||
ctrlClick("right");
|
||||
navigator.vibrate?.(10);
|
||||
}, HOLD_MS);
|
||||
} else if (ctrl.pointers.size === 2) {
|
||||
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
|
||||
if (ctrl.held.has("left")) ctrlButton("left", false);
|
||||
ctrl.g = { kind: "scroll", ...centroid() };
|
||||
}
|
||||
}
|
||||
// The panel a gesture started on; if that stops being the one in use, the gesture
|
||||
// does nothing more (except let go of what it holds).
|
||||
function ctrlAimedAt() { return ctrlMode() === "abs" ? panelKey(ctrlTarget().panel) : "headset"; }
|
||||
function ctrlSameTarget(g) { return g.target && g.target === ctrlAimedAt(); }
|
||||
function centroid() {
|
||||
const ps = [...ctrl.pointers.values()];
|
||||
return { x: ps.reduce((a, p) => a + p.x, 0) / ps.length, y: ps.reduce((a, p) => a + p.y, 0) / ps.length };
|
||||
}
|
||||
function ctrlTouchMove(e) {
|
||||
const prev = ctrl.pointers.get(e.pointerId);
|
||||
if (!prev || !ctrl.g) return;
|
||||
const g = ctrl.g, dx = e.clientX - prev.x, dy = e.clientY - prev.y;
|
||||
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
|
||||
if (g.kind === "scroll") {
|
||||
const c = centroid();
|
||||
// Content follows the fingers, as on the phone itself.
|
||||
ctrlSend([{ scroll: [-(c.x - g.x) * SCROLL_SPEED, -(c.y - g.y) * SCROLL_SPEED] }]);
|
||||
g.x = c.x; g.y = c.y;
|
||||
return;
|
||||
}
|
||||
if (g.kind === "maybe" && Math.hypot(e.clientX - g.x, e.clientY - g.y) > TAP_MOVE) {
|
||||
clearTimeout(g.timer);
|
||||
if (!ctrlSameTarget(g)) { g.kind = "done"; return; }
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(g.e);
|
||||
if (f) { ctrlMoveTo(...f); ctrlButton("left", true); }
|
||||
g.kind = "drag";
|
||||
} else g.kind = "move";
|
||||
}
|
||||
if (g.kind === "drag") { const f = ctrlFraction(e); if (f) ctrlMoveTo(...f); }
|
||||
else if (g.kind === "move") ctrlMoveBy(dx * TOUCH_SPEED, dy * TOUCH_SPEED);
|
||||
}
|
||||
function ctrlTouchUp(e) {
|
||||
const g = ctrl.g;
|
||||
ctrl.pointers.delete(e.pointerId);
|
||||
if (!g) return;
|
||||
if (g.kind === "scroll") g.kind = "done"; // the finger left behind doesn't jump the scroll or drag
|
||||
if (g.kind === "maybe" && !ctrlSameTarget(g)) g.kind = "done";
|
||||
if (g.kind === "maybe" && ctrl.pointers.size === 0) {
|
||||
clearTimeout(g.timer);
|
||||
if (ctrlMode() === "abs") { const f = ctrlFraction(g.e); if (!f) { ctrl.g = null; return; } ctrlMoveTo(...f); }
|
||||
ctrlClick("left");
|
||||
} else if (g.kind === "drag") ctrlButton("left", false);
|
||||
if (ctrl.pointers.size === 0) ctrl.g = null;
|
||||
}
|
||||
// The system took the touch (a swipe home, a call): no tap, and let go of anything held.
|
||||
function ctrlTouchCancel(e) {
|
||||
ctrl.pointers.delete(e.pointerId);
|
||||
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
|
||||
if (ctrl.held.has("left")) ctrlButton("left", false);
|
||||
if (ctrl.g) ctrl.g.kind = "done";
|
||||
if (ctrl.pointers.size === 0) ctrl.g = null;
|
||||
}
|
||||
const canvasEl = $("canvas");
|
||||
canvasEl.addEventListener("pointerdown", e => {
|
||||
if (!ctrl.on) return;
|
||||
e.preventDefault();
|
||||
$("viewer").focus({ preventScroll: true });
|
||||
try { canvasEl.setPointerCapture(e.pointerId); } catch { /* a pointer that's already gone */ }
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (e.pointerType === "touch") return ctrlTouchDown(e);
|
||||
if (ctrlMode() === "abs" && !ctrlFraction(e)) return; // on the bars around the picture
|
||||
ctrlMouse(e);
|
||||
ctrlButton(CTRL_BUTTONS[e.button] || "left", true);
|
||||
});
|
||||
canvasEl.addEventListener("pointermove", e => {
|
||||
if (!ctrl.on || !ctrlTarget().ok) return;
|
||||
if (e.pointerType === "touch") return ctrlTouchMove(e);
|
||||
ctrlMouse(e);
|
||||
});
|
||||
for (const type of ["pointerup", "pointercancel"]) canvasEl.addEventListener(type, e => {
|
||||
if (!ctrl.on) return;
|
||||
if (e.pointerType === "touch") return type === "pointercancel" ? ctrlTouchCancel(e) : ctrlTouchUp(e);
|
||||
if (type === "pointercancel") return [...ctrl.held].forEach(b => ctrlButton(b, false)); // no button is named
|
||||
const name = CTRL_BUTTONS[e.button] || "left";
|
||||
if (ctrl.held.has(name)) ctrlButton(name, false);
|
||||
});
|
||||
canvasEl.addEventListener("contextmenu", e => { if (ctrl.on) e.preventDefault(); });
|
||||
$("viewer").addEventListener("wheel", e => {
|
||||
if (!ctrl.on || !ctrlTarget().ok) return;
|
||||
e.preventDefault();
|
||||
const unit = e.deltaMode === 1 ? 40 : e.deltaMode === 2 ? 800 : 1;
|
||||
ctrlSend([{ scroll: [e.deltaX * unit, e.deltaY * unit] }]);
|
||||
}, { passive: false });
|
||||
// Keys while the view has focus go to the Frame (⌘ as Ctrl on a Mac); click elsewhere to stop.
|
||||
function ctrlKey(e, down) {
|
||||
if (!ctrl.on || document.activeElement !== $("viewer") || (down && !ctrlTarget().ok)) return;
|
||||
let code = EVDEV[e.code];
|
||||
if (IS_MAC && (e.code === "MetaLeft" || e.code === "MetaRight")) code = 29;
|
||||
if (!code) return;
|
||||
e.preventDefault();
|
||||
e.stopPropagation(); // it's for the Frame, not Frame Control's own shortcuts
|
||||
if (down && e.repeat && [29, 42, 54, 56, 97, 100, 125, 126].includes(code)) return;
|
||||
if (!down && !ctrl.keys.has(code)) return;
|
||||
ctrlKeyEvent(code, down);
|
||||
}
|
||||
$("viewer").addEventListener("keydown", e => ctrlKey(e, true));
|
||||
$("viewer").addEventListener("keyup", e => ctrlKey(e, false));
|
||||
|
||||
// Phone keyboard: the field types plain text; the buttons send single keys and clicks.
|
||||
function ctrlTap(code) { ctrlKeyEvent(code, true); ctrlKeyEvent(code, false); }
|
||||
let ctrlWarned = false;
|
||||
function ctrlText(text) {
|
||||
const plain = text.replace(/[^\x20-\x7e\n\t]/g, "");
|
||||
if (plain !== text && !ctrlWarned) {
|
||||
ctrlWarned = true;
|
||||
toast("Control types plain letters, numbers and symbols. For accents and emoji, use Keyboard and trackpad below.");
|
||||
}
|
||||
// Small pieces: the Frame types them key by key, and a release queued behind mustn't wait long.
|
||||
for (const part of plain.match(/[\s\S]{1,100}/g) || []) ctrlSend([{ text: part }]);
|
||||
}
|
||||
$("ctrlType").addEventListener("beforeinput", e => {
|
||||
if (e.isComposing) return;
|
||||
e.preventDefault();
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (e.inputType === "insertLineBreak" || e.inputType === "insertParagraph") return ctrlTap(28);
|
||||
if (e.inputType === "deleteContentBackward") return ctrlTap(14);
|
||||
if (e.inputType === "deleteContentForward") return ctrlTap(111);
|
||||
const text = e.data ?? e.dataTransfer?.getData("text/plain") ?? "";
|
||||
if (text) ctrlText(text);
|
||||
});
|
||||
$("ctrlType").addEventListener("compositionend", e => { if (e.data) ctrlText(e.data); $("ctrlType").value = ""; });
|
||||
$("ctrlKeys").addEventListener("click", e => {
|
||||
const k = e.target.closest("[data-ctrl-key]"), c = e.target.closest("[data-ctrl-click]");
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (k) ctrlTap(+k.dataset.ctrlKey);
|
||||
if (c) ctrlClick(c.dataset.ctrlClick);
|
||||
});
|
||||
|
||||
// ---- controls ----
|
||||
$("refreshAll").onclick = () => { refresh(); loadTitles(); }; // titles too: the Frame may have been asleep at start
|
||||
document.addEventListener("keydown", e => {
|
||||
@@ -1602,6 +2112,42 @@ function upload(file, mode) {
|
||||
xhr.send(file);
|
||||
});
|
||||
}
|
||||
async function refreshMedia(selectId) {
|
||||
const data = await api("/api/media", { action: "list" });
|
||||
const selected = selectId || $("mediaFile").value;
|
||||
$("mediaFile").replaceChildren();
|
||||
if (!data.files.length) $("mediaFile").add(new Option("Send media to begin", ""));
|
||||
for (const file of data.files) $("mediaFile").add(new Option(file.name, file.id));
|
||||
if (data.files.some(f => f.id === selected)) $("mediaFile").value = selected;
|
||||
const p = data.player;
|
||||
$("mediaState").textContent = p.error || `${p.state}${p.file ? ": " + p.file : ""}`;
|
||||
return data;
|
||||
}
|
||||
$("mediaChoose").onclick = () => $("mediaInput").click();
|
||||
$("mediaInput").onchange = () => act("Send media", async () => {
|
||||
const file = $("mediaInput").files[0];
|
||||
if (!file) return;
|
||||
const result = await upload(file, "media");
|
||||
await refreshMedia(result.id);
|
||||
$("mediaInput").value = "";
|
||||
return result;
|
||||
}, $("mediaChoose"));
|
||||
$("mediaRefresh").onclick = () => act("Refresh media", () => refreshMedia(), $("mediaRefresh"));
|
||||
$("mediaPlay").onclick = () => act("Play media", async () => {
|
||||
const id = $("mediaFile").value;
|
||||
if (!id) throw new Error("Send or select a media file first");
|
||||
const result = await api("/api/media", { action: "play", id,
|
||||
layout: $("mediaLayout").value, theatre: $("mediaTheatre").checked });
|
||||
$("mediaState").textContent = "Starting…";
|
||||
setTimeout(() => refreshMedia().catch(e => { $("mediaState").textContent = e.message; }), 1500);
|
||||
return result;
|
||||
}, $("mediaPlay"));
|
||||
$("mediaStop").onclick = () => act("Stop media", async () => {
|
||||
const result = await api("/api/media", { action: "stop" });
|
||||
await refreshMedia();
|
||||
return result;
|
||||
}, $("mediaStop"));
|
||||
|
||||
let apkLookup = 0;
|
||||
async function checkApkAlternatives(apk) {
|
||||
const lookup = ++apkLookup;
|
||||
@@ -2329,6 +2875,7 @@ async function openShot(s) {
|
||||
await new Promise((ok, bad) => { img.onload = ok; img.onerror = () => bad(new Error("not an image")); img.src = url; });
|
||||
if (gen !== viewGen) return;
|
||||
lastImg = img; lastSource = "shot"; lastShot = { blob, file: s.file };
|
||||
desk.shown = null; // a screenshot, not a panel: Control has nothing to act on here
|
||||
draw();
|
||||
$("srcBadge").textContent = `Screenshot · ${shotApp(s.appid)}`;
|
||||
$("stamp").hidden = false; $("stamp").textContent = new Date(s.time * 1000).toLocaleString();
|
||||
|
||||
+206
-6
@@ -46,6 +46,7 @@ import frame_apk_versions # noqa: E402
|
||||
import frame_catalog # noqa: E402
|
||||
import frame_host # noqa: E402
|
||||
import frame_macview # noqa: E402
|
||||
import frame_media # noqa: E402
|
||||
import frame_report # noqa: E402
|
||||
import frame_store # noqa: E402
|
||||
import frame_telemetry # noqa: E402
|
||||
@@ -399,6 +400,7 @@ _stream_proc = None
|
||||
|
||||
|
||||
def stream_command(query):
|
||||
"""The ffmpeg that streams H.264: the headset view, or one panel's own window (src=panel)."""
|
||||
q = parse_qs(query)
|
||||
try:
|
||||
height = int((q.get("h") or ["720"])[0])
|
||||
@@ -408,6 +410,16 @@ def stream_command(query):
|
||||
if height not in STREAM_HEIGHTS or fps not in STREAM_FPS:
|
||||
raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400)
|
||||
rate = 3 if height == 720 else 6 # Mbit/s
|
||||
if q.get("src") == ["panel"]:
|
||||
# The panel's own pixels (x11grab of its window: gamescope keeps them), fitted
|
||||
# to the height asked for. It stays still however the wearer moves their head.
|
||||
window, display = panel_target(q)
|
||||
return (f"DISPLAY={display} ffmpeg -hide_banner -loglevel error -nostdin -f x11grab -framerate {fps} "
|
||||
f"-window_id {window} -i {display} "
|
||||
f"-vf \"scale=-2:'trunc(min({height},ih)/2)*2',format=yuv420p\" -c:v libx264 -preset ultrafast "
|
||||
f"-tune zerolatency -g {fps * 2} -bf 0 -b:v {rate}M -maxrate {rate}M -bufsize {rate // 2 or 1}M "
|
||||
f"-x264-params aud=1:repeat-headers=1 -f h264 - & p=$!; "
|
||||
f"exec >&-; cat >/dev/null; kill $p 2>/dev/null; wait $p")
|
||||
return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; "
|
||||
f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} "
|
||||
f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency "
|
||||
@@ -627,7 +639,7 @@ class InputAgent:
|
||||
"""
|
||||
|
||||
def __init__(self, source=HERE / "frame_input_agent.py", packages=None):
|
||||
self.source, self.proc, self.lock = source, None, threading.Lock()
|
||||
self.source, self.proc, self.lock, self.write_lock = source, None, threading.Lock(), threading.Lock()
|
||||
self.packages = kdeconnect_packages() if packages is None else packages
|
||||
# generation counts stop()s; launching is the generation a launch is under way for.
|
||||
self.status, self.launching, self.generation = {"state": "off"}, None, 0
|
||||
@@ -800,8 +812,10 @@ class InputAgent:
|
||||
self.start()
|
||||
elif ready and events:
|
||||
try:
|
||||
proc.stdin.write((json.dumps(events) + "\n").encode())
|
||||
proc.stdin.flush()
|
||||
# One writer at a time: two devices sending at once mustn't tear a line.
|
||||
with self.write_lock:
|
||||
proc.stdin.write((json.dumps(events) + "\n").encode())
|
||||
proc.stdin.flush()
|
||||
sent = True
|
||||
except (BrokenPipeError, OSError, ValueError):
|
||||
pass # _watch reports how it ended
|
||||
@@ -841,6 +855,115 @@ def remote_input(body):
|
||||
return _input.send([input_event(e) for e in events])
|
||||
|
||||
|
||||
# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
|
||||
|
||||
PANEL_WINDOW = re.compile(r"^\d{1,10}$")
|
||||
PANEL_DISPLAY = re.compile(r"^:\d{1,2}$")
|
||||
TOUCH_BUTTONS = ("left", "right", "middle")
|
||||
|
||||
|
||||
def panels():
|
||||
"""The headset's app panels (window, display, name, size) and which has focus."""
|
||||
return json.loads(ssh("python3 - panels", stdin=(HERE / "frame_touch.py").read_text(), timeout=20))
|
||||
|
||||
|
||||
def panel_target(q):
|
||||
window, display = (q.get("window") or [""])[0], (q.get("display") or [""])[0]
|
||||
if not PANEL_WINDOW.match(window) or not PANEL_DISPLAY.match(display):
|
||||
raise Failure("window must be a window id and display an X display such as :1", 400)
|
||||
return window, display
|
||||
|
||||
|
||||
def panel_capture(query):
|
||||
"""One frame of a panel's own window, as PNG (its pixels, without the room around it)."""
|
||||
window, display = panel_target(parse_qs(query))
|
||||
return ssh(f"DISPLAY={display} timeout 10 ffmpeg -hide_banner -loglevel error -nostdin -f x11grab "
|
||||
f"-window_id {window} -i {display} -frames:v 1 -f image2pipe -c:v png -", timeout=20, text=False)
|
||||
|
||||
|
||||
def touch_event(event):
|
||||
"""A frame_touch.py event with only the fields it knows, in range."""
|
||||
if not isinstance(event, dict):
|
||||
raise Failure("each touch event must be an object", 400)
|
||||
|
||||
def num(name, limit):
|
||||
value = event.get(name)
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise Failure(f"{name} must be a number", 400)
|
||||
return max(-limit, min(limit, round(float(value), 4)))
|
||||
out = {}
|
||||
if "fx" in event or "fy" in event:
|
||||
if "window" not in event:
|
||||
raise Failure("a position needs the panel's window and display", 400)
|
||||
out.update(fx=num("fx", 1), fy=num("fy", 1))
|
||||
# Any event can name the panel it's meant for; the Frame drops it if another has focus.
|
||||
if "window" in event:
|
||||
window, display = event.get("window"), event.get("display")
|
||||
if isinstance(window, bool) or not isinstance(window, int) or window <= 0:
|
||||
raise Failure("window must be the panel's window id", 400)
|
||||
if not isinstance(display, str) or not PANEL_DISPLAY.match(display):
|
||||
raise Failure("display must be an X display such as :1", 400)
|
||||
out.update(window=window, display=display)
|
||||
for name in ("dx", "dy"):
|
||||
if name in event:
|
||||
out[name] = num(name, INPUT_MOVE_LIMIT)
|
||||
if "button" in event:
|
||||
if event["button"] not in TOUCH_BUTTONS:
|
||||
raise Failure(f"button must be one of {', '.join(TOUCH_BUTTONS)}", 400)
|
||||
out["button"], out["down"] = event["button"], event.get("down") is not False
|
||||
if "scroll" in event:
|
||||
sc = event["scroll"]
|
||||
if not isinstance(sc, list) or len(sc) != 2:
|
||||
raise Failure("scroll must be [dx, dy]", 400)
|
||||
out["scroll"] = [num_value(v, 5000) for v in sc]
|
||||
if "key" in event:
|
||||
key = event["key"]
|
||||
if isinstance(key, bool) or not isinstance(key, int) or not 0 < key < 768:
|
||||
raise Failure("key must be a Linux key code", 400)
|
||||
out["key"], out["down"] = key, event.get("down") is not False
|
||||
if "text" in event:
|
||||
text = event["text"]
|
||||
if not isinstance(text, str) or not 0 < len(text) <= INPUT_TEXT_LIMIT:
|
||||
raise Failure(f"text must be 1 to {INPUT_TEXT_LIMIT} characters", 400)
|
||||
out["text"] = text
|
||||
if not set(out) - {"window", "display"}:
|
||||
raise Failure("touch event has nothing to do", 400)
|
||||
return out
|
||||
|
||||
|
||||
def num_value(value, limit):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise Failure("scroll values must be numbers", 400)
|
||||
return max(-limit, min(limit, round(float(value), 2)))
|
||||
|
||||
|
||||
class TouchAgent(InputAgent):
|
||||
"""frame_touch.py on the Frame, fed events over one long-lived ssh. Nothing to install:
|
||||
it uses gamescope's own input socket and the libei that's on the image."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(source=HERE / "frame_touch.py", packages=[])
|
||||
|
||||
def deliver(self, report, force=False):
|
||||
return ""
|
||||
|
||||
def command(self, folder=""):
|
||||
code = base64.b64encode(self.source.read_bytes()).decode()
|
||||
return "python3 -u -c " + shlex.quote(
|
||||
f"import base64;exec(compile(base64.b64decode('{code}'),'frame_touch','exec'))")
|
||||
|
||||
|
||||
_touch = TouchAgent()
|
||||
|
||||
|
||||
def remote_touch(body):
|
||||
"""{"events": [...]} points, clicks, scrolls and types into the focused panel."""
|
||||
events = body.get("events", [])
|
||||
if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
|
||||
raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
|
||||
return _touch.send([touch_event(e) for e in events])
|
||||
|
||||
|
||||
def flatpak(body):
|
||||
app, action = str(body.get("id", "")), body.get("action")
|
||||
if not FLATPAK_ID.match(app):
|
||||
@@ -1609,6 +1732,75 @@ def _sweep_one(prefix, d):
|
||||
pass
|
||||
|
||||
|
||||
# ---- Our Frame-side media player -----------------------------------------
|
||||
|
||||
_MEDIA_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def media(body):
|
||||
action = body.get("action")
|
||||
if action not in ("list", "status", "play", "stop"):
|
||||
raise Failure("Media action must be list, status, play or stop", 400)
|
||||
if action == "play":
|
||||
identity = body.get("id")
|
||||
if not isinstance(identity, str) or not re.fullmatch(r"[0-9a-f]{32}/[^/\\\x00]+", identity):
|
||||
raise Failure("Invalid media id", 400)
|
||||
if body.get("layout", "auto") not in frame_media.LAYOUTS:
|
||||
raise Failure("Invalid media layout", 400)
|
||||
if type(body.get("theatre", False)) is not bool:
|
||||
raise Failure("theatre must be true or false", 400)
|
||||
with _MEDIA_LOCK:
|
||||
# Ship only our small stdlib modules, atomically, to the user account.
|
||||
sources = {name: (HERE / name).read_text() for name in (
|
||||
"frame_media.py", "frame_media_player.py", "frame_media_remote.py", "frame_splat.py")}
|
||||
installer = """import json, os, pathlib, sys, tempfile
|
||||
root = pathlib.Path.home()/'.local/share/frame-control/media'
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
for name, source in json.load(sys.stdin).items():
|
||||
path = root/name
|
||||
fd, temp = tempfile.mkstemp(dir=root, prefix=name+'.')
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
f.write(source)
|
||||
os.replace(temp, path)
|
||||
"""
|
||||
ssh("python3 -c " + shlex.quote(installer), stdin=json.dumps(sources))
|
||||
try:
|
||||
out = ssh("python3 ~/.local/share/frame-control/media/frame_media_remote.py",
|
||||
stdin=json.dumps(body), timeout=75) # remote worst case: ffprobe 30 + reset-failed 10 + systemd-run 15 s
|
||||
except Failure as e:
|
||||
for line in reversed(getattr(e, "stdout", "").splitlines()):
|
||||
try:
|
||||
detail = json.loads(line).get("error")
|
||||
except (ValueError, AttributeError):
|
||||
continue
|
||||
if detail:
|
||||
raise Failure(detail) from None
|
||||
raise
|
||||
return json.loads(out)
|
||||
|
||||
|
||||
def push_media(path):
|
||||
# Validate the format, but leave layout selection until playback (ffprobe
|
||||
# can then read metadata on the Frame, where it is installed).
|
||||
name = Path(path).name
|
||||
frame_media.plan(name, "mono")
|
||||
if name.startswith(".") or "\\" in name:
|
||||
raise Failure("Rename the file: media names can't start with a dot or contain a backslash", 400)
|
||||
token = secrets.token_hex(16)
|
||||
dest = "Videos/FrameControl/" + token + "/"
|
||||
ssh("mkdir -p ~/" + dest)
|
||||
try:
|
||||
push_file(path, dest)
|
||||
except Exception:
|
||||
try:
|
||||
ssh("rm -rf ~/" + dest)
|
||||
except Exception:
|
||||
pass # keep the copy error; an empty folder isn't listed as media
|
||||
raise
|
||||
return {"message": "Media sent. Choose its layout and press Play.",
|
||||
"id": token + "/" + name}
|
||||
|
||||
|
||||
# ---- Mac in the headset (frame_macview.py) ----------------------------------
|
||||
|
||||
# The tunnel gets its own connection: the shared master's options would win
|
||||
@@ -1675,9 +1867,9 @@ def agent_approval(body):
|
||||
return frame_agent.approvals.decide(body.get("confirmation"), body.get("accept"))
|
||||
|
||||
|
||||
POST = {"/api/agent/call": agent_call, "/api/agent/approval": agent_approval,
|
||||
POST = {"/api/media": media, "/api/agent/call": agent_call, "/api/agent/approval": agent_approval,
|
||||
"/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
|
||||
"/api/input": remote_input,
|
||||
"/api/input": remote_input, "/api/touch": remote_touch,
|
||||
"/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
|
||||
"/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
|
||||
"/api/webinstall/cancel": webinstall_cancel,
|
||||
@@ -1811,6 +2003,10 @@ class Handler(BaseHTTPRequestHandler):
|
||||
self.send_json(title_job(url.query))
|
||||
elif path == "/api/licenses":
|
||||
self.send_json({"notices": licenses()})
|
||||
elif path == "/api/panels":
|
||||
self.send_json(panels())
|
||||
elif path == "/api/touch":
|
||||
self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
|
||||
elif path == "/api/input":
|
||||
self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
|
||||
elif path == "/api/job":
|
||||
@@ -1842,6 +2038,8 @@ class Handler(BaseHTTPRequestHandler):
|
||||
self.send_bytes(*shot_image(url.query))
|
||||
elif path == "/api/stream":
|
||||
self.stream_video(url.query)
|
||||
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["panel"]:
|
||||
self.send_bytes(panel_capture(url.query), "image/png", headers=[("X-Capture-Source", "panel")])
|
||||
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]:
|
||||
self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")])
|
||||
elif path == "/api/screenshot":
|
||||
@@ -1918,7 +2116,7 @@ class Handler(BaseHTTPRequestHandler):
|
||||
proc.wait()
|
||||
errors.seek(0)
|
||||
err = strip_ansi(errors.read().decode(errors="replace")).strip()
|
||||
raise Failure(err or "The headset view sent no video for 20 s")
|
||||
raise Failure(err or "The Frame sent no video for 20 s")
|
||||
chunk = first
|
||||
try:
|
||||
self.send_response(200)
|
||||
@@ -1972,6 +2170,8 @@ class Handler(BaseHTTPRequestHandler):
|
||||
raise Failure("upload interrupted", 400)
|
||||
f.write(chunk)
|
||||
remaining -= len(chunk)
|
||||
if mode == "media":
|
||||
return push_media(dest)
|
||||
if mode == "apkinfo":
|
||||
# Read an APK for a report without installing it.
|
||||
try:
|
||||
|
||||
Reference in new issue
Block a user