mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 02:00:19 +02:00
Live view: a Desktop view that stays still, and Control to tap on the Frame (#46)
* Live view: a Desktop view that stays still, and Control to tap on the Frame The live view gets a second source and a way to use the Frame from it: - Desktop: the app panel in use in the headset, streamed from its own window (x11grab of gamescope's redirected window), so it doesn't move as the wearer looks around. A picker shows any other panel, view only. - Control: on the Desktop view a tap or click lands exactly where you put it; drag is a mouse drag, press and hold right-clicks, two fingers scroll, and on a computer the mouse, wheel and keyboard work directly. On the headset view the view is a trackpad. A text field and key row type from a phone. Input goes through gamescope's own EIS socket (the way Steam feeds Remote Play input) with the libei already on the image: ui/frame_touch.py, over the same long-lived ssh machinery as the keyboard agent, nothing to install. It reaches the panel that has focus on either X display, which the KDE Connect route can't. Verified on the Frame and from the iPhone app in the Simulator. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Control: fixes from review - Keys held on the Frame are released with buttons when Control stops or the view loses focus; keys for the Frame no longer trigger Frame Control's own shortcuts. - Taps only act when the picture on screen is the panel in use; positions, presses, keys, text and scrolls name their panel (display and window: ids repeat across :0 and :1, told apart by pid), and the Frame drops them if focus has moved on. Releases always go. - While connecting, a tap keeps its position; on an error only releases wait and retries back off; trimming a long queue never drops a release. - Lifting one of two scrolling fingers ends the scroll; a cancelled touch isn't a tap; clicks and holds on the bars around the picture do nothing. - A capture loop from before a Live restart can't stop the new video. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Control: close the targeting gaps from the second review - The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope packs ":1" into the first value), so a window id repeated across :0 and :1 can't be mistaken; the pid is only the fallback. - Presses, keys, text and scrolls read focus afresh on the Frame; only moves use a reading up to a second old. - A gesture remembers the panel it started on and does nothing more if that stops being the one in use; a press with no panel to aim at isn't sent. - Opening a screenshot clears the panel Control would act on; switching to another app releases held keys and buttons. - Trimming keeps a click with its position; the error backoff holds for new input too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Control: fixes from the SWE-2 Max review - The Frame side tracks keys as well as buttons and lets go of both when the session ends. - A stale tap tells the page, which re-reads the panels at once. - A paused input device waits instead of ending the session; only a disconnect does. An OS error on one event skips it. - Presses check focus with two property reads and do the full lookup only when it changed. - Writes to an agent's stdin are serialized, so two devices sending at once can't tear a line (the keyboard agent too). - Connecting gives up with a message after 15 s instead of hanging on "Connecting…"; text goes in 100-character pieces so releases don't wait behind a long paste; a cancelled mouse gesture releases what's held. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Control: stale clears, pauses reconverge, pastes split per request - The Frame says it has caught up as soon as an aimed event lands after a stale one, so the page stops re-reading the panels. - After a device pause it lets go of everything it holds (releases that arrived while paused were dropped), and waits for the device once per batch, not once per event. - The quick focus check no longer freshens the panel geometry's age. - Each request carries at most about 100 characters of text. - Turning Control off while it connects doesn't report an error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * frame_touch: build the socket path on the Frame, so Windows can import it for tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * frame_touch: any event that goes through clears the stale flag A trackpad move names no panel, so waiting for an aimed event could leave the page re-reading panels for the rest of the session; a release still aimed at the old panel doesn't count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex SouthwellandClaude Opus 5.5 authored and GitHub committed
1 parent
ef56025ad1
commit
e702adbd77
8 files changed
+1680
-32
No files matched your search
@@ -0,0 +1,446 @@
|
||||
"""Touch and direct input for the Steam Frame's panels. Frame Control's server runs this ON the Frame.
|
||||
|
||||
gamescope, the Frame's compositor, serves Valve's own input injection: an EIS
|
||||
socket (libei's server side), which Steam uses to feed it Remote Play input.
|
||||
This connects to it with libei, which is on the SteamOS image, and points,
|
||||
clicks, scrolls and types into the panel that has focus in the headset: the
|
||||
one the wearer last used. No install, and it reaches every panel, on either
|
||||
of gamescope's X displays (see docs/streaming.md).
|
||||
|
||||
python3 frame_touch.py focus print the focused panel as JSON
|
||||
python3 frame_touch.py panels print every app panel as JSON, and which has focus
|
||||
python3 frame_touch.py read events on stdin, one JSON object (or list) per line:
|
||||
{"fx": 0.5, "fy": 0.2, "window": 123, "display": ":1"}
|
||||
pointer to that fraction of that panel; any event can
|
||||
name its panel, and goes nowhere if another has focus
|
||||
{"dx": 4, "dy": -2} pointer by that much
|
||||
{"button": "left", "down": true} left, right or middle; "down" false releases
|
||||
{"scroll": [0, 120]} by pixels; positive y scrolls down
|
||||
{"key": 30, "down": true} a Linux (evdev) key code, as the page maps KeyboardEvent.code
|
||||
{"text": "hello"} printable ASCII, typed on a US layout
|
||||
|
||||
Status goes to stdout, one JSON object per line: {"state": "ready" | "error", ...}.
|
||||
Standard library only (ctypes for libei), like the rest of what runs on the Frame.
|
||||
"""
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
SOCKET = "/run/user/{uid}/gamescope-0-ei" # filled in on the Frame (Windows has no getuid; the tests import this)
|
||||
BUTTONS = {"left": 0x110, "right": 0x111, "middle": 0x112} # BTN_LEFT, BTN_RIGHT, BTN_MIDDLE
|
||||
SHIFT = 42 # KEY_LEFTSHIFT
|
||||
# Printable ASCII on a US layout: character -> (evdev key code, shifted).
|
||||
ROWS = [("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
|
||||
("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)]
|
||||
ASCII = {" ": (57, False), "\n": (28, False), "\t": (15, False)}
|
||||
for plain, shifted, first in ROWS:
|
||||
for i, (a, b) in enumerate(zip(plain, shifted)):
|
||||
ASCII[a], ASCII[b] = (first + i, False), (first + i, True)
|
||||
|
||||
# libei's event types and device capabilities (libei.h, libei 1.4).
|
||||
EV_CONNECT, EV_DISCONNECT, EV_SEAT_ADDED, EV_DEVICE_ADDED, EV_DEVICE_REMOVED = 1, 2, 3, 5, 6
|
||||
EV_DEVICE_PAUSED, EV_DEVICE_RESUMED = 7, 8
|
||||
CAP_POINTER, CAP_ABSOLUTE, CAP_KEYBOARD, CAP_SCROLL, CAP_BUTTON = 1, 2, 4, 16, 32
|
||||
|
||||
|
||||
def say(state, **more):
|
||||
print(json.dumps({"state": state, **more}), flush=True)
|
||||
|
||||
|
||||
# ---- which panel has focus -----------------------------------------------------
|
||||
|
||||
def xprop_root(display, name):
|
||||
try:
|
||||
out = subprocess.run(["xprop", "-root", name], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return []
|
||||
values = out.split("=", 1)[1] if "=" in out else ""
|
||||
return [int(v) for v in values.replace(",", " ").split() if v.isdigit()]
|
||||
|
||||
|
||||
def window_info(display, window):
|
||||
"""Name and geometry of a window on one X display, or None if it isn't there."""
|
||||
try:
|
||||
which = ["-root"] if window == "root" else ["-id", str(window)]
|
||||
out = subprocess.run(["xwininfo", *which], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
if "IsViewable" not in out:
|
||||
return None
|
||||
info = {}
|
||||
for line in out.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("xwininfo: Window id:"):
|
||||
info["name"] = line.split('"', 1)[1].rsplit('"', 1)[0] if '"' in line else ""
|
||||
for key, field in (("Absolute upper-left X:", "x"), ("Absolute upper-left Y:", "y"),
|
||||
("Width:", "width"), ("Height:", "height")):
|
||||
if line.startswith(key):
|
||||
info[field] = int(line.split(":", 1)[1])
|
||||
return info if "width" in info else None
|
||||
|
||||
|
||||
def displays():
|
||||
return sorted(f":{n[1:]}" for n in os.listdir("/tmp/.X11-unix") if n[1:].isdigit())
|
||||
|
||||
|
||||
def window_pid(display, window):
|
||||
try:
|
||||
out = subprocess.run(["xprop", "-id", str(window), "_NET_WM_PID"], env=dict(os.environ, DISPLAY=display),
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
value = out.rsplit("=", 1)[-1].strip() if "=" in out else ""
|
||||
return int(value) if value.isdigit() else None
|
||||
|
||||
|
||||
def locate(window, pid):
|
||||
"""The display a focusable window is on, with its name and geometry.
|
||||
|
||||
Window ids are per X server, so :0 and :1 can both have one; the pid gamescope
|
||||
lists with it (GAMESCOPE_FOCUSABLE_WINDOWS) tells them apart.
|
||||
"""
|
||||
found = []
|
||||
for display in displays():
|
||||
info = window_info(display, window)
|
||||
if info:
|
||||
found.append((display, info))
|
||||
if len(found) > 1 and pid:
|
||||
found = [f for f in found if window_pid(f[0], window) == pid] or found
|
||||
if not found:
|
||||
return None
|
||||
display, info = found[0]
|
||||
root = window_info(display, "root") or {}
|
||||
return {"window": window, "display": display, **info,
|
||||
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
|
||||
|
||||
|
||||
def focusable():
|
||||
"""gamescope's focusable windows as (window, app id, pid)."""
|
||||
t = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
|
||||
return [tuple(t[i:i + 3]) for i in range(0, len(t) - 2, 3)]
|
||||
|
||||
|
||||
def focus_display():
|
||||
"""The display of the focused window, from GAMESCOPE_FOCUS_DISPLAY on :0's root.
|
||||
|
||||
gamescope writes the name (":1") as 32-bit items, so its first four bytes land,
|
||||
little-endian, in the first value: 12602 is 0x313A, ":1" (steamcompmgr.cpp;
|
||||
seen 2026-09-29).
|
||||
"""
|
||||
values = xprop_root(":0", "GAMESCOPE_FOCUS_DISPLAY")
|
||||
if not values:
|
||||
return None
|
||||
name = (values[0] & 0xFFFFFFFF).to_bytes(4, "little").split(b"\0", 1)[0].decode("ascii", "replace")
|
||||
return name if name[:1] == ":" and name[1:].isdigit() else None
|
||||
|
||||
|
||||
def focus_now():
|
||||
"""Just which window and display have focus: two property reads, for checking a press."""
|
||||
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
|
||||
return (window or None, focus_display() if window else None)
|
||||
|
||||
|
||||
def focus():
|
||||
"""The panel that has focus in the headset: window, display, name and sizes (gamescope
|
||||
publishes the window and its display on :0's root)."""
|
||||
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
|
||||
if not window:
|
||||
return {"window": None}
|
||||
app, pid = next(((a, p) for w, a, p in focusable() if w == window), (None, None))
|
||||
display = focus_display()
|
||||
info = window_info(display, window) if display else None
|
||||
if info:
|
||||
root = window_info(display, "root") or {}
|
||||
panel = {"window": window, "display": display, **info,
|
||||
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
|
||||
else:
|
||||
panel = locate(window, pid) # no display published: tell them apart by pid
|
||||
return {**panel, "app": app} if panel else {"window": None}
|
||||
|
||||
|
||||
def panels():
|
||||
"""Every app panel (gamescope's focusable windows), for watching one that hasn't focus."""
|
||||
now = focus()
|
||||
found = []
|
||||
for window, app, pid in focusable():
|
||||
panel = locate(window, pid)
|
||||
if panel and panel["width"] > 1 and panel["height"] > 1 and \
|
||||
not any(f["window"] == window and f["display"] == panel["display"] for f in found):
|
||||
panel.pop("root", None)
|
||||
found.append({**panel, "app": app, "focused": (window, panel["display"]) ==
|
||||
(now.get("window"), now.get("display"))})
|
||||
return {"focus": now.get("window"), "focus_display": now.get("display"), "panels": found}
|
||||
|
||||
|
||||
def to_root(panel, fx, fy):
|
||||
"""A point given as a fraction of the panel, in the root coordinates gamescope's pointer uses.
|
||||
|
||||
gamescope fits each panel's window to its display, so a 1920x1080 window on a
|
||||
1280x720 display takes pointer positions at two thirds scale (verified 2026-09-29).
|
||||
"""
|
||||
rw, rh = panel["root"]
|
||||
w, h = panel["width"], panel["height"]
|
||||
s = min(rw / w, rh / h)
|
||||
ox, oy = (rw - w * s) / 2, (rh - h * s) / 2
|
||||
fx, fy = min(max(fx, 0.0), 1.0), min(max(fy, 0.0), 1.0)
|
||||
return ox + fx * (w * s - 1), oy + fy * (h * s - 1)
|
||||
|
||||
|
||||
# ---- gamescope's input socket ----------------------------------------------------
|
||||
|
||||
def libei():
|
||||
L = ctypes.CDLL("libei.so.1")
|
||||
vp, c = ctypes.c_void_p, ctypes
|
||||
sig = {
|
||||
"ei_new_sender": (vp, [vp]), "ei_configure_name": (None, [vp, c.c_char_p]),
|
||||
"ei_setup_backend_socket": (c.c_int, [vp, c.c_char_p]), "ei_get_fd": (c.c_int, [vp]),
|
||||
"ei_dispatch": (None, [vp]), "ei_get_event": (vp, [vp]), "ei_event_get_type": (c.c_int, [vp]),
|
||||
"ei_event_unref": (vp, [vp]), "ei_event_get_seat": (vp, [vp]), "ei_event_get_device": (vp, [vp]),
|
||||
"ei_device_has_capability": (c.c_bool, [vp, c.c_int]), "ei_now": (c.c_uint64, [vp]),
|
||||
"ei_device_start_emulating": (None, [vp, c.c_uint32]), "ei_device_stop_emulating": (None, [vp]),
|
||||
"ei_device_frame": (None, [vp, c.c_uint64]),
|
||||
"ei_device_pointer_motion": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_pointer_motion_absolute": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_button_button": (None, [vp, c.c_uint32, c.c_bool]),
|
||||
"ei_device_scroll_delta": (None, [vp, c.c_double, c.c_double]),
|
||||
"ei_device_keyboard_key": (None, [vp, c.c_uint32, c.c_bool]),
|
||||
"ei_unref": (vp, [vp]),
|
||||
}
|
||||
for name, (res, args) in sig.items():
|
||||
f = getattr(L, name)
|
||||
f.restype, f.argtypes = res, args
|
||||
return L
|
||||
|
||||
|
||||
class Gamescope:
|
||||
"""One connection to gamescope's EIS socket and its virtual input device."""
|
||||
|
||||
def __init__(self):
|
||||
self.L = L = libei()
|
||||
self.ei = L.ei_new_sender(None)
|
||||
L.ei_configure_name(self.ei, b"Frame Control")
|
||||
if L.ei_setup_backend_socket(self.ei, SOCKET.format(uid=os.getuid()).encode()) != 0:
|
||||
raise RuntimeError("Couldn't reach gamescope's input socket. Is the headset on?")
|
||||
self.fd = L.ei_get_fd(self.ei)
|
||||
self.device, self.sequence, self.held, self.keys, self.alive = None, 0, set(), set(), True
|
||||
|
||||
def pump(self, wait=0.0):
|
||||
"""Handle gamescope's events; False once it has disconnected."""
|
||||
select.select([self.fd], [], [], wait)
|
||||
self.L.ei_dispatch(self.ei)
|
||||
alive = True
|
||||
while True:
|
||||
ev = self.L.ei_get_event(self.ei)
|
||||
if not ev:
|
||||
return alive
|
||||
kind = self.L.ei_event_get_type(ev)
|
||||
if kind == EV_SEAT_ADDED:
|
||||
seat = self.L.ei_event_get_seat(ev)
|
||||
# Variadic, ending in 0 (NULL): ask for everything we send.
|
||||
self.L.ei_seat_bind_capabilities(ctypes.c_void_p(seat), *map(ctypes.c_int, (
|
||||
CAP_POINTER, CAP_ABSOLUTE, CAP_BUTTON, CAP_SCROLL, CAP_KEYBOARD, 0)))
|
||||
elif kind == EV_DEVICE_RESUMED:
|
||||
device = self.L.ei_event_get_device(ev)
|
||||
if self.L.ei_device_has_capability(device, CAP_ABSOLUTE):
|
||||
self.sequence += 1
|
||||
self.L.ei_device_start_emulating(device, self.sequence)
|
||||
self.device = device
|
||||
# Releases that arrived while it was paused were dropped: let go of
|
||||
# everything now, so the headset and this agent agree nothing is held.
|
||||
if self.held or self.keys:
|
||||
self.release_all()
|
||||
elif kind in (EV_DEVICE_PAUSED, EV_DEVICE_REMOVED):
|
||||
if self.L.ei_event_get_device(ev) == self.device:
|
||||
self.device = None
|
||||
elif kind == EV_DISCONNECT:
|
||||
self.device, alive, self.alive = None, False, False
|
||||
self.L.ei_event_unref(ev)
|
||||
|
||||
def wait_ready(self, timeout=5):
|
||||
end = time.time() + timeout
|
||||
while self.device is None and time.time() < end:
|
||||
if not self.pump(0.1):
|
||||
break
|
||||
if self.device is None:
|
||||
raise RuntimeError("gamescope closed its input socket" if not self.alive
|
||||
else "gamescope didn't offer an input device")
|
||||
|
||||
def frame(self):
|
||||
self.L.ei_device_frame(self.device, self.L.ei_now(self.ei))
|
||||
self.L.ei_dispatch(self.ei)
|
||||
|
||||
def move_to(self, x, y):
|
||||
self.L.ei_device_pointer_motion_absolute(self.device, x, y)
|
||||
self.frame()
|
||||
|
||||
def move_by(self, dx, dy):
|
||||
self.L.ei_device_pointer_motion(self.device, dx, dy)
|
||||
self.frame()
|
||||
|
||||
def button(self, name, down):
|
||||
code = BUTTONS[name]
|
||||
if down == (code in self.held):
|
||||
return # already in that state
|
||||
self.L.ei_device_button_button(self.device, code, down)
|
||||
self.frame()
|
||||
(self.held.add if down else self.held.discard)(code)
|
||||
|
||||
def scroll(self, dx, dy):
|
||||
self.L.ei_device_scroll_delta(self.device, dx, dy)
|
||||
self.frame()
|
||||
|
||||
def key(self, code, down):
|
||||
self.L.ei_device_keyboard_key(self.device, code, down)
|
||||
self.frame()
|
||||
(self.keys.add if down else self.keys.discard)(code)
|
||||
# Paced: a burst of keys can reach the app out of order (seen 2026-09-29).
|
||||
time.sleep(0.008)
|
||||
|
||||
def text(self, text):
|
||||
for ch in text:
|
||||
if ch not in ASCII:
|
||||
continue
|
||||
code, shifted = ASCII[ch]
|
||||
if shifted:
|
||||
self.key(SHIFT, True)
|
||||
self.key(code, True)
|
||||
self.key(code, False)
|
||||
if shifted:
|
||||
self.key(SHIFT, False)
|
||||
|
||||
def release_all(self):
|
||||
"""Let go of every button and key still down, so nothing stays held in the headset."""
|
||||
for code in list(self.held):
|
||||
name = next(n for n, c in BUTTONS.items() if c == code)
|
||||
self.button(name, False)
|
||||
for code in list(self.keys):
|
||||
self.key(code, False)
|
||||
|
||||
|
||||
# ---- events from the server --------------------------------------------------------
|
||||
|
||||
def events(line):
|
||||
try:
|
||||
data = json.loads(line)
|
||||
except ValueError:
|
||||
return []
|
||||
return [e for e in (data if isinstance(data, list) else [data]) if isinstance(e, dict)]
|
||||
|
||||
|
||||
def number(value, limit=100000.0):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise ValueError("not a number")
|
||||
return max(-limit, min(limit, float(value)))
|
||||
|
||||
|
||||
STALE = [False] # whether the last status said a tap went nowhere
|
||||
|
||||
|
||||
def aimed_elsewhere(event, panel):
|
||||
"""Whether an event names a panel that isn't the one with focus now."""
|
||||
if "window" not in event:
|
||||
return False
|
||||
return (panel.get("window"), panel.get("display")) != (event.get("window"), event.get("display"))
|
||||
|
||||
|
||||
def apply(gs, event, panel):
|
||||
"""Send one event; returns the focused panel it checked against (looked up at most once a second).
|
||||
|
||||
Positions and presses name the panel they were meant for. If focus has moved to
|
||||
another panel since, they go nowhere, so a tap can't land on the wrong one;
|
||||
releases always go, so nothing stays held.
|
||||
"""
|
||||
# Moves may use a focus reading up to a second old; anything that acts (a press, key,
|
||||
# text or scroll) reads it afresh, so it can't land on a panel that took focus since.
|
||||
acts = any(k in event for k in ("button", "key", "text", "scroll")) and event.get("down") is not False
|
||||
if "window" in event:
|
||||
if panel and acts and focus_now() == (panel.get("window"), panel.get("display")):
|
||||
pass # still the same panel (its geometry is re-read on the usual one-second schedule)
|
||||
elif acts or not panel or time.time() - panel.get("_at", 0) > 1 or aimed_elsewhere(event, panel):
|
||||
panel = {**focus(), "_at": time.time()}
|
||||
stale = aimed_elsewhere(event, panel) if "window" in event else False
|
||||
if stale and not (event.get("down") is False and ("button" in event or "key" in event)):
|
||||
say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True) # the page re-syncs
|
||||
STALE[0] = True
|
||||
return panel
|
||||
if STALE[0] and not stale:
|
||||
# Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
|
||||
STALE[0] = False
|
||||
say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
|
||||
if "fx" in event and panel and panel.get("window"):
|
||||
gs.move_to(*to_root(panel, number(event["fx"], 1), number(event["fy"], 1)))
|
||||
if "dx" in event or "dy" in event:
|
||||
gs.move_by(number(event.get("dx", 0), 2000), number(event.get("dy", 0), 2000))
|
||||
if event.get("button") in BUTTONS:
|
||||
gs.button(event["button"], event.get("down") is not False)
|
||||
if isinstance(event.get("scroll"), list) and len(event["scroll"]) == 2:
|
||||
gs.scroll(number(event["scroll"][0], 5000), number(event["scroll"][1], 5000))
|
||||
if isinstance(event.get("key"), int) and not isinstance(event["key"], bool) and 0 < event["key"] < 768:
|
||||
gs.key(event["key"], event.get("down") is not False)
|
||||
if isinstance(event.get("text"), str):
|
||||
gs.text(event["text"][:500])
|
||||
return panel
|
||||
|
||||
|
||||
def main():
|
||||
if sys.argv[1:] == ["focus"]:
|
||||
print(json.dumps(focus()))
|
||||
return 0
|
||||
if sys.argv[1:] == ["panels"]:
|
||||
print(json.dumps(panels()))
|
||||
return 0
|
||||
try:
|
||||
gs = Gamescope()
|
||||
gs.wait_ready()
|
||||
except (OSError, RuntimeError) as e:
|
||||
say("error", message=str(e))
|
||||
return 1
|
||||
say("ready", focus=focus().get("window"))
|
||||
stdin, pending, panel = sys.stdin.fileno(), b"", None
|
||||
try:
|
||||
while True:
|
||||
ready, _, _ = select.select([stdin, gs.fd], [], [], 30)
|
||||
if gs.fd in ready and not gs.pump():
|
||||
say("error", message="gamescope closed its input socket")
|
||||
return 1
|
||||
if stdin not in ready:
|
||||
continue
|
||||
chunk = os.read(stdin, 65536)
|
||||
if not chunk:
|
||||
return 0 # the server went away
|
||||
*lines, pending = (pending + chunk).split(b"\n")
|
||||
for line in lines:
|
||||
waited = False
|
||||
for event in events(line):
|
||||
if gs.device is None and waited:
|
||||
continue # still paused: don't wait again for each event of this batch
|
||||
if gs.device is None:
|
||||
waited = True
|
||||
# Paused (gamescope can pause the device): wait a moment; drop this
|
||||
# event if it doesn't come back. Only a disconnect ends the session.
|
||||
try:
|
||||
gs.wait_ready(2)
|
||||
except RuntimeError:
|
||||
if not gs.alive:
|
||||
raise
|
||||
continue
|
||||
try:
|
||||
panel = apply(gs, event, panel)
|
||||
except (ValueError, KeyError, TypeError, OSError):
|
||||
continue # the server checks events; skip anything odd
|
||||
except RuntimeError as e:
|
||||
say("error", message=str(e))
|
||||
return 1
|
||||
finally:
|
||||
if gs.device is not None:
|
||||
gs.release_all() # never leave a button held down in the headset
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+508
-23
@@ -147,6 +147,18 @@
|
||||
.viewer .badge { position: absolute; left: 12px; top: 12px; font-size: 11px; letter-spacing: 1.2px; text-transform: uppercase;
|
||||
background: rgba(0,0,0,.6); color: var(--text); padding: 4px 9px; border-radius: 2px; }
|
||||
.viewer .badge.live { left: auto; right: 12px; background: var(--bad); color: #fff; font-weight: 700; }
|
||||
.view-actions { display: flex; gap: 10px; }
|
||||
.ctrl-status { margin-top: 10px; font-size: 13px; color: var(--blue); }
|
||||
.ctrl-status::before { content: "● "; }
|
||||
.ctrl-status.warn { color: var(--warn); }
|
||||
/* Control: the view takes taps, clicks, drags and keys, and none of them zoom or pan it. */
|
||||
.viewer.control { outline: 1px solid var(--blue); }
|
||||
.viewer.control canvas { cursor: crosshair; touch-action: none; }
|
||||
.viewer.control.rel canvas { cursor: move; }
|
||||
.viewer.immersive { position: fixed; z-index: 40; inset: env(safe-area-inset-top) env(safe-area-inset-right) 0 env(safe-area-inset-left);
|
||||
max-height: none; aspect-ratio: auto; border-radius: 0; }
|
||||
.ctrl-keys { margin-top: 10px; }
|
||||
#panelPick { max-width: 240px; height: 30px; }
|
||||
.viewer .stamp { position: absolute; right: 12px; bottom: 12px; font-size: 11px; color: #c7d5e0;
|
||||
background: rgba(0,0,0,.6); padding: 3px 8px; border-radius: 2px; }
|
||||
.viewer .asleep { position: absolute; inset: auto 0 0 0; padding: 14px 18px; text-align: center; color: var(--text);
|
||||
@@ -390,6 +402,9 @@
|
||||
.grid-top, .grid-3, .and-grid, .and-col { gap: 14px; }
|
||||
.toolbar { gap: 8px; }
|
||||
.toolbar .spacer { display: none; }
|
||||
.view-actions { flex: 1 1 100%; display: grid; grid-template-columns: repeat(4, 1fr); gap: 6px; }
|
||||
.view-actions button { justify-content: center; padding: 0 4px; }
|
||||
#panelPick { flex: 1 1 100%; max-width: none; }
|
||||
.shelf-head { flex-wrap: wrap; row-gap: 8px; }
|
||||
.shelf { grid-template-columns: repeat(auto-fill, minmax(100px, 1fr)); gap: 10px; }
|
||||
.cat-grid { grid-template-columns: 1fr; }
|
||||
@@ -469,16 +484,20 @@
|
||||
<div class="toolbar">
|
||||
<div class="seg" aria-label="What to capture">
|
||||
<button data-view="headset" class="on" title="What the headset shows, via the SteamVR compositor">Headset view</button>
|
||||
<button data-view="flat" title="gamescope's 2D layer: the desktop panel and Steam's flat UI">Desktop panel</button>
|
||||
<button data-view="flat" title="The app panel in use, flat: it stays still however the wearer looks around">Desktop</button>
|
||||
</div>
|
||||
<select id="panelPick" class="small" hidden aria-label="Which panel to show"></select>
|
||||
<div class="seg" id="eyeSeg" aria-label="Crop">
|
||||
<button data-eye="left" class="on" title="One eye, like pointing a camera into one lens">Left eye</button>
|
||||
<button data-eye="both" title="Both eyes side by side">Both eyes</button>
|
||||
</div>
|
||||
<div class="spacer"></div>
|
||||
<button class="action" id="shotBtn">Capture</button>
|
||||
<button id="liveBtn" title="Keep updating: video of the headset view, or repeated captures of the desktop panel">Live</button>
|
||||
<button id="saveBtn" disabled>Save</button>
|
||||
<span class="view-actions">
|
||||
<button class="action" id="shotBtn">Capture</button>
|
||||
<button id="liveBtn" title="Keep updating, as video">Live</button>
|
||||
<button id="ctrlBtn" title="Control the Frame by tapping or clicking on the view (C)">Control</button>
|
||||
<button id="saveBtn" disabled>Save</button>
|
||||
</span>
|
||||
</div>
|
||||
<div class="viewer" id="viewer">
|
||||
<div class="empty" id="viewerEmpty">
|
||||
@@ -500,6 +519,25 @@
|
||||
<div class="asleep" id="asleep" hidden>The headset is showing a blank frame. It's probably asleep or not being worn. Put it on and capture again.</div>
|
||||
<div class="loading"><div class="spinner"></div></div>
|
||||
</div>
|
||||
<div class="ctrl-status" id="ctrlBadge" role="status" hidden></div>
|
||||
<div class="pad-keys ctrl-keys" id="ctrlKeys" hidden aria-label="Keys and clicks for the Frame">
|
||||
<input id="ctrlType" type="text" placeholder="Type here to type on the Frame" autocomplete="off" autocapitalize="off"
|
||||
autocorrect="off" spellcheck="false" enterkeyhint="send">
|
||||
<span class="seg pad-clicks">
|
||||
<button data-ctrl-click="left" title="Left click where the pointer is">Click</button>
|
||||
<button data-ctrl-click="right" title="Right click where the pointer is">Right-click</button>
|
||||
</span>
|
||||
<span class="seg pad-special">
|
||||
<button data-ctrl-key="1" title="Escape">Esc</button>
|
||||
<button data-ctrl-key="15" title="Tab">Tab</button>
|
||||
<button data-ctrl-key="105" title="Left arrow" aria-label="Left arrow">←</button>
|
||||
<button data-ctrl-key="103" title="Up arrow" aria-label="Up arrow">↑</button>
|
||||
<button data-ctrl-key="108" title="Down arrow" aria-label="Down arrow">↓</button>
|
||||
<button data-ctrl-key="106" title="Right arrow" aria-label="Right arrow">→</button>
|
||||
<button data-ctrl-key="14" title="Backspace" aria-label="Backspace">⌫</button>
|
||||
<button data-ctrl-key="28" title="Enter">Enter</button>
|
||||
</span>
|
||||
</div>
|
||||
<div class="hint" id="viewHint"></div>
|
||||
</section>
|
||||
|
||||
@@ -943,9 +981,9 @@ const QUICK = [["Remmina", "org.remmina.Remmina"], ["Moonlight", "com.moonlight_
|
||||
const CDN = "https://cdn.cloudflare.steamstatic.com/steam/apps";
|
||||
const HINTS = {
|
||||
headset: "What the lenses show, composited by SteamVR: the room, floating panels, dashboard and controllers. Live streams it as video (one eye, about 30 fps); Capture takes a still of both eyes. Captures show everything on screen, including anything private.",
|
||||
flat: "gamescope's 2D layer: the desktop panel and Steam's flat UI, without the room or VR scene.",
|
||||
flat: "The app panel in use in the headset, flat and still: it doesn't move as the wearer looks around. Pick another panel to watch it. Control lets you tap or click right where you want on it.",
|
||||
};
|
||||
const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", shot: "Screenshot" };
|
||||
const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", panel: "Desktop", shot: "Screenshot" };
|
||||
let state = null, view = "headset", eye = "left", live = false, liveTimer = null, volTimer = null;
|
||||
let lastImg = null, lastSource = null, lastShot = null, viewGen = 0;
|
||||
|
||||
@@ -1205,8 +1243,12 @@ function setView(v) {
|
||||
view = v;
|
||||
document.querySelectorAll("[data-view]").forEach(b => b.classList.toggle("on", b.dataset.view === v));
|
||||
$("eyeSeg").style.visibility = v === "headset" && !video.ctl ? "visible" : "hidden";
|
||||
$("eyeSeg").style.display = v === "flat" ? "none" : ""; // the Desktop view has no eyes to pick
|
||||
$("viewHint").textContent = HINTS[v];
|
||||
if (live && changed) { toggleLive(false); toggleLive(true); } // video for the headset, captures for the panel
|
||||
$("panelPick").hidden = v !== "flat";
|
||||
if (v === "flat") watchPanels(); else clearTimeout(desk.timer);
|
||||
if (live && changed) { toggleLive(false); toggleLive(true); }
|
||||
ctrlShow();
|
||||
}
|
||||
function setEye(e) {
|
||||
eye = e;
|
||||
@@ -1233,20 +1275,27 @@ function setZoom(z, cx, cy) {
|
||||
$("zoomIn").onclick = () => setZoom(zoom * 1.5);
|
||||
$("zoomOut").onclick = () => setZoom(zoom / 1.5);
|
||||
$("zoomFit").onclick = () => setZoom(1);
|
||||
$("fullBtn").onclick = () => document.fullscreenElement ? document.exitFullscreen() : $("viewer").requestFullscreen();
|
||||
// Full screen, or on phones (no element full screen there) the viewer filling the window.
|
||||
$("fullBtn").onclick = () => {
|
||||
if (document.fullscreenElement) return document.exitFullscreen();
|
||||
if ($("viewer").requestFullscreen && !TOUCH) return $("viewer").requestFullscreen().catch(() => $("viewer").classList.toggle("immersive"));
|
||||
$("viewer").classList.toggle("immersive");
|
||||
setZoom(1);
|
||||
};
|
||||
document.addEventListener("fullscreenchange", () => setZoom(1));
|
||||
$("viewer").addEventListener("wheel", e => {
|
||||
if ($("canvas").hidden) return;
|
||||
if ($("canvas").hidden || ctrl.on) return;
|
||||
e.preventDefault();
|
||||
const v = $("viewer").getBoundingClientRect();
|
||||
setZoom(zoom * Math.exp(-e.deltaY * 0.002), e.clientX - v.left, e.clientY - v.top);
|
||||
}, { passive: false });
|
||||
$("canvas").addEventListener("dblclick", e => {
|
||||
if (ctrl.on) return;
|
||||
const v = $("viewer").getBoundingClientRect();
|
||||
setZoom(zoom > 1 ? 1 : 2.5, e.clientX - v.left, e.clientY - v.top);
|
||||
});
|
||||
$("canvas").addEventListener("pointerdown", e => {
|
||||
if (zoom === 1) return;
|
||||
if (zoom === 1 || ctrl.on) return;
|
||||
const start = { x: e.clientX - panX, y: e.clientY - panY };
|
||||
$("viewer").classList.add("panning");
|
||||
$("canvas").setPointerCapture(e.pointerId);
|
||||
@@ -1256,7 +1305,10 @@ $("canvas").addEventListener("pointerdown", e => {
|
||||
$("canvas").addEventListener("pointerup", up, { once: true });
|
||||
});
|
||||
document.addEventListener("keydown", e => {
|
||||
if (page !== "home" || $("canvas").hidden || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
|
||||
if (page !== "home" || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
|
||||
if (ctrl.on && document.activeElement === $("viewer")) return; // keys are going to the Frame
|
||||
if (e.key === "c") return $("ctrlBtn").click();
|
||||
if ($("canvas").hidden) return;
|
||||
if (e.key === "+" || e.key === "=") setZoom(zoom * 1.5);
|
||||
else if (e.key === "-") setZoom(zoom / 1.5);
|
||||
else if (e.key === "0") setZoom(1);
|
||||
@@ -1298,8 +1350,14 @@ async function capture() {
|
||||
if (!live) $("viewer").classList.add("busy"); // no spinner flashing over a live stream
|
||||
let url = null;
|
||||
try {
|
||||
const r = await fetch(view === "headset" ? "/api/screenshot?view=headset" : "/api/screenshot",
|
||||
{ headers: {"X-Frame-UI": UI_KEY} });
|
||||
let url_ = "/api/screenshot?view=headset";
|
||||
var shotPanel = null;
|
||||
if (view === "flat") {
|
||||
shotPanel = deskPanel() || (await loadPanels(), deskPanel());
|
||||
if (!shotPanel) throw new Error("no app panel is open in the headset");
|
||||
url_ = `/api/screenshot?view=panel&window=${shotPanel.window}&display=${encodeURIComponent(shotPanel.display)}`;
|
||||
}
|
||||
const r = await fetch(url_, { headers: {"X-Frame-UI": UI_KEY} });
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
|
||||
const source = r.headers.get("X-Capture-Source") || "gamescope";
|
||||
url = URL.createObjectURL(await r.blob());
|
||||
@@ -1308,6 +1366,9 @@ async function capture() {
|
||||
if (gen !== viewGen) return true;
|
||||
lastImg = img; lastSource = source; lastShot = null;
|
||||
draw();
|
||||
desk.shown = source === "panel" ? panelKey(shotPanel) : null;
|
||||
if (source === "panel") $("srcBadge").textContent = `Desktop · ${shotPanel?.name || "panel"}`;
|
||||
ctrlShow();
|
||||
$("stamp").hidden = false; $("stamp").textContent = new Date().toLocaleTimeString();
|
||||
$("saveBtn").disabled = false;
|
||||
return true;
|
||||
@@ -1320,26 +1381,29 @@ async function capture() {
|
||||
$("viewer").classList.remove("busy");
|
||||
}
|
||||
}
|
||||
let liveFailures = 0;
|
||||
async function liveLoop() {
|
||||
if (!live) return;
|
||||
liveFailures = (await capture()) ? 0 : liveFailures + 1;
|
||||
let liveFailures = 0, liveGen = 0;
|
||||
async function liveLoop(gen = liveGen) {
|
||||
if (!live || gen !== liveGen) return;
|
||||
const ok = await capture();
|
||||
if (gen !== liveGen) return; // Live was restarted (say, as video) while this capture ran
|
||||
liveFailures = ok ? 0 : liveFailures + 1;
|
||||
if (liveFailures >= 5) {
|
||||
toggleLive(false);
|
||||
toast("Live stopped after 5 failed captures. Is the headset on?", true);
|
||||
toast(view === "flat" ? "Live stopped: couldn't capture the panel." : "Live stopped after 5 failed captures. Is the headset on?", true);
|
||||
return;
|
||||
}
|
||||
if (live) liveTimer = setTimeout(liveLoop, view === "headset" ? 300 : 700);
|
||||
if (live) liveTimer = setTimeout(() => liveLoop(gen), view === "headset" ? 300 : 700);
|
||||
}
|
||||
function toggleLive(on) {
|
||||
live = on;
|
||||
liveGen++;
|
||||
liveFailures = 0;
|
||||
$("liveBtn").classList.toggle("on", live);
|
||||
$("liveBadge").hidden = !live;
|
||||
clearTimeout(liveTimer);
|
||||
stopVideo();
|
||||
if (!live) return;
|
||||
if (view === "headset" && "VideoDecoder" in window) {
|
||||
if ("VideoDecoder" in window && (view === "headset" || deskPanel())) {
|
||||
const started = startVideo();
|
||||
const ctl = video.ctl;
|
||||
started.catch(e => {
|
||||
@@ -1384,9 +1448,10 @@ async function startVideo() {
|
||||
if (!shown++) {
|
||||
viewGen++; // a capture still in flight mustn't replace the video
|
||||
lastImg = null; lastSource = "video"; lastShot = null;
|
||||
desk.shown = panelKey(panel); ctrlShow();
|
||||
$("viewer").classList.remove("busy");
|
||||
c.hidden = false; $("viewerEmpty").hidden = true; $("zoombar").hidden = false; $("asleep").hidden = true;
|
||||
$("srcBadge").hidden = false; $("srcBadge").textContent = "Headset view · video";
|
||||
$("srcBadge").hidden = false; $("srcBadge").textContent = `${video.label} · video`;
|
||||
$("stamp").hidden = false; $("saveBtn").disabled = false;
|
||||
}
|
||||
frames++;
|
||||
@@ -1398,7 +1463,11 @@ async function startVideo() {
|
||||
},
|
||||
error: e => log("Video decoder: " + e.message, "e"),
|
||||
});
|
||||
const r = await fetch(`/api/stream?${STREAM_QUERY}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
|
||||
const panel = view === "flat" ? deskPanel() : null;
|
||||
desk.shown = null; // until this stream's first picture
|
||||
video.label = panel ? `Desktop · ${panel.name || "panel"}` : "Headset view";
|
||||
const query = panel ? `src=panel&window=${panel.window}&display=${encodeURIComponent(panel.display)}&${STREAM_QUERY}` : STREAM_QUERY;
|
||||
const r = await fetch(`/api/stream?${query}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
|
||||
const reader = r.body.getReader();
|
||||
let buf = new Uint8Array(0), scan = 0, auStart = -1;
|
||||
@@ -1443,7 +1512,7 @@ function stopVideo() {
|
||||
if (video.dec && video.dec.state !== "closed") video.dec.close();
|
||||
video.ctl = video.dec = null;
|
||||
$("viewer").classList.remove("busy");
|
||||
if (lastSource === "video") { $("srcBadge").textContent = "Headset view · video (stopped)"; $("stamp").hidden = true; }
|
||||
if (lastSource === "video") { $("srcBadge").textContent = `${video.label || "Headset view"} · video (stopped)`; $("stamp").hidden = true; }
|
||||
$("eyeSeg").style.visibility = view === "headset" ? "visible" : "hidden";
|
||||
}
|
||||
$("shotBtn").onclick = () => capture();
|
||||
@@ -1463,6 +1532,421 @@ function download(blob, name) {
|
||||
document.querySelectorAll("[data-view]").forEach(b => b.onclick = () => setView(b.dataset.view));
|
||||
document.querySelectorAll("[data-eye]").forEach(b => b.onclick = () => setEye(b.dataset.eye));
|
||||
|
||||
// ---- Desktop view: the app panel in use in the headset (or one picked to watch) ----
|
||||
// gamescope says which panel has focus: the one the wearer last used. That's the one
|
||||
// Control reaches, so the view follows it unless you pick another to watch.
|
||||
// Panels are "display/window": window ids are per X display, and the Frame has two.
|
||||
// desk.shown is the panel whose picture is on screen now; Control only acts on that one.
|
||||
const desk = { panels: [], focus: null, pick: "", timer: null, loaded: false, shown: null };
|
||||
const panelKey = p => p ? `${p.display}/${p.window}` : null;
|
||||
function deskPanel() {
|
||||
const want = desk.pick || desk.focus;
|
||||
// Nothing in use and nothing picked: show the first panel rather than nothing.
|
||||
return desk.panels.find(p => panelKey(p) === want) || (!desk.pick && !desk.focus && desk.panels[0]) || null;
|
||||
}
|
||||
async function loadPanels() {
|
||||
try {
|
||||
const before = panelKey(deskPanel());
|
||||
const r = await api("/api/panels");
|
||||
desk.panels = r.panels || []; desk.loaded = true;
|
||||
desk.focus = r.focus ? `${r.focus_display}/${r.focus}` : null;
|
||||
if (desk.pick && !desk.panels.some(p => panelKey(p) === desk.pick)) desk.pick = "";
|
||||
renderPanelPick();
|
||||
const now = panelKey(deskPanel());
|
||||
if (view === "flat" && live && now !== before) { toggleLive(false); toggleLive(true); }
|
||||
ctrlShow();
|
||||
} catch (e) {
|
||||
log("Couldn't list the headset's panels: " + e.message, "e");
|
||||
}
|
||||
}
|
||||
function watchPanels() {
|
||||
clearTimeout(desk.timer);
|
||||
if (view !== "flat") return;
|
||||
loadPanels().finally(() => { if (view === "flat") desk.timer = setTimeout(watchPanels, document.hidden ? 10000 : 3000); });
|
||||
}
|
||||
function renderPanelPick() {
|
||||
const sel = $("panelPick");
|
||||
sel.innerHTML = `<option value="">${desk.focus ? "Panel in use" : "Panel in use (none yet)"}</option>` +
|
||||
desk.panels.map(p => `<option value="${esc(panelKey(p))}">${esc(p.name || "Untitled")}${p.focused ? " (in use)" : ""}</option>`).join("");
|
||||
sel.value = desk.pick;
|
||||
}
|
||||
$("panelPick").onchange = () => {
|
||||
desk.pick = $("panelPick").value;
|
||||
ctrlShow();
|
||||
if (live) { toggleLive(false); toggleLive(true); } else if (lastImg || lastSource) capture();
|
||||
};
|
||||
|
||||
// ---- Control: tap or click on the view to use the Frame ----
|
||||
// Through gamescope's own input (frame_touch.py). On the Desktop view a tap lands
|
||||
// exactly where you tapped; on the headset view, which moves with the wearer's head,
|
||||
// the view is a trackpad for the pointer instead.
|
||||
const ctrl = { on: false, queue: [], sending: false, state: "off", message: "", move: null, rel: [0, 0], raf: 0,
|
||||
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
|
||||
const EVDEV = { Escape: 1, Minus: 12, Equal: 13, Backspace: 14, Tab: 15, BracketLeft: 26, BracketRight: 27, Enter: 28,
|
||||
ControlLeft: 29, Semicolon: 39, Quote: 40, Backquote: 41, ShiftLeft: 42, Backslash: 43, Comma: 51, Period: 52,
|
||||
Slash: 53, ShiftRight: 54, NumpadMultiply: 55, AltLeft: 56, Space: 57, CapsLock: 58, F11: 87, F12: 88,
|
||||
IntlBackslash: 86, NumpadEnter: 96, ControlRight: 97, NumpadDivide: 98, AltRight: 100, Home: 102, ArrowUp: 103,
|
||||
PageUp: 104, ArrowLeft: 105, ArrowRight: 106, End: 107, ArrowDown: 108, PageDown: 109, Insert: 110, Delete: 111,
|
||||
MetaLeft: 125, MetaRight: 126, ContextMenu: 127, NumpadSubtract: 74, NumpadAdd: 78, NumpadDecimal: 83 };
|
||||
[..."qwertyuiop"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 16 + i);
|
||||
[..."asdfghjkl"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 30 + i);
|
||||
[..."zxcvbnm"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 44 + i);
|
||||
[..."1234567890"].forEach((c, i) => EVDEV["Digit" + c] = 2 + i);
|
||||
for (let i = 1; i <= 10; i++) EVDEV["F" + i] = 58 + i;
|
||||
[[7, 71], [8, 72], [9, 73], [4, 75], [5, 76], [6, 77], [1, 79], [2, 80], [3, 81], [0, 82]].forEach(([n, k]) => EVDEV["Numpad" + n] = k);
|
||||
const CTRL_BUTTONS = ["left", "middle", "right"]; // MouseEvent.button 0, 1, 2
|
||||
|
||||
function ctrlMode() { return view === "flat" ? "abs" : "rel"; }
|
||||
// Where taps go on the Desktop view, or why they can't.
|
||||
function ctrlTarget() {
|
||||
if (ctrlMode() === "rel") return { ok: true };
|
||||
const p = deskPanel();
|
||||
if (!desk.loaded) return { why: "Finding the headset's panels…" };
|
||||
if (!p) return { why: "No app panel is open in the headset." };
|
||||
if (!desk.focus) return { why: "Nobody's using a panel in the headset yet. Taps work once one is in use." };
|
||||
if (panelKey(p) !== desk.focus) return { why: "Only the panel in use takes taps. Pick “Panel in use” to control it." };
|
||||
if (desk.shown !== desk.focus) return { why: live ? "Waiting for the picture of the panel in use…"
|
||||
: "The panel in use has changed. Press Capture or Live to see it first." };
|
||||
return { ok: true, panel: p };
|
||||
}
|
||||
function ctrlShow() {
|
||||
const v = $("viewer"), b = $("ctrlBadge");
|
||||
$("ctrlBtn").classList.toggle("on", ctrl.on);
|
||||
v.classList.toggle("control", ctrl.on);
|
||||
v.classList.toggle("rel", ctrl.on && ctrlMode() === "rel");
|
||||
$("ctrlKeys").hidden = !ctrl.on;
|
||||
b.hidden = !ctrl.on;
|
||||
if (!ctrl.on) return;
|
||||
const t = ctrlTarget();
|
||||
let text = ctrlMode() === "abs" ? `Tap or click to use ${t.panel ? t.panel.name || "the panel" : "the panel"}`
|
||||
: "Drag to move the pointer, tap to click, two fingers to scroll";
|
||||
let warn = false;
|
||||
if (!t.ok) { text = t.why; warn = true; }
|
||||
else if (ctrl.state === "error") { text = ctrl.message || "Control isn't working"; warn = true; }
|
||||
else if (ctrl.state === "starting") text = "Connecting…";
|
||||
b.textContent = text;
|
||||
b.classList.toggle("warn", warn);
|
||||
}
|
||||
function ctrlToggle(on) {
|
||||
if (!on) ctrlRelease();
|
||||
ctrl.on = on;
|
||||
if (on) {
|
||||
setZoom(1);
|
||||
ctrlPoll(0);
|
||||
if (view === "flat" && !desk.loaded) watchPanels();
|
||||
if (!TOUCH) $("viewer").focus({ preventScroll: true });
|
||||
}
|
||||
if (window.frameApp && window.frameApp.captureKeys) window.frameApp.captureKeys(on && document.activeElement === $("viewer"));
|
||||
ctrlShow();
|
||||
}
|
||||
$("ctrlBtn").onclick = () => ctrlToggle(!ctrl.on);
|
||||
// Until it's ready, check on the Frame side every half second (for up to 15 s).
|
||||
async function ctrlPoll(n) {
|
||||
if (!ctrl.on) return;
|
||||
try {
|
||||
const r = await api(n ? "/api/touch" : "/api/touch?start=1");
|
||||
ctrl.state = r.state; ctrl.message = r.message || "";
|
||||
} catch (e) { ctrl.state = "error"; ctrl.message = e.message; }
|
||||
if (!ctrl.on) return; // turned off while connecting
|
||||
ctrlShow();
|
||||
if (ctrl.state !== "ready" && ctrl.state !== "error" && n < 30) return setTimeout(() => ctrlPoll(n + 1), 500);
|
||||
if (ctrl.state === "ready") return ctrlFlush();
|
||||
if (ctrl.state !== "error") {
|
||||
ctrl.state = "error"; ctrl.message = "The Frame didn't answer. Turn Control off and on to try again.";
|
||||
ctrlShow();
|
||||
}
|
||||
}
|
||||
$("viewer").tabIndex = 0;
|
||||
$("viewer").addEventListener("focus", () => { if (ctrl.on && window.frameApp?.captureKeys) window.frameApp.captureKeys(true); });
|
||||
$("viewer").addEventListener("blur", () => { ctrlRelease(); if (window.frameApp?.captureKeys) window.frameApp.captureKeys(false); });
|
||||
document.addEventListener("visibilitychange", () => { if (document.hidden) ctrlRelease(); });
|
||||
window.addEventListener("blur", () => ctrlRelease()); // another app took the keyboard
|
||||
|
||||
const isMoveEvent = e => ("fx" in e || "dx" in e) && !("button" in e) && !("key" in e);
|
||||
const isRelease = e => ("button" in e || "key" in e) && e.down === false;
|
||||
// While it connects, keep clicks and keys, each with the last position before it, so a
|
||||
// tap still lands where it was made; lone moves are stale by then.
|
||||
function ctrlKeepable(events) {
|
||||
const out = [];
|
||||
let move = null;
|
||||
for (const e of events) {
|
||||
if (isMoveEvent(e)) { if ("fx" in e) move = e; continue; }
|
||||
if (move && "button" in e && e.down !== false) out.push(move);
|
||||
move = null;
|
||||
out.push(e);
|
||||
}
|
||||
if (move) out.push(move); // the last position: a click queued after it needs it
|
||||
return out;
|
||||
}
|
||||
function ctrlSend(events) {
|
||||
ctrl.queue.push(...events.map(e => isMoveEvent(e) ? e : ctrlAim(e)).filter(Boolean));
|
||||
// Keep order; if the Frame is slow to answer, drop pointer moves first, then the oldest
|
||||
// presses and scrolls, never a release.
|
||||
if (ctrl.queue.length > 300) {
|
||||
const q = ctrlKeepable(ctrl.queue), keep = new Set();
|
||||
q.forEach((e, i) => { if (isRelease(e) || i >= q.length - 200) keep.add(i); });
|
||||
// A kept press keeps the position before it, so it lands where it was made.
|
||||
for (const i of [...keep]) if (i > 0 && "button" in q[i] && q[i].down !== false && "fx" in q[i - 1]) keep.add(i - 1);
|
||||
ctrl.queue = q.filter((e, i) => keep.has(i));
|
||||
}
|
||||
ctrlFlush();
|
||||
}
|
||||
async function ctrlFlush() {
|
||||
if (ctrl.sending || !ctrl.queue.length) return;
|
||||
// Backing off after a failure: new input waits for the same retry.
|
||||
const wait = (ctrl.retryAt || 0) - Date.now();
|
||||
if (wait > 0) {
|
||||
clearTimeout(ctrl.retry);
|
||||
ctrl.retry = setTimeout(ctrlFlush, wait);
|
||||
return;
|
||||
}
|
||||
ctrl.sending = true;
|
||||
// Up to 200 events, and at most about 100 characters of text: the Frame types a batch
|
||||
// key by key before reading the next, and a release mustn't wait behind a long paste.
|
||||
let n = 0, chars = 0;
|
||||
while (n < ctrl.queue.length && n < 200 && (n === 0 || chars + (ctrl.queue[n].text?.length || 0) <= 100)) {
|
||||
chars += ctrl.queue[n].text?.length || 0;
|
||||
n++;
|
||||
}
|
||||
const batch = ctrl.queue.splice(0, n);
|
||||
let again = 0;
|
||||
try {
|
||||
const r = await api("/api/touch", { events: batch });
|
||||
ctrl.state = r.state; ctrl.message = r.message || "";
|
||||
if (r.stale && typeof loadPanels === "function") loadPanels(); // focus moved on: catch up now
|
||||
if (!r.sent && r.state === "error") {
|
||||
// Broken: presses would be stale by the time it's back; releases still matter.
|
||||
ctrl.queue.unshift(...batch.filter(isRelease));
|
||||
again = 2000;
|
||||
} else if (!r.sent) {
|
||||
ctrl.queue.unshift(...ctrlKeepable(batch)); // connecting: send them once it's ready
|
||||
again = 300;
|
||||
}
|
||||
} catch (e) {
|
||||
ctrl.state = "error"; ctrl.message = e.message;
|
||||
ctrl.queue.unshift(...batch.filter(isRelease));
|
||||
again = 1000; // try the releases again in a moment, not in a tight loop
|
||||
} finally {
|
||||
ctrl.sending = false;
|
||||
ctrlShow();
|
||||
clearTimeout(ctrl.retry);
|
||||
ctrl.retryAt = again ? Date.now() + again : 0;
|
||||
if (again) { if (ctrl.queue.length) ctrl.retry = setTimeout(ctrlFlush, again); }
|
||||
else if (ctrl.queue.length) ctrlFlush();
|
||||
}
|
||||
}
|
||||
// Pointer moves: at most one per animation frame.
|
||||
function ctrlMoveTo(fx, fy) {
|
||||
const p = ctrlTarget().panel;
|
||||
if (!p) return;
|
||||
ctrl.move = { fx, fy, window: p.window, display: p.display };
|
||||
ctrlSchedule();
|
||||
}
|
||||
function ctrlMoveBy(dx, dy) { ctrl.rel[0] += dx; ctrl.rel[1] += dy; ctrlSchedule(); }
|
||||
function ctrlSchedule() { if (!ctrl.raf) ctrl.raf = requestAnimationFrame(ctrlFlushMoves); }
|
||||
function ctrlFlushMoves() {
|
||||
ctrl.raf = 0;
|
||||
const out = [];
|
||||
if (ctrl.move) { out.push(ctrl.move); ctrl.move = null; }
|
||||
if (ctrl.rel[0] || ctrl.rel[1]) { out.push({ dx: Math.round(ctrl.rel[0]), dy: Math.round(ctrl.rel[1]) }); ctrl.rel = [0, 0]; }
|
||||
if (out.length) ctrlSend(out);
|
||||
}
|
||||
// On the Desktop view, presses name the panel they're for, so the Frame drops them if
|
||||
// focus has moved on (releases always go).
|
||||
// A press with no panel to aim at (the Desktop view lost its target) isn't sent at all.
|
||||
function ctrlAim(e) {
|
||||
if (ctrlMode() !== "abs" || e.down === false) return e;
|
||||
const p = ctrlTarget().panel;
|
||||
return p ? { ...e, window: p.window, display: p.display } : null;
|
||||
}
|
||||
function ctrlButton(name, down) {
|
||||
ctrlFlushMoves(); // the click lands where the pointer was sent
|
||||
if (down) ctrl.held.add(name); else ctrl.held.delete(name);
|
||||
ctrlSend([ctrlAim({ button: name, down })]);
|
||||
}
|
||||
function ctrlKeyEvent(code, down) {
|
||||
ctrlFlushMoves();
|
||||
if (down) ctrl.keys.add(code); else ctrl.keys.delete(code);
|
||||
ctrlSend([ctrlAim({ key: code, down })]);
|
||||
}
|
||||
function ctrlClick(name = "left") { ctrlButton(name, true); ctrlButton(name, false); }
|
||||
function ctrlRelease() {
|
||||
for (const b of [...ctrl.held]) ctrlButton(b, false);
|
||||
for (const k of [...ctrl.keys]) ctrlKeyEvent(k, false);
|
||||
if (ctrl.g && ctrl.g.timer) clearTimeout(ctrl.g.timer);
|
||||
ctrl.pointers.clear(); ctrl.g = null;
|
||||
}
|
||||
// Where a pointer event falls on the panel, as fractions (the canvas box includes any letterboxing).
|
||||
function ctrlFraction(e) {
|
||||
const c = $("canvas"), r = c.getBoundingClientRect();
|
||||
const scale = Math.min(r.width / c.width, r.height / c.height);
|
||||
const w = c.width * scale, h = c.height * scale;
|
||||
const x = (e.clientX - r.left - (r.width - w) / 2) / w, y = (e.clientY - r.top - (r.height - h) / 2) / h;
|
||||
return x >= 0 && x <= 1 && y >= 0 && y <= 1 ? [x, y] : null;
|
||||
}
|
||||
|
||||
// Mouse (and pen) on a computer: the view is the Frame's screen.
|
||||
function ctrlMouse(e) {
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(e);
|
||||
if (f) ctrlMoveTo(...f);
|
||||
} else if (e.type === "pointermove" && (e.movementX || e.movementY)) {
|
||||
ctrlMoveBy(e.movementX * 1.5, e.movementY * 1.5);
|
||||
}
|
||||
}
|
||||
// Touch: tap to click, press and hold to right-click, two fingers to scroll. Dragging is a
|
||||
// mouse drag on the Desktop view (sliders, selecting text) and moves the pointer on the headset view.
|
||||
const TAP_MOVE = 8, HOLD_MS = 550, TOUCH_SPEED = 1.6, SCROLL_SPEED = 2;
|
||||
function ctrlTouchDown(e) {
|
||||
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
|
||||
if (ctrl.pointers.size === 1) {
|
||||
const g = ctrl.g = { kind: "maybe", x: e.clientX, y: e.clientY, e, target: ctrlAimedAt() };
|
||||
g.timer = setTimeout(() => {
|
||||
if (ctrl.g !== g || g.kind !== "maybe" || !ctrlSameTarget(g)) return;
|
||||
g.kind = "held";
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(g.e);
|
||||
if (!f) return; // on the bars around the picture: nothing to press
|
||||
ctrlMoveTo(...f);
|
||||
}
|
||||
ctrlClick("right");
|
||||
navigator.vibrate?.(10);
|
||||
}, HOLD_MS);
|
||||
} else if (ctrl.pointers.size === 2) {
|
||||
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
|
||||
if (ctrl.held.has("left")) ctrlButton("left", false);
|
||||
ctrl.g = { kind: "scroll", ...centroid() };
|
||||
}
|
||||
}
|
||||
// The panel a gesture started on; if that stops being the one in use, the gesture
|
||||
// does nothing more (except let go of what it holds).
|
||||
function ctrlAimedAt() { return ctrlMode() === "abs" ? panelKey(ctrlTarget().panel) : "headset"; }
|
||||
function ctrlSameTarget(g) { return g.target && g.target === ctrlAimedAt(); }
|
||||
function centroid() {
|
||||
const ps = [...ctrl.pointers.values()];
|
||||
return { x: ps.reduce((a, p) => a + p.x, 0) / ps.length, y: ps.reduce((a, p) => a + p.y, 0) / ps.length };
|
||||
}
|
||||
function ctrlTouchMove(e) {
|
||||
const prev = ctrl.pointers.get(e.pointerId);
|
||||
if (!prev || !ctrl.g) return;
|
||||
const g = ctrl.g, dx = e.clientX - prev.x, dy = e.clientY - prev.y;
|
||||
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
|
||||
if (g.kind === "scroll") {
|
||||
const c = centroid();
|
||||
// Content follows the fingers, as on the phone itself.
|
||||
ctrlSend([{ scroll: [-(c.x - g.x) * SCROLL_SPEED, -(c.y - g.y) * SCROLL_SPEED] }]);
|
||||
g.x = c.x; g.y = c.y;
|
||||
return;
|
||||
}
|
||||
if (g.kind === "maybe" && Math.hypot(e.clientX - g.x, e.clientY - g.y) > TAP_MOVE) {
|
||||
clearTimeout(g.timer);
|
||||
if (!ctrlSameTarget(g)) { g.kind = "done"; return; }
|
||||
if (ctrlMode() === "abs") {
|
||||
const f = ctrlFraction(g.e);
|
||||
if (f) { ctrlMoveTo(...f); ctrlButton("left", true); }
|
||||
g.kind = "drag";
|
||||
} else g.kind = "move";
|
||||
}
|
||||
if (g.kind === "drag") { const f = ctrlFraction(e); if (f) ctrlMoveTo(...f); }
|
||||
else if (g.kind === "move") ctrlMoveBy(dx * TOUCH_SPEED, dy * TOUCH_SPEED);
|
||||
}
|
||||
function ctrlTouchUp(e) {
|
||||
const g = ctrl.g;
|
||||
ctrl.pointers.delete(e.pointerId);
|
||||
if (!g) return;
|
||||
if (g.kind === "scroll") g.kind = "done"; // the finger left behind doesn't jump the scroll or drag
|
||||
if (g.kind === "maybe" && !ctrlSameTarget(g)) g.kind = "done";
|
||||
if (g.kind === "maybe" && ctrl.pointers.size === 0) {
|
||||
clearTimeout(g.timer);
|
||||
if (ctrlMode() === "abs") { const f = ctrlFraction(g.e); if (!f) { ctrl.g = null; return; } ctrlMoveTo(...f); }
|
||||
ctrlClick("left");
|
||||
} else if (g.kind === "drag") ctrlButton("left", false);
|
||||
if (ctrl.pointers.size === 0) ctrl.g = null;
|
||||
}
|
||||
// The system took the touch (a swipe home, a call): no tap, and let go of anything held.
|
||||
function ctrlTouchCancel(e) {
|
||||
ctrl.pointers.delete(e.pointerId);
|
||||
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
|
||||
if (ctrl.held.has("left")) ctrlButton("left", false);
|
||||
if (ctrl.g) ctrl.g.kind = "done";
|
||||
if (ctrl.pointers.size === 0) ctrl.g = null;
|
||||
}
|
||||
const canvasEl = $("canvas");
|
||||
canvasEl.addEventListener("pointerdown", e => {
|
||||
if (!ctrl.on) return;
|
||||
e.preventDefault();
|
||||
$("viewer").focus({ preventScroll: true });
|
||||
try { canvasEl.setPointerCapture(e.pointerId); } catch { /* a pointer that's already gone */ }
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (e.pointerType === "touch") return ctrlTouchDown(e);
|
||||
if (ctrlMode() === "abs" && !ctrlFraction(e)) return; // on the bars around the picture
|
||||
ctrlMouse(e);
|
||||
ctrlButton(CTRL_BUTTONS[e.button] || "left", true);
|
||||
});
|
||||
canvasEl.addEventListener("pointermove", e => {
|
||||
if (!ctrl.on || !ctrlTarget().ok) return;
|
||||
if (e.pointerType === "touch") return ctrlTouchMove(e);
|
||||
ctrlMouse(e);
|
||||
});
|
||||
for (const type of ["pointerup", "pointercancel"]) canvasEl.addEventListener(type, e => {
|
||||
if (!ctrl.on) return;
|
||||
if (e.pointerType === "touch") return type === "pointercancel" ? ctrlTouchCancel(e) : ctrlTouchUp(e);
|
||||
if (type === "pointercancel") return [...ctrl.held].forEach(b => ctrlButton(b, false)); // no button is named
|
||||
const name = CTRL_BUTTONS[e.button] || "left";
|
||||
if (ctrl.held.has(name)) ctrlButton(name, false);
|
||||
});
|
||||
canvasEl.addEventListener("contextmenu", e => { if (ctrl.on) e.preventDefault(); });
|
||||
$("viewer").addEventListener("wheel", e => {
|
||||
if (!ctrl.on || !ctrlTarget().ok) return;
|
||||
e.preventDefault();
|
||||
const unit = e.deltaMode === 1 ? 40 : e.deltaMode === 2 ? 800 : 1;
|
||||
ctrlSend([{ scroll: [e.deltaX * unit, e.deltaY * unit] }]);
|
||||
}, { passive: false });
|
||||
// Keys while the view has focus go to the Frame (⌘ as Ctrl on a Mac); click elsewhere to stop.
|
||||
function ctrlKey(e, down) {
|
||||
if (!ctrl.on || document.activeElement !== $("viewer") || (down && !ctrlTarget().ok)) return;
|
||||
let code = EVDEV[e.code];
|
||||
if (IS_MAC && (e.code === "MetaLeft" || e.code === "MetaRight")) code = 29;
|
||||
if (!code) return;
|
||||
e.preventDefault();
|
||||
e.stopPropagation(); // it's for the Frame, not Frame Control's own shortcuts
|
||||
if (down && e.repeat && [29, 42, 54, 56, 97, 100, 125, 126].includes(code)) return;
|
||||
if (!down && !ctrl.keys.has(code)) return;
|
||||
ctrlKeyEvent(code, down);
|
||||
}
|
||||
$("viewer").addEventListener("keydown", e => ctrlKey(e, true));
|
||||
$("viewer").addEventListener("keyup", e => ctrlKey(e, false));
|
||||
|
||||
// Phone keyboard: the field types plain text; the buttons send single keys and clicks.
|
||||
function ctrlTap(code) { ctrlKeyEvent(code, true); ctrlKeyEvent(code, false); }
|
||||
let ctrlWarned = false;
|
||||
function ctrlText(text) {
|
||||
const plain = text.replace(/[^\x20-\x7e\n\t]/g, "");
|
||||
if (plain !== text && !ctrlWarned) {
|
||||
ctrlWarned = true;
|
||||
toast("Control types plain letters, numbers and symbols. For accents and emoji, use Keyboard and trackpad below.");
|
||||
}
|
||||
// Small pieces: the Frame types them key by key, and a release queued behind mustn't wait long.
|
||||
for (const part of plain.match(/[\s\S]{1,100}/g) || []) ctrlSend([{ text: part }]);
|
||||
}
|
||||
$("ctrlType").addEventListener("beforeinput", e => {
|
||||
if (e.isComposing) return;
|
||||
e.preventDefault();
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (e.inputType === "insertLineBreak" || e.inputType === "insertParagraph") return ctrlTap(28);
|
||||
if (e.inputType === "deleteContentBackward") return ctrlTap(14);
|
||||
if (e.inputType === "deleteContentForward") return ctrlTap(111);
|
||||
const text = e.data ?? e.dataTransfer?.getData("text/plain") ?? "";
|
||||
if (text) ctrlText(text);
|
||||
});
|
||||
$("ctrlType").addEventListener("compositionend", e => { if (e.data) ctrlText(e.data); $("ctrlType").value = ""; });
|
||||
$("ctrlKeys").addEventListener("click", e => {
|
||||
const k = e.target.closest("[data-ctrl-key]"), c = e.target.closest("[data-ctrl-click]");
|
||||
if (!ctrlTarget().ok) return ctrlShow();
|
||||
if (k) ctrlTap(+k.dataset.ctrlKey);
|
||||
if (c) ctrlClick(c.dataset.ctrlClick);
|
||||
});
|
||||
|
||||
// ---- controls ----
|
||||
$("refreshAll").onclick = () => { refresh(); loadTitles(); }; // titles too: the Frame may have been asleep at start
|
||||
document.addEventListener("keydown", e => {
|
||||
@@ -2391,6 +2875,7 @@ async function openShot(s) {
|
||||
await new Promise((ok, bad) => { img.onload = ok; img.onerror = () => bad(new Error("not an image")); img.src = url; });
|
||||
if (gen !== viewGen) return;
|
||||
lastImg = img; lastSource = "shot"; lastShot = { blob, file: s.file };
|
||||
desk.shown = null; // a screenshot, not a panel: Control has nothing to act on here
|
||||
draw();
|
||||
$("srcBadge").textContent = `Screenshot · ${shotApp(s.appid)}`;
|
||||
$("stamp").hidden = false; $("stamp").textContent = new Date(s.time * 1000).toLocaleString();
|
||||
|
||||
+133
-5
@@ -400,6 +400,7 @@ _stream_proc = None
|
||||
|
||||
|
||||
def stream_command(query):
|
||||
"""The ffmpeg that streams H.264: the headset view, or one panel's own window (src=panel)."""
|
||||
q = parse_qs(query)
|
||||
try:
|
||||
height = int((q.get("h") or ["720"])[0])
|
||||
@@ -409,6 +410,16 @@ def stream_command(query):
|
||||
if height not in STREAM_HEIGHTS or fps not in STREAM_FPS:
|
||||
raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400)
|
||||
rate = 3 if height == 720 else 6 # Mbit/s
|
||||
if q.get("src") == ["panel"]:
|
||||
# The panel's own pixels (x11grab of its window: gamescope keeps them), fitted
|
||||
# to the height asked for. It stays still however the wearer moves their head.
|
||||
window, display = panel_target(q)
|
||||
return (f"DISPLAY={display} ffmpeg -hide_banner -loglevel error -nostdin -f x11grab -framerate {fps} "
|
||||
f"-window_id {window} -i {display} "
|
||||
f"-vf \"scale=-2:'trunc(min({height},ih)/2)*2',format=yuv420p\" -c:v libx264 -preset ultrafast "
|
||||
f"-tune zerolatency -g {fps * 2} -bf 0 -b:v {rate}M -maxrate {rate}M -bufsize {rate // 2 or 1}M "
|
||||
f"-x264-params aud=1:repeat-headers=1 -f h264 - & p=$!; "
|
||||
f"exec >&-; cat >/dev/null; kill $p 2>/dev/null; wait $p")
|
||||
return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; "
|
||||
f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} "
|
||||
f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency "
|
||||
@@ -628,7 +639,7 @@ class InputAgent:
|
||||
"""
|
||||
|
||||
def __init__(self, source=HERE / "frame_input_agent.py", packages=None):
|
||||
self.source, self.proc, self.lock = source, None, threading.Lock()
|
||||
self.source, self.proc, self.lock, self.write_lock = source, None, threading.Lock(), threading.Lock()
|
||||
self.packages = kdeconnect_packages() if packages is None else packages
|
||||
# generation counts stop()s; launching is the generation a launch is under way for.
|
||||
self.status, self.launching, self.generation = {"state": "off"}, None, 0
|
||||
@@ -801,8 +812,10 @@ class InputAgent:
|
||||
self.start()
|
||||
elif ready and events:
|
||||
try:
|
||||
proc.stdin.write((json.dumps(events) + "\n").encode())
|
||||
proc.stdin.flush()
|
||||
# One writer at a time: two devices sending at once mustn't tear a line.
|
||||
with self.write_lock:
|
||||
proc.stdin.write((json.dumps(events) + "\n").encode())
|
||||
proc.stdin.flush()
|
||||
sent = True
|
||||
except (BrokenPipeError, OSError, ValueError):
|
||||
pass # _watch reports how it ended
|
||||
@@ -842,6 +855,115 @@ def remote_input(body):
|
||||
return _input.send([input_event(e) for e in events])
|
||||
|
||||
|
||||
# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
|
||||
|
||||
PANEL_WINDOW = re.compile(r"^\d{1,10}$")
|
||||
PANEL_DISPLAY = re.compile(r"^:\d{1,2}$")
|
||||
TOUCH_BUTTONS = ("left", "right", "middle")
|
||||
|
||||
|
||||
def panels():
|
||||
"""The headset's app panels (window, display, name, size) and which has focus."""
|
||||
return json.loads(ssh("python3 - panels", stdin=(HERE / "frame_touch.py").read_text(), timeout=20))
|
||||
|
||||
|
||||
def panel_target(q):
|
||||
window, display = (q.get("window") or [""])[0], (q.get("display") or [""])[0]
|
||||
if not PANEL_WINDOW.match(window) or not PANEL_DISPLAY.match(display):
|
||||
raise Failure("window must be a window id and display an X display such as :1", 400)
|
||||
return window, display
|
||||
|
||||
|
||||
def panel_capture(query):
|
||||
"""One frame of a panel's own window, as PNG (its pixels, without the room around it)."""
|
||||
window, display = panel_target(parse_qs(query))
|
||||
return ssh(f"DISPLAY={display} timeout 10 ffmpeg -hide_banner -loglevel error -nostdin -f x11grab "
|
||||
f"-window_id {window} -i {display} -frames:v 1 -f image2pipe -c:v png -", timeout=20, text=False)
|
||||
|
||||
|
||||
def touch_event(event):
|
||||
"""A frame_touch.py event with only the fields it knows, in range."""
|
||||
if not isinstance(event, dict):
|
||||
raise Failure("each touch event must be an object", 400)
|
||||
|
||||
def num(name, limit):
|
||||
value = event.get(name)
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise Failure(f"{name} must be a number", 400)
|
||||
return max(-limit, min(limit, round(float(value), 4)))
|
||||
out = {}
|
||||
if "fx" in event or "fy" in event:
|
||||
if "window" not in event:
|
||||
raise Failure("a position needs the panel's window and display", 400)
|
||||
out.update(fx=num("fx", 1), fy=num("fy", 1))
|
||||
# Any event can name the panel it's meant for; the Frame drops it if another has focus.
|
||||
if "window" in event:
|
||||
window, display = event.get("window"), event.get("display")
|
||||
if isinstance(window, bool) or not isinstance(window, int) or window <= 0:
|
||||
raise Failure("window must be the panel's window id", 400)
|
||||
if not isinstance(display, str) or not PANEL_DISPLAY.match(display):
|
||||
raise Failure("display must be an X display such as :1", 400)
|
||||
out.update(window=window, display=display)
|
||||
for name in ("dx", "dy"):
|
||||
if name in event:
|
||||
out[name] = num(name, INPUT_MOVE_LIMIT)
|
||||
if "button" in event:
|
||||
if event["button"] not in TOUCH_BUTTONS:
|
||||
raise Failure(f"button must be one of {', '.join(TOUCH_BUTTONS)}", 400)
|
||||
out["button"], out["down"] = event["button"], event.get("down") is not False
|
||||
if "scroll" in event:
|
||||
sc = event["scroll"]
|
||||
if not isinstance(sc, list) or len(sc) != 2:
|
||||
raise Failure("scroll must be [dx, dy]", 400)
|
||||
out["scroll"] = [num_value(v, 5000) for v in sc]
|
||||
if "key" in event:
|
||||
key = event["key"]
|
||||
if isinstance(key, bool) or not isinstance(key, int) or not 0 < key < 768:
|
||||
raise Failure("key must be a Linux key code", 400)
|
||||
out["key"], out["down"] = key, event.get("down") is not False
|
||||
if "text" in event:
|
||||
text = event["text"]
|
||||
if not isinstance(text, str) or not 0 < len(text) <= INPUT_TEXT_LIMIT:
|
||||
raise Failure(f"text must be 1 to {INPUT_TEXT_LIMIT} characters", 400)
|
||||
out["text"] = text
|
||||
if not set(out) - {"window", "display"}:
|
||||
raise Failure("touch event has nothing to do", 400)
|
||||
return out
|
||||
|
||||
|
||||
def num_value(value, limit):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
|
||||
raise Failure("scroll values must be numbers", 400)
|
||||
return max(-limit, min(limit, round(float(value), 2)))
|
||||
|
||||
|
||||
class TouchAgent(InputAgent):
|
||||
"""frame_touch.py on the Frame, fed events over one long-lived ssh. Nothing to install:
|
||||
it uses gamescope's own input socket and the libei that's on the image."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(source=HERE / "frame_touch.py", packages=[])
|
||||
|
||||
def deliver(self, report, force=False):
|
||||
return ""
|
||||
|
||||
def command(self, folder=""):
|
||||
code = base64.b64encode(self.source.read_bytes()).decode()
|
||||
return "python3 -u -c " + shlex.quote(
|
||||
f"import base64;exec(compile(base64.b64decode('{code}'),'frame_touch','exec'))")
|
||||
|
||||
|
||||
_touch = TouchAgent()
|
||||
|
||||
|
||||
def remote_touch(body):
|
||||
"""{"events": [...]} points, clicks, scrolls and types into the focused panel."""
|
||||
events = body.get("events", [])
|
||||
if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
|
||||
raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
|
||||
return _touch.send([touch_event(e) for e in events])
|
||||
|
||||
|
||||
def flatpak(body):
|
||||
app, action = str(body.get("id", "")), body.get("action")
|
||||
if not FLATPAK_ID.match(app):
|
||||
@@ -1747,7 +1869,7 @@ def agent_approval(body):
|
||||
|
||||
POST = {"/api/media": media, "/api/agent/call": agent_call, "/api/agent/approval": agent_approval,
|
||||
"/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
|
||||
"/api/input": remote_input,
|
||||
"/api/input": remote_input, "/api/touch": remote_touch,
|
||||
"/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
|
||||
"/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
|
||||
"/api/webinstall/cancel": webinstall_cancel,
|
||||
@@ -1881,6 +2003,10 @@ class Handler(BaseHTTPRequestHandler):
|
||||
self.send_json(title_job(url.query))
|
||||
elif path == "/api/licenses":
|
||||
self.send_json({"notices": licenses()})
|
||||
elif path == "/api/panels":
|
||||
self.send_json(panels())
|
||||
elif path == "/api/touch":
|
||||
self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
|
||||
elif path == "/api/input":
|
||||
self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
|
||||
elif path == "/api/job":
|
||||
@@ -1912,6 +2038,8 @@ class Handler(BaseHTTPRequestHandler):
|
||||
self.send_bytes(*shot_image(url.query))
|
||||
elif path == "/api/stream":
|
||||
self.stream_video(url.query)
|
||||
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["panel"]:
|
||||
self.send_bytes(panel_capture(url.query), "image/png", headers=[("X-Capture-Source", "panel")])
|
||||
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]:
|
||||
self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")])
|
||||
elif path == "/api/screenshot":
|
||||
@@ -1988,7 +2116,7 @@ class Handler(BaseHTTPRequestHandler):
|
||||
proc.wait()
|
||||
errors.seek(0)
|
||||
err = strip_ansi(errors.read().decode(errors="replace")).strip()
|
||||
raise Failure(err or "The headset view sent no video for 20 s")
|
||||
raise Failure(err or "The Frame sent no video for 20 s")
|
||||
chunk = first
|
||||
try:
|
||||
self.send_response(200)
|
||||
|
||||
Reference in new issue
Block a user