Files
Alex SouthwellandClaude Opus 5.5 e702adbd77 Live view: a Desktop view that stays still, and Control to tap on the Frame (#46)
* Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from review

- Keys held on the Frame are released with buttons when Control stops or
  the view loses focus; keys for the Frame no longer trigger Frame Control's
  own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
  presses, keys, text and scrolls name their panel (display and window: ids
  repeat across :0 and :1, told apart by pid), and the Frame drops them if
  focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
  and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
  isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: close the targeting gaps from the second review

- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
  packs ":1" into the first value), so a window id repeated across :0 and :1
  can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
  use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
  stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
  another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
  input too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from the SWE-2 Max review

- The Frame side tracks keys as well as buttons and lets go of both when the
  session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
  disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
  when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
  can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
  "Connecting…"; text goes in 100-character pieces so releases don't wait
  behind a long paste; a cancelled mouse gesture releases what's held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: stale clears, pauses reconverge, pastes split per request

- The Frame says it has caught up as soon as an aimed event lands after a
  stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
  arrived while paused were dropped), and waits for the device once per
  batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: build the socket path on the Frame, so Windows can import it for tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: any event that goes through clears the stale flag

A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:13:21 +10:00

284 lines
13 KiB
Python

"""Touch: the live view's Control (ui/frame_touch.py on the Frame, and the server's checks).
Run: python3 -m unittest discover -s tests
"""
import shutil
import subprocess
import sys
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "ui"))
class Mapping(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def test_panel_bigger_than_its_display_is_scaled(self):
# Verified 2026-09-29: a 1920x1080 window on :1 (1280x720) took Accept at
# window (1828, 1020) as pointer (1219, 680).
panel = {"root": [1280, 720], "width": 1920, "height": 1080}
x, y = self.t.to_root(panel, 1828 / 1920, 1020 / 1080)
self.assertAlmostEqual(x, 1218.7, delta=1)
self.assertAlmostEqual(y, 679.4, delta=1)
def test_same_size_is_one_to_one_and_clamped(self):
panel = {"root": [1280, 720], "width": 1280, "height": 720}
self.assertEqual(self.t.to_root(panel, 0, 0), (0, 0))
self.assertEqual(self.t.to_root(panel, 1, 1), (1279, 719))
self.assertEqual(self.t.to_root(panel, -3, 7), (0, 719))
def test_other_shapes_are_letterboxed(self):
panel = {"root": [1280, 720], "width": 800, "height": 800} # square: bars left and right
x, y = self.t.to_root(panel, 0, 0.5)
self.assertAlmostEqual(x, 280, delta=0.5)
self.assertAlmostEqual(y, 359.5, delta=0.5)
def test_same_id_on_both_displays_is_told_apart_by_pid(self):
t = self.t
saved = t.displays, t.window_info, t.window_pid
self.addCleanup(lambda: (setattr(t, "displays", saved[0]), setattr(t, "window_info", saved[1]),
setattr(t, "window_pid", saved[2])))
t.displays = lambda: [":0", ":1"]
t.window_info = lambda d, w: {"name": f"on {d}", "width": 1280 if w != "root" else 1920, "height": 720}
t.window_pid = lambda d, w: {":0": 111, ":1": 222}[d]
self.assertEqual(t.locate(5, 222)["display"], ":1")
self.assertEqual(t.locate(5, 111)["display"], ":0")
self.assertEqual(t.locate(5, None)["display"], ":0")
def test_focus_display_is_decoded(self):
t = self.t
saved = t.xprop_root
self.addCleanup(lambda: setattr(t, "xprop_root", saved))
for values, want in (([12602, 0, 58], ":1"), ([12346], ":0"), ([], None), ([0x41], None)):
t.xprop_root = lambda d, n, v=values: v
self.assertEqual(t.focus_display(), want)
def test_ascii_table_covers_printable_characters(self):
for code in range(0x20, 0x7F):
self.assertIn(chr(code), self.t.ASCII, chr(code))
self.assertEqual(self.t.ASCII["a"], (30, False))
self.assertEqual(self.t.ASCII["A"], (30, True))
self.assertEqual(self.t.ASCII["?"], (53, True))
self.assertEqual(self.t.ASCII["1"], (2, False))
self.assertEqual(self.t.ASCII["0"], (11, False))
def test_events_parsing(self):
self.assertEqual(self.t.events(b'{"dx": 1}'), [{"dx": 1}])
self.assertEqual(self.t.events(b'[{"dx": 1}, 5]'), [{"dx": 1}])
self.assertEqual(self.t.events(b"nope"), [])
class FakeGamescope:
def __init__(self):
self.calls = []
def __getattr__(self, name):
return lambda *a: self.calls.append((name, *a))
class Apply(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def setUp(self):
self.focused = {"window": 7, "display": ":1", "root": [1280, 720], "width": 1280, "height": 720, "name": "x"}
saved = self.t.focus, self.t.say, self.t.focus_now
self.t.STALE[0] = False
self.said = []
self.t.focus = lambda: dict(self.focused)
self.t.focus_now = lambda: (self.focused["window"], self.focused["display"])
self.t.say = lambda state, **more: self.said.append((state, more))
self.addCleanup(lambda: (setattr(self.t, "focus", saved[0]), setattr(self.t, "say", saved[1]),
setattr(self.t, "focus_now", saved[2])))
def test_tap_moves_then_clicks_in_order(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, {"button": "left", "down": True, "window": 7, "display": ":1"},
{"button": "left", "down": False}):
panel = self.t.apply(gs, e, panel)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "button", "button"])
self.assertEqual(gs.calls[1][1:], ("left", True))
def test_a_tap_meant_for_another_panel_goes_nowhere(self):
# Focus moved on: the position and the press are dropped; the release still goes.
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"button": "left", "down": True, "window": 99, "display": ":1"},
{"key": 30, "down": True, "window": 7, "display": ":0"}, {"button": "left", "down": False, "window": 99, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(gs.calls, [("button", "left", False)])
self.assertEqual(self.said[0], ("ready", {"focus": 7, "display": ":1", "stale": True}))
def test_presses_read_focus_afresh(self):
# A move may use a recent reading; a press checks again, so a panel that just
# took focus doesn't get a click meant for another.
gs, calls = FakeGamescope(), []
self.t.focus = lambda: calls.append(1) or dict(self.focused)
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, None)
panel = self.t.apply(gs, {"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1)
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1) # same panel still: the quick check was enough
self.focused["window"] = 8
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 2)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "move_to", "button"])
def test_stale_is_said_once_and_cleared(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"},
{"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(self.said, [("ready", {"focus": 7, "display": ":1", "stale": True}),
("ready", {"focus": 7, "display": ":1"})])
def test_stale_clears_on_a_trackpad_move_but_not_on_a_stale_release(self):
gs = FakeGamescope()
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, None)
panel = self.t.apply(gs, {"button": "left", "down": False, "window": 99, "display": ":1"}, panel)
self.assertTrue(self.t.STALE[0]) # that release was still aimed at the old panel
self.t.apply(gs, {"dx": 3, "dy": 0}, panel)
self.assertFalse(self.t.STALE[0])
self.assertEqual(self.said[-1][1].get("stale"), None)
def test_same_window_id_on_the_other_display_is_another_panel(self):
gs = FakeGamescope()
self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)
self.assertEqual(gs.calls, [])
def test_relative_scroll_keys_text(self):
gs = FakeGamescope()
for e in ({"dx": 5, "dy": -3}, {"scroll": [0, 120]}, {"key": 30, "down": True}, {"text": "hi"},
{"key": True}, {"button": "sideways"}):
self.t.apply(gs, e, None)
self.assertEqual([c[0] for c in gs.calls], ["move_by", "scroll", "key", "text"])
def test_everything_held_is_let_go(self):
class Held(self.t.Gamescope):
def __init__(self):
self.held, self.keys, self.log = set(), set(), []
def frame(self):
pass
g = Held()
g.L = type("L", (), {"ei_device_button_button": lambda *a: g.log.append(("button",) + a[2:]),
"ei_device_keyboard_key": lambda *a: g.log.append(("key",) + a[2:])})()
g.device = object()
g.button("left", True)
g.key(42, True)
g.release_all()
self.assertEqual(g.log[-2:], [("button", 0x110, False), ("key", 42, False)])
self.assertEqual((g.held, g.keys), (set(), set()))
def test_resuming_after_a_pause_lets_go_of_everything(self):
t = self.t
log, queue = [], [1]
class L:
def __getattr__(self, name):
if name == "ei_get_event":
return lambda ei: queue.pop(0) if queue else None
return {"ei_event_get_type": lambda ev: t.EV_DEVICE_RESUMED, "ei_event_get_device": lambda ev: "dev",
"ei_device_has_capability": lambda d, c: True,
"ei_device_button_button": lambda d, c, down: log.append(("button", c, down)),
"ei_device_keyboard_key": lambda d, c, down: log.append(("key", c, down))}.get(name, lambda *a: 0)
g = t.Gamescope.__new__(t.Gamescope)
g.L, g.ei, g.fd, g.device, g.sequence, g.alive = L(), None, None, None, 0, True
g.held, g.keys = {0x110}, {42}
g.frame = lambda: None
old = t.select.select
t.select.select = lambda *a: ([], [], [])
try:
g.pump()
finally:
t.select.select = old
self.assertEqual(sorted(log), [("button", 0x110, False), ("key", 42, False)])
self.assertEqual(g.device, "dev")
def test_text_uses_shift_for_capitals(self):
class Keys(self.t.Gamescope):
def __init__(self):
self.pressed = []
def key(self, code, down):
self.pressed.append((code, down))
k = Keys()
k.text("Hié") # the é has no key on a US layout and is left out
self.assertEqual(k.pressed, [(42, True), (35, True), (35, False), (42, False), (23, True), (23, False)])
class ServerChecks(unittest.TestCase):
@classmethod
def setUpClass(cls):
import server
cls.s = server
def test_touch_event_keeps_known_fields(self):
ev = self.s.touch_event
self.assertEqual(ev({"fx": 0.5, "fy": 2, "window": 5, "display": ":1"}), {"fx": 0.5, "fy": 1.0, "window": 5, "display": ":1"})
self.assertEqual(ev({"button": "left", "window": 5, "display": ":0"}), {"button": "left", "down": True, "window": 5, "display": ":0"})
self.assertEqual(ev({"button": "right"}), {"button": "right", "down": True})
self.assertEqual(ev({"key": 30, "down": False}), {"key": 30, "down": False})
self.assertEqual(ev({"scroll": [0, 1e9]}), {"scroll": [0.0, 5000.0]})
self.assertEqual(ev({"dx": 3, "other": 1}), {"dx": 3.0})
def test_touch_event_rejects_bad_ones(self):
for bad in (None, {}, {"fx": 0.5, "fy": 0.5}, {"fx": "1", "fy": 0, "window": 1, "display": ":1"}, {"button": "side"},
{"fx": 0.5, "fy": 0.5, "window": 1}, {"fx": 0.5, "fy": 0.5, "window": 1, "display": ":1;x"},
{"window": 1, "display": ":1"},
{"key": 0}, {"key": 999}, {"key": True}, {"scroll": [1]}, {"text": ""}, {"text": "x" * 501},
{"fx": 0.1, "fy": 0.1, "window": True, "display": ":1"}):
with self.assertRaises(self.s.Failure, msg=repr(bad)):
self.s.touch_event(bad)
def test_panel_stream_is_the_window_and_checked(self):
cmd = self.s.stream_command("src=panel&window=10485777&display=:1&h=720&fps=30")
self.assertIn("DISPLAY=:1 ffmpeg", cmd)
self.assertIn("-window_id 10485777 -i :1", cmd)
for bad in ("src=panel&window=1;rm&display=:1", "src=panel&window=1&display=:1;x", "src=panel&display=:1"):
with self.assertRaises(self.s.Failure):
self.s.stream_command(bad + "&h=720&fps=30")
def test_touch_agent_runs_the_helper_with_nothing_to_copy(self):
agent = self.s.TouchAgent()
self.assertEqual(agent.deliver(lambda m: None), "")
cmd = agent.command()
self.assertTrue(cmd.startswith("python3 -u -c '"))
self.assertIn("frame_touch", cmd)
def test_batch_limit(self):
with self.assertRaises(self.s.Failure):
self.s.remote_touch({"events": [{"dx": 1}] * (self.s.INPUT_BATCH_LIMIT + 1)})
@unittest.skipUnless(shutil.which("node"), "needs node")
class PageGestures(unittest.TestCase):
"""Control's gestures and queue (tests/page/ctrl_gestures.mjs runs the real functions from index.html)."""
def test_gestures(self):
r = subprocess.run(["node", str(ROOT / "tests" / "page" / "ctrl_gestures.mjs")], capture_output=True, text=True,
timeout=60)
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
class Script(unittest.TestCase):
def test_helper_compiles_on_the_frames_python(self):
# The Frame runs it with its own python3 (3.13 on SteamOS 0.4.1); stdlib and ctypes only.
src = (ROOT / "ui/frame_touch.py").read_text()
compile(src, "frame_touch.py", "exec")
for mod in ("import ctypes", "import json", "import select"):
self.assertIn(mod, src)
self.assertNotIn("import requests", src)
if __name__ == "__main__":
unittest.main()