Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it: - Desktop: the app panel in use in the headset, streamed from its own window (x11grab of gamescope's redirected window), so it doesn't move as the wearer looks around. A picker shows any other panel, view only. - Control: on the Desktop view a tap or click lands exactly where you put it; drag is a mouse drag, press and hold right-clicks, two fingers scroll, and on a computer the mouse, wheel and keyboard work directly. On the headset view the view is a trackpad. A text field and key row type from a phone. Input goes through gamescope's own EIS socket (the way Steam feeds Remote Play input) with the libei already on the image: ui/frame_touch.py, over the same long-lived ssh machinery as the keyboard agent, nothing to install. It reaches the panel that has focus on either X display, which the KDE Connect route can't. Verified on the Frame and from the iPhone app in the Simulator. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 03:00:18 +02:00 · 2026-09-29 08:23:55 +10:00
1 parent 41f08ac9a2
commit e7c81733b1
@@ -35,8 +35,8 @@ See what the headset sees, install games and Android apps, move files and text a
 <tr>
 <td width="50%" valign="top">

-**👓 Headset view**<br>
-Live video of what the lenses show (about 30 fps), or a still of both eyes. Zoom, pan, full screen, save as PNG.
+**👓 Headset view and Desktop**<br>
+Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.

 </td>
 <td width="50%" valign="top">
@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
 | Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
 | **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
 | **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
+| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
+| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
 | **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
 | Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
 | `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
@@ -6,6 +6,7 @@ This covers three directions, plus input:
 - **B. Mac → Frame**: use the Mac's desktop inside the headset.
 - **C. iPhone → Frame**: mirror the phone inside the headset.
 - **Input**: type and point in the Frame from the Mac or iPhone.
+- **Live view and Control**: watch a panel flat and tap on it to use it.

 The confidence labels are the same as in [ssh.md](ssh.md).

@@ -162,13 +163,68 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
  and ignores injected pointer motion; `:1` holds apps such as Chromium and
  takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
  There's also a `gamescope-0-ei` (libei) socket.
- **Not yet tested:** typing and clicking as seen in the headset, and whether
-  it reaches the KDE desktop panel (Plasma is its own session).
+- Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
+  panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
+  Control's own panels, Steam's UI) and, since XTest positions are clamped to
+  `:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
+  the live view (below) has neither limit.
+- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
+  own session).
 - **Known limit:** keys and clicks typed while the link is reconnecting wait
  and are sent once it's back, but anything sent in the moment the Wi-Fi
  drops, before SSH notices, can be lost. Confirming every event would add a
  round trip to each pointer move.

+## Live view and Control: watch a panel and tap on it
+
+**Built: Home → Desktop / Headset view → Control.** The live view has two
+sources:
+
+- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
+  moves with the wearer's head, so Control makes the view a trackpad: drag to
+  move the pointer, tap to click, press and hold to right-click, two fingers to
+  scroll. With a mouse, moving over the view moves the pointer.
+- **Desktop**: the app panel in use in the headset, from its own window, so it
+  stays still however the wearer looks around. Control makes taps and clicks
+  land exactly where you put them. Dragging is a mouse drag, press and hold is a
+  right-click, two fingers scroll, and on a computer the mouse, wheel and
+  keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
+  other panel, view only.
+
+Below the view, a text field and key buttons type on the Frame from a phone.
+
+How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
+
+- **Input goes through gamescope's own injection.** gamescope serves an EIS
+  socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
+  it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
+  talks to it with `ctypes`: nothing to install. gamescope offers one device,
+  "Gamescope Virtual Input", with relative and absolute pointer, buttons,
+  scroll and keyboard (Linux key codes; no text capability, so the text field
+  types printable ASCII on a US layout). Its absolute region is unbounded; the
+  pointer uses the focused panel's display coordinates, and gamescope fits each
+  window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
+  positions at two thirds scale. Taps on a 1280×720 page landed on the exact
+  pixel.
+- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
+  root), on either X display. In the OpenVR backend focus moves only on SteamVR
+  overlay events (the controller's laser entering or clicking a panel), or to a
+  new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
+  with `gamescopectl focus_info`, which writes to the journal). Neither
+  `GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
+  gamescope command does. So Control follows the wearer: whatever they last
+  used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
+  gets a connector of its own and doesn't hold focus.
+- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
+  apart).
+- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
+  -window_id <window> -i :<display>` works on gamescope's redirected windows,
+  while grabbing the root gives black. It streams as H.264 like the headset view
+  (about 30 fps at 720p).
+- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
+  text box in the panel and the text field typed into it; a trackpad move went
+  exactly (+40, +25).
+
 Our own `uinput` keyboard and mouse would also work (`steamos` is in the
 `input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
 remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.
@@ -0,0 +1,132 @@
+// Control on the live view (tap, drag, hold, scroll, type), run in node against the real
+// functions from ui/index.html with a fake canvas and a fake server.
+import { readFileSync } from "fs";
+const src = readFileSync(new URL("../../ui/index.html", import.meta.url), "utf8");
+const grab = name => {
+  const one = src.match(new RegExp(`\\n((?:async )?function ${name}\\(.*\\}\\n)`));  // one-line function
+  if (one) return one[1];
+  const m = src.match(new RegExp(`(?:\\nconst ${name} = [^\\n]*\\n)|((?:async )?function ${name}\\([\\s\\S]*?\\n}\\n)`));
+  if (!m) throw new Error("not found: " + name);
+  return m[0];
+};
+const NAMES = ["isMoveEvent", "TAP_MOVE", "ctrlSend", "ctrlFlush", "ctrlMoveTo", "ctrlMoveBy", "ctrlSchedule",
+  "ctrlFlushMoves", "ctrlButton", "ctrlClick", "ctrlRelease", "ctrlFraction", "ctrlTouchDown", "centroid",
+  "ctrlTouchMove", "ctrlTouchUp", "ctrlTap", "ctrlText"];
+const code = NAMES.map(grab).join("");
+const fail = msg => { console.log("FAIL " + msg); process.exit(1); };
+const tick = (ms = 0) => new Promise(r => setTimeout(r, ms));
+
+function page({ mode = "abs", rect = { left: 0, top: 0, width: 640, height: 360 }, api } = {}) {
+  const ctrl = { on: true, queue: [], sending: false, state: "ready", message: "", move: null, rel: [0, 0], raf: 0,
+                 held: new Set(), pointers: new Map(), g: null, retry: null };
+  const sent = [];
+  const canvas = { width: 1280, height: 720, getBoundingClientRect: () => rect };
+  const env = {
+    ctrl, $: () => canvas, ctrlMode: () => mode, ctrlShow: () => {}, toast: () => {},
+    ctrlTarget: () => (mode === "abs" ? { ok: true, panel: { window: 42 } } : { ok: true }),
+    api: api || (async (path, body) => { sent.push(...body.events); return { state: "ready", sent: true }; }),
+    requestAnimationFrame: cb => { setTimeout(cb, 0); return 1; },
+    navigator: {},
+  };
+  const fns = new Function(...Object.keys(env), `let ctrlWarned = false;\n${code}
+    return { ctrlTouchDown, ctrlTouchMove, ctrlTouchUp, ctrlSend, ctrlText, ctrlButton, ctrlRelease, ctrlFlushMoves };`)(...Object.values(env));
+  const at = (id, x, y) => ({ pointerId: id, clientX: x, clientY: y });
+  return { ctrl, sent, ...fns, at };
+}
+
+// A tap lands where it was tapped: pointer there first, then the click.
+{
+  const p = page();
+  p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
+  await tick(10);
+  const [move, down, up] = p.sent;
+  if (!(move.fx === 0.5 && Math.abs(move.fy - 0.25) < 1e-9 && move.window === 42)) fail("tap position " + JSON.stringify(move));
+  if (!(down.button === "left" && down.down && up.button === "left" && up.down === false && p.sent.length === 3))
+    fail("tap click " + JSON.stringify(p.sent));
+}
+// The panel is letterboxed in a taller view: taps map inside it, taps on the bars do nothing.
+{
+  const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } });  // 640x360 picture, 60px bars
+  p.ctrlTouchDown(p.at(1, 320, 150)); p.ctrlTouchUp(p.at(1, 320, 150));
+  p.ctrlTouchDown(p.at(2, 320, 10)); p.ctrlTouchUp(p.at(2, 320, 10));
+  await tick(10);
+  if (!(p.sent.length === 3 && Math.abs(p.sent[0].fy - 0.25) < 1e-9)) fail("letterbox " + JSON.stringify(p.sent));
+}
+// A drag is a mouse drag: down where it started, moves, up at the end.
+{
+  const p = page();
+  p.ctrlTouchDown(p.at(1, 100, 100));
+  p.ctrlTouchMove(p.at(1, 140, 100)); await tick(5);
+  p.ctrlTouchMove(p.at(1, 200, 120)); await tick(5);
+  p.ctrlTouchUp(p.at(1, 200, 120)); await tick(10);
+  const kinds = p.sent.map(e => "fx" in e ? "move" : `${e.button}-${e.down ? "down" : "up"}`);
+  if (!(kinds[0] === "move" && kinds[1] === "left-down" && kinds.at(-1) === "left-up" && kinds.includes("move", 2)))
+    fail("drag " + kinds.join(","));
+  if (Math.abs(p.sent[0].fx - 100 / 640) > 1e-9) fail("drag starts where the finger went down");
+  if (p.ctrl.held.size) fail("drag left the button held");
+}
+// Press and hold is a right-click where the finger is.
+{
+  const p = page();
+  p.ctrlTouchDown(p.at(1, 64, 36));
+  await tick(620);
+  p.ctrlTouchUp(p.at(1, 64, 36)); await tick(10);
+  const buttons = p.sent.filter(e => "button" in e).map(e => `${e.button}-${e.down}`);
+  if (buttons.join() !== "right-true,right-false") fail("hold " + buttons.join());
+  if (!(p.sent[0].fx === 0.1)) fail("hold position " + JSON.stringify(p.sent[0]));
+}
+// Two fingers scroll, and the content follows them (fingers up scrolls down).
+{
+  const p = page();
+  p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 200, 200));
+  p.ctrlTouchMove(p.at(1, 100, 180)); p.ctrlTouchMove(p.at(2, 200, 180));
+  p.ctrlTouchUp(p.at(1, 100, 180)); p.ctrlTouchUp(p.at(2, 200, 180));
+  await tick(10);
+  const dy = p.sent.filter(e => e.scroll).reduce((a, e) => a + e.scroll[1], 0);
+  if (!(dy === 40 && !p.sent.some(e => "button" in e))) fail("scroll " + JSON.stringify(p.sent));
+}
+// On the headset view it's a trackpad: drags move the pointer, taps click where it is.
+{
+  const p = page({ mode: "rel" });
+  p.ctrlTouchDown(p.at(1, 100, 100));
+  p.ctrlTouchMove(p.at(1, 110, 100)); p.ctrlTouchMove(p.at(1, 120, 105));
+  p.ctrlTouchUp(p.at(1, 120, 105)); await tick(10);
+  p.ctrlTouchDown(p.at(1, 50, 50)); p.ctrlTouchUp(p.at(1, 50, 50)); await tick(10);
+  const moved = p.sent.filter(e => "dx" in e).reduce((a, e) => [a[0] + e.dx, a[1] + e.dy], [0, 0]);
+  if (!(moved[0] === 32 && moved[1] === 8)) fail("trackpad move " + JSON.stringify(moved));
+  if (p.sent.some(e => "fx" in e)) fail("trackpad sent an absolute position");
+  if (p.sent.filter(e => e.button === "left").length !== 2) fail("trackpad tap " + JSON.stringify(p.sent));
+}
+// Text: plain ASCII only, in chunks the server takes.
+{
+  const p = page();
+  p.ctrlText("héllo " + "x".repeat(600));
+  await tick(10);
+  const text = p.sent.map(e => e.text).join("");
+  if (!(text === "hllo " + "x".repeat(600) && p.sent.every(e => e.text.length <= 500))) fail("text " + text.length);
+}
+// Not connected yet, or the request fails: clicks and keys wait, stale moves don't; a release is never lost.
+{
+  let calls = 0;
+  const p = page({ api: async () => { calls++; if (calls === 1) return { state: "starting", sent: false }; throw new Error("offline"); } });
+  p.ctrlSend([{ fx: 0.1, fy: 0.1, window: 42 }, { button: "left", down: true }, { key: 30, down: true }]);
+  await tick(5);
+  const q = p.ctrl.queue;
+  if (!(q.length === 2 && q[0].button === "left" && q[1].key === 30)) fail("kept while starting " + JSON.stringify(q));
+  clearTimeout(p.ctrl.retry);
+  p.ctrl.queue = [];
+  p.ctrlSend([{ button: "left", down: false }, { key: 31, down: true }, { dx: 3, dy: 1 }]);
+  await tick(5);
+  if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].button === "left" && p.ctrl.queue[0].down === false))
+    fail("release kept on failure " + JSON.stringify(p.ctrl.queue));
+  if (calls !== 2) fail("retried in a tight loop: " + calls + " requests");
+  clearTimeout(p.ctrl.retry);
+}
+// Turning Control off (or the view losing focus) lets go of anything held.
+{
+  const p = page();
+  p.ctrlButton("left", true);
+  p.ctrlRelease(); await tick(10);
+  if (!(p.sent.at(-1).button === "left" && p.sent.at(-1).down === false && !p.ctrl.held.size)) fail("release " + JSON.stringify(p.sent));
+}
+console.log("control gestures ok");
@@ -0,0 +1,172 @@
+"""Touch: the live view's Control (ui/frame_touch.py on the Frame, and the server's checks).
+
+Run: python3 -m unittest discover -s tests
+"""
+import shutil
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT / "ui"))
+
+
+class Mapping(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        import frame_touch
+        cls.t = frame_touch
+
+    def test_panel_bigger_than_its_display_is_scaled(self):
+        # Verified 2026-09-29: a 1920x1080 window on :1 (1280x720) took Accept at
+        # window (1828, 1020) as pointer (1219, 680).
+        panel = {"root": [1280, 720], "width": 1920, "height": 1080}
+        x, y = self.t.to_root(panel, 1828 / 1920, 1020 / 1080)
+        self.assertAlmostEqual(x, 1218.7, delta=1)
+        self.assertAlmostEqual(y, 679.4, delta=1)
+
+    def test_same_size_is_one_to_one_and_clamped(self):
+        panel = {"root": [1280, 720], "width": 1280, "height": 720}
+        self.assertEqual(self.t.to_root(panel, 0, 0), (0, 0))
+        self.assertEqual(self.t.to_root(panel, 1, 1), (1279, 719))
+        self.assertEqual(self.t.to_root(panel, -3, 7), (0, 719))
+
+    def test_other_shapes_are_letterboxed(self):
+        panel = {"root": [1280, 720], "width": 800, "height": 800}  # square: bars left and right
+        x, y = self.t.to_root(panel, 0, 0.5)
+        self.assertAlmostEqual(x, 280, delta=0.5)
+        self.assertAlmostEqual(y, 359.5, delta=0.5)
+
+    def test_ascii_table_covers_printable_characters(self):
+        for code in range(0x20, 0x7F):
+            self.assertIn(chr(code), self.t.ASCII, chr(code))
+        self.assertEqual(self.t.ASCII["a"], (30, False))
+        self.assertEqual(self.t.ASCII["A"], (30, True))
+        self.assertEqual(self.t.ASCII["?"], (53, True))
+        self.assertEqual(self.t.ASCII["1"], (2, False))
+        self.assertEqual(self.t.ASCII["0"], (11, False))
+
+    def test_events_parsing(self):
+        self.assertEqual(self.t.events(b'{"dx": 1}'), [{"dx": 1}])
+        self.assertEqual(self.t.events(b'[{"dx": 1}, 5]'), [{"dx": 1}])
+        self.assertEqual(self.t.events(b"nope"), [])
+
+
+class FakeGamescope:
+    def __init__(self):
+        self.calls = []
+
+    def __getattr__(self, name):
+        return lambda *a: self.calls.append((name, *a))
+
+
+class Apply(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        import frame_touch
+        cls.t = frame_touch
+
+    def setUp(self):
+        self.focused = {"window": 7, "display": ":1", "root": [1280, 720], "width": 1280, "height": 720, "name": "x"}
+        self.old_focus, self.old_say = self.t.focus, self.t.say
+        self.said = []
+        self.t.focus = lambda: dict(self.focused)
+        self.t.say = lambda state, **more: self.said.append((state, more))
+        self.addCleanup(lambda: (setattr(self.t, "focus", self.old_focus), setattr(self.t, "say", self.old_say)))
+
+    def test_tap_moves_then_clicks_in_order(self):
+        gs, panel = FakeGamescope(), None
+        for e in ({"fx": 0.5, "fy": 0.5, "window": 7}, {"button": "left", "down": True}, {"button": "left", "down": False}):
+            panel = self.t.apply(gs, e, panel)
+        self.assertEqual([c[0] for c in gs.calls], ["move_to", "button", "button"])
+        self.assertEqual(gs.calls[1][1:], ("left", True))
+
+    def test_a_tap_meant_for_another_panel_goes_nowhere(self):
+        gs = FakeGamescope()
+        self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 99}, None)
+        self.assertEqual(gs.calls, [])
+        self.assertEqual(self.said, [("ready", {"focus": 7, "stale": True})])
+
+    def test_relative_scroll_keys_text(self):
+        gs = FakeGamescope()
+        for e in ({"dx": 5, "dy": -3}, {"scroll": [0, 120]}, {"key": 30, "down": True}, {"text": "hi"},
+                  {"key": True}, {"button": "sideways"}):
+            self.t.apply(gs, e, None)
+        self.assertEqual([c[0] for c in gs.calls], ["move_by", "scroll", "key", "text"])
+
+    def test_text_uses_shift_for_capitals(self):
+        class Keys(self.t.Gamescope):
+            def __init__(self):
+                self.pressed = []
+
+            def key(self, code, down):
+                self.pressed.append((code, down))
+        k = Keys()
+        k.text("Hié")  # the é has no key on a US layout and is left out
+        self.assertEqual(k.pressed, [(42, True), (35, True), (35, False), (42, False), (23, True), (23, False)])
+
+
+class ServerChecks(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        import server
+        cls.s = server
+
+    def test_touch_event_keeps_known_fields(self):
+        ev = self.s.touch_event
+        self.assertEqual(ev({"fx": 0.5, "fy": 2, "window": 5}), {"fx": 0.5, "fy": 1.0, "window": 5})
+        self.assertEqual(ev({"button": "right"}), {"button": "right", "down": True})
+        self.assertEqual(ev({"key": 30, "down": False}), {"key": 30, "down": False})
+        self.assertEqual(ev({"scroll": [0, 1e9]}), {"scroll": [0.0, 5000.0]})
+        self.assertEqual(ev({"dx": 3, "other": 1}), {"dx": 3.0})
+
+    def test_touch_event_rejects_bad_ones(self):
+        for bad in (None, {}, {"fx": 0.5, "fy": 0.5}, {"fx": "1", "fy": 0, "window": 1}, {"button": "side"},
+                    {"key": 0}, {"key": 999}, {"key": True}, {"scroll": [1]}, {"text": ""}, {"text": "x" * 501},
+                    {"fx": 0.1, "fy": 0.1, "window": True}):
+            with self.assertRaises(self.s.Failure, msg=repr(bad)):
+                self.s.touch_event(bad)
+
+    def test_panel_stream_is_the_window_and_checked(self):
+        cmd = self.s.stream_command("src=panel&window=10485777&display=:1&h=720&fps=30")
+        self.assertIn("DISPLAY=:1 ffmpeg", cmd)
+        self.assertIn("-window_id 10485777 -i :1", cmd)
+        for bad in ("src=panel&window=1;rm&display=:1", "src=panel&window=1&display=:1;x", "src=panel&display=:1"):
+            with self.assertRaises(self.s.Failure):
+                self.s.stream_command(bad + "&h=720&fps=30")
+
+    def test_touch_agent_runs_the_helper_with_nothing_to_copy(self):
+        agent = self.s.TouchAgent()
+        self.assertEqual(agent.deliver(lambda m: None), "")
+        cmd = agent.command()
+        self.assertTrue(cmd.startswith("python3 -u -c '"))
+        self.assertIn("frame_touch", cmd)
+
+    def test_batch_limit(self):
+        with self.assertRaises(self.s.Failure):
+            self.s.remote_touch({"events": [{"dx": 1}] * (self.s.INPUT_BATCH_LIMIT + 1)})
+
+
+@unittest.skipUnless(shutil.which("node"), "needs node")
+class PageGestures(unittest.TestCase):
+    """Control's gestures and queue (tests/page/ctrl_gestures.mjs runs the real functions from index.html)."""
+
+    def test_gestures(self):
+        r = subprocess.run(["node", str(ROOT / "tests" / "page" / "ctrl_gestures.mjs")], capture_output=True, text=True,
+                           timeout=60)
+        self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
+
+
+class Script(unittest.TestCase):
+    def test_helper_compiles_on_the_frames_python(self):
+        # The Frame runs it with its own python3 (3.13 on SteamOS 0.4.1); stdlib and ctypes only.
+        src = (ROOT / "ui/frame_touch.py").read_text()
+        compile(src, "frame_touch.py", "exec")
+        for mod in ("import ctypes", "import json", "import select"):
+            self.assertIn(mod, src)
+        self.assertNotIn("import requests", src)
+
+
+if __name__ == "__main__":
+    unittest.main()
@@ -0,0 +1,350 @@
+"""Touch and direct input for the Steam Frame's panels. Frame Control's server runs this ON the Frame.
+
+gamescope, the Frame's compositor, serves Valve's own input injection: an EIS
+socket (libei's server side), which Steam uses to feed it Remote Play input.
+This connects to it with libei, which is on the SteamOS image, and points,
+clicks, scrolls and types into the panel that has focus in the headset: the
+one the wearer last used. No install, and it reaches every panel, on either
+of gamescope's X displays (see docs/streaming.md).
+
+  python3 frame_touch.py focus   print the focused panel as JSON
+  python3 frame_touch.py panels  print every app panel as JSON, and which has focus
+  python3 frame_touch.py         read events on stdin, one JSON object (or list) per line:
+    {"fx": 0.5, "fy": 0.2, "window": 123}  pointer to that fraction of the focused panel
+                                           (ignored if another panel has focus since)
+    {"dx": 4, "dy": -2}                    pointer by that much
+    {"button": "left", "down": true}       left, right or middle; "down" false releases
+    {"scroll": [0, 120]}                   by pixels; positive y scrolls down
+    {"key": 30, "down": true}              a Linux (evdev) key code, as the page maps KeyboardEvent.code
+    {"text": "hello"}                      printable ASCII, typed on a US layout
+
+Status goes to stdout, one JSON object per line: {"state": "ready" | "error", ...}.
+Standard library only (ctypes for libei), like the rest of what runs on the Frame.
+"""
+import ctypes
+import json
+import os
+import select
+import subprocess
+import sys
+import time
+
+SOCKET = f"/run/user/{os.getuid()}/gamescope-0-ei"
+BUTTONS = {"left": 0x110, "right": 0x111, "middle": 0x112}  # BTN_LEFT, BTN_RIGHT, BTN_MIDDLE
+SHIFT = 42  # KEY_LEFTSHIFT
+# Printable ASCII on a US layout: character -> (evdev key code, shifted).
+ROWS = [("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
+        ("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)]
+ASCII = {" ": (57, False), "\n": (28, False), "\t": (15, False)}
+for plain, shifted, first in ROWS:
+    for i, (a, b) in enumerate(zip(plain, shifted)):
+        ASCII[a], ASCII[b] = (first + i, False), (first + i, True)
+
+# libei's event types and device capabilities (libei.h, libei 1.4).
+EV_CONNECT, EV_DISCONNECT, EV_SEAT_ADDED, EV_DEVICE_ADDED, EV_DEVICE_REMOVED = 1, 2, 3, 5, 6
+EV_DEVICE_PAUSED, EV_DEVICE_RESUMED = 7, 8
+CAP_POINTER, CAP_ABSOLUTE, CAP_KEYBOARD, CAP_SCROLL, CAP_BUTTON = 1, 2, 4, 16, 32
+
+
+def say(state, **more):
+    print(json.dumps({"state": state, **more}), flush=True)
+
+
+# ---- which panel has focus -----------------------------------------------------
+
+def xprop_root(display, name):
+    try:
+        out = subprocess.run(["xprop", "-root", name], env=dict(os.environ, DISPLAY=display),
+                             capture_output=True, text=True, timeout=5).stdout
+    except (OSError, subprocess.SubprocessError):
+        return []
+    values = out.split("=", 1)[1] if "=" in out else ""
+    return [int(v) for v in values.replace(",", " ").split() if v.isdigit()]
+
+
+def window_info(display, window):
+    """Name and geometry of a window on one X display, or None if it isn't there."""
+    try:
+        which = ["-root"] if window == "root" else ["-id", str(window)]
+        out = subprocess.run(["xwininfo", *which], env=dict(os.environ, DISPLAY=display),
+                             capture_output=True, text=True, timeout=5).stdout
+    except (OSError, subprocess.SubprocessError):
+        return None
+    if "IsViewable" not in out:
+        return None
+    info = {}
+    for line in out.splitlines():
+        line = line.strip()
+        if line.startswith("xwininfo: Window id:"):
+            info["name"] = line.split('"', 1)[1].rsplit('"', 1)[0] if '"' in line else ""
+        for key, field in (("Absolute upper-left X:", "x"), ("Absolute upper-left Y:", "y"),
+                           ("Width:", "width"), ("Height:", "height")):
+            if line.startswith(key):
+                info[field] = int(line.split(":", 1)[1])
+    return info if "width" in info else None
+
+
+def displays():
+    return sorted(f":{n[1:]}" for n in os.listdir("/tmp/.X11-unix") if n[1:].isdigit())
+
+
+def focus():
+    """The panel that has focus in the headset: its window, display, name and sizes.
+
+    gamescope publishes it on :0's root (GAMESCOPE_FOCUSED_WINDOW). The window can be
+    on any of its Xwayland displays; the pid in GAMESCOPE_FOCUSABLE_WINDOWS (window,
+    app id, pid triples) tells which one when ids repeat across displays.
+    """
+    window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
+    if not window:
+        return {"window": None}
+    triples = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
+    app = next((triples[i + 1] for i in range(0, len(triples) - 2, 3) if triples[i] == window), None)
+    for display in displays():
+        info = window_info(display, window)
+        if info:
+            root = window_info(display, "root") or {}
+            return {"window": window, "display": display, "app": app, **info,
+                    "root": [root.get("width", info["width"]), root.get("height", info["height"])]}
+    return {"window": None}
+
+
+def panels():
+    """Every app panel (gamescope's focusable windows), for watching one that hasn't focus."""
+    focused = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
+    triples = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
+    found, seen = [], set()
+    for i in range(0, len(triples) - 2, 3):
+        window, app = triples[i], triples[i + 1]
+        if window in seen:
+            continue
+        seen.add(window)
+        for display in displays():
+            info = window_info(display, window)
+            if info and info["width"] > 1 and info["height"] > 1:
+                found.append({"window": window, "display": display, "app": app, **info,
+                              "focused": window == focused})
+                break
+    return {"focus": focused or None, "panels": found}
+
+
+def to_root(panel, fx, fy):
+    """A point given as a fraction of the panel, in the root coordinates gamescope's pointer uses.
+
+    gamescope fits each panel's window to its display, so a 1920x1080 window on a
+    1280x720 display takes pointer positions at two thirds scale (verified 2026-09-29).
+    """
+    rw, rh = panel["root"]
+    w, h = panel["width"], panel["height"]
+    s = min(rw / w, rh / h)
+    ox, oy = (rw - w * s) / 2, (rh - h * s) / 2
+    fx, fy = min(max(fx, 0.0), 1.0), min(max(fy, 0.0), 1.0)
+    return ox + fx * (w * s - 1), oy + fy * (h * s - 1)
+
+
+# ---- gamescope's input socket ----------------------------------------------------
+
+def libei():
+    L = ctypes.CDLL("libei.so.1")
+    vp, c = ctypes.c_void_p, ctypes
+    sig = {
+        "ei_new_sender": (vp, [vp]), "ei_configure_name": (None, [vp, c.c_char_p]),
+        "ei_setup_backend_socket": (c.c_int, [vp, c.c_char_p]), "ei_get_fd": (c.c_int, [vp]),
+        "ei_dispatch": (None, [vp]), "ei_get_event": (vp, [vp]), "ei_event_get_type": (c.c_int, [vp]),
+        "ei_event_unref": (vp, [vp]), "ei_event_get_seat": (vp, [vp]), "ei_event_get_device": (vp, [vp]),
+        "ei_device_has_capability": (c.c_bool, [vp, c.c_int]), "ei_now": (c.c_uint64, [vp]),
+        "ei_device_start_emulating": (None, [vp, c.c_uint32]), "ei_device_stop_emulating": (None, [vp]),
+        "ei_device_frame": (None, [vp, c.c_uint64]),
+        "ei_device_pointer_motion": (None, [vp, c.c_double, c.c_double]),
+        "ei_device_pointer_motion_absolute": (None, [vp, c.c_double, c.c_double]),
+        "ei_device_button_button": (None, [vp, c.c_uint32, c.c_bool]),
+        "ei_device_scroll_delta": (None, [vp, c.c_double, c.c_double]),
+        "ei_device_keyboard_key": (None, [vp, c.c_uint32, c.c_bool]),
+        "ei_unref": (vp, [vp]),
+    }
+    for name, (res, args) in sig.items():
+        f = getattr(L, name)
+        f.restype, f.argtypes = res, args
+    return L
+
+
+class Gamescope:
+    """One connection to gamescope's EIS socket and its virtual input device."""
+
+    def __init__(self):
+        self.L = L = libei()
+        self.ei = L.ei_new_sender(None)
+        L.ei_configure_name(self.ei, b"Frame Control")
+        if L.ei_setup_backend_socket(self.ei, SOCKET.encode()) != 0:
+            raise RuntimeError("Couldn't reach gamescope's input socket. Is the headset on?")
+        self.fd = L.ei_get_fd(self.ei)
+        self.device, self.sequence, self.held = None, 0, set()
+
+    def pump(self, wait=0.0):
+        """Handle gamescope's events; False once it has disconnected."""
+        select.select([self.fd], [], [], wait)
+        self.L.ei_dispatch(self.ei)
+        alive = True
+        while True:
+            ev = self.L.ei_get_event(self.ei)
+            if not ev:
+                return alive
+            kind = self.L.ei_event_get_type(ev)
+            if kind == EV_SEAT_ADDED:
+                seat = self.L.ei_event_get_seat(ev)
+                # Variadic, ending in 0 (NULL): ask for everything we send.
+                self.L.ei_seat_bind_capabilities(ctypes.c_void_p(seat), *map(ctypes.c_int, (
+                    CAP_POINTER, CAP_ABSOLUTE, CAP_BUTTON, CAP_SCROLL, CAP_KEYBOARD, 0)))
+            elif kind == EV_DEVICE_RESUMED:
+                device = self.L.ei_event_get_device(ev)
+                if self.L.ei_device_has_capability(device, CAP_ABSOLUTE):
+                    self.sequence += 1
+                    self.L.ei_device_start_emulating(device, self.sequence)
+                    self.device = device
+            elif kind in (EV_DEVICE_PAUSED, EV_DEVICE_REMOVED):
+                if self.L.ei_event_get_device(ev) == self.device:
+                    self.device = None
+            elif kind == EV_DISCONNECT:
+                self.device, alive = None, False
+            self.L.ei_event_unref(ev)
+
+    def wait_ready(self, timeout=5):
+        end = time.time() + timeout
+        while self.device is None and time.time() < end:
+            if not self.pump(0.1):
+                break
+        if self.device is None:
+            raise RuntimeError("gamescope didn't offer an input device")
+
+    def frame(self):
+        self.L.ei_device_frame(self.device, self.L.ei_now(self.ei))
+        self.L.ei_dispatch(self.ei)
+
+    def move_to(self, x, y):
+        self.L.ei_device_pointer_motion_absolute(self.device, x, y)
+        self.frame()
+
+    def move_by(self, dx, dy):
+        self.L.ei_device_pointer_motion(self.device, dx, dy)
+        self.frame()
+
+    def button(self, name, down):
+        code = BUTTONS[name]
+        if down == (code in self.held):
+            return  # already in that state
+        self.L.ei_device_button_button(self.device, code, down)
+        self.frame()
+        (self.held.add if down else self.held.discard)(code)
+
+    def scroll(self, dx, dy):
+        self.L.ei_device_scroll_delta(self.device, dx, dy)
+        self.frame()
+
+    def key(self, code, down):
+        self.L.ei_device_keyboard_key(self.device, code, down)
+        self.frame()
+        # Paced: a burst of keys can reach the app out of order (seen 2026-09-29).
+        time.sleep(0.008)
+
+    def text(self, text):
+        for ch in text:
+            if ch not in ASCII:
+                continue
+            code, shifted = ASCII[ch]
+            if shifted:
+                self.key(SHIFT, True)
+            self.key(code, True)
+            self.key(code, False)
+            if shifted:
+                self.key(SHIFT, False)
+
+    def release_all(self):
+        for code in list(self.held):
+            name = next(n for n, c in BUTTONS.items() if c == code)
+            self.button(name, False)
+
+
+# ---- events from the server --------------------------------------------------------
+
+def events(line):
+    try:
+        data = json.loads(line)
+    except ValueError:
+        return []
+    return [e for e in (data if isinstance(data, list) else [data]) if isinstance(e, dict)]
+
+
+def number(value, limit=100000.0):
+    if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
+        raise ValueError("not a number")
+    return max(-limit, min(limit, float(value)))
+
+
+def apply(gs, event, panel):
+    """Send one event; returns the panel it was aimed at (refreshed if focus moved)."""
+    if "fx" in event:
+        window = event.get("window")
+        if not panel or panel.get("window") != window or time.time() - panel.get("_at", 0) > 1:
+            panel = {**focus(), "_at": time.time()}
+        if panel.get("window") is None or panel["window"] != window:
+            say("ready", focus=panel.get("window"), stale=True)  # the page re-syncs its view
+            return panel
+        gs.move_to(*to_root(panel, number(event["fx"], 1), number(event["fy"], 1)))
+    if "dx" in event or "dy" in event:
+        gs.move_by(number(event.get("dx", 0), 2000), number(event.get("dy", 0), 2000))
+    if event.get("button") in BUTTONS:
+        gs.button(event["button"], event.get("down") is not False)
+    if isinstance(event.get("scroll"), list) and len(event["scroll"]) == 2:
+        gs.scroll(number(event["scroll"][0], 5000), number(event["scroll"][1], 5000))
+    if isinstance(event.get("key"), int) and not isinstance(event["key"], bool) and 0 < event["key"] < 768:
+        gs.key(event["key"], event.get("down") is not False)
+    if isinstance(event.get("text"), str):
+        gs.text(event["text"][:500])
+    return panel
+
+
+def main():
+    if sys.argv[1:] == ["focus"]:
+        print(json.dumps(focus()))
+        return 0
+    if sys.argv[1:] == ["panels"]:
+        print(json.dumps(panels()))
+        return 0
+    try:
+        gs = Gamescope()
+        gs.wait_ready()
+    except (OSError, RuntimeError) as e:
+        say("error", message=str(e))
+        return 1
+    say("ready", focus=focus().get("window"))
+    stdin, pending, panel = sys.stdin.fileno(), b"", None
+    try:
+        while True:
+            ready, _, _ = select.select([stdin, gs.fd], [], [], 30)
+            if gs.fd in ready and not gs.pump():
+                say("error", message="gamescope closed its input socket")
+                return 1
+            if stdin not in ready:
+                continue
+            chunk = os.read(stdin, 65536)
+            if not chunk:
+                return 0  # the server went away
+            *lines, pending = (pending + chunk).split(b"\n")
+            for line in lines:
+                for event in events(line):
+                    if gs.device is None:
+                        gs.wait_ready(2)
+                    try:
+                        panel = apply(gs, event, panel)
+                    except (ValueError, KeyError, TypeError):
+                        continue  # the server checks events; skip anything odd
+    except RuntimeError as e:
+        say("error", message=str(e))
+        return 1
+    finally:
+        if gs.device is not None:
+            gs.release_all()  # never leave a button held down in the headset
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -146,6 +146,18 @@
  .viewer .badge { position: absolute; left: 12px; top: 12px; font-size: 11px; letter-spacing: 1.2px; text-transform: uppercase;
                   background: rgba(0,0,0,.6); color: var(--text); padding: 4px 9px; border-radius: 2px; }
  .viewer .badge.live { left: auto; right: 12px; background: var(--bad); color: #fff; font-weight: 700; }
+  .view-actions { display: flex; gap: 10px; }
+  .ctrl-status { margin-top: 10px; font-size: 13px; color: var(--blue); }
+  .ctrl-status::before { content: "● "; }
+  .ctrl-status.warn { color: var(--warn); }
+  /* Control: the view takes taps, clicks, drags and keys, and none of them zoom or pan it. */
+  .viewer.control { outline: 1px solid var(--blue); }
+  .viewer.control canvas { cursor: crosshair; touch-action: none; }
+  .viewer.control.rel canvas { cursor: move; }
+  .viewer.immersive { position: fixed; z-index: 40; inset: env(safe-area-inset-top) env(safe-area-inset-right) 0 env(safe-area-inset-left);
+                      max-height: none; aspect-ratio: auto; border-radius: 0; }
+  .ctrl-keys { margin-top: 10px; }
+  #panelPick { max-width: 240px; height: 30px; }
  .viewer .stamp { position: absolute; right: 12px; bottom: 12px; font-size: 11px; color: #c7d5e0;
                   background: rgba(0,0,0,.6); padding: 3px 8px; border-radius: 2px; }
  .viewer .asleep { position: absolute; inset: auto 0 0 0; padding: 14px 18px; text-align: center; color: var(--text);
@@ -363,6 +375,9 @@
    .grid-top, .grid-3, .and-grid, .and-col { gap: 14px; }
    .toolbar { gap: 8px; }
    .toolbar .spacer { display: none; }
+    .view-actions { flex: 1 1 100%; display: grid; grid-template-columns: repeat(4, 1fr); gap: 6px; }
+    .view-actions button { justify-content: center; padding: 0 4px; }
+    #panelPick { flex: 1 1 100%; max-width: none; }
    .shelf-head { flex-wrap: wrap; row-gap: 8px; }
    .shelf { grid-template-columns: repeat(auto-fill, minmax(100px, 1fr)); gap: 10px; }
    .cat-grid { grid-template-columns: 1fr; }
@@ -423,16 +438,20 @@
      <div class="toolbar">
        <div class="seg" aria-label="What to capture">
          <button data-view="headset" class="on" title="What the headset shows, via the SteamVR compositor">Headset view</button>
-          <button data-view="flat" title="gamescope's 2D layer: the desktop panel and Steam's flat UI">Desktop panel</button>
+          <button data-view="flat" title="The app panel in use, flat: it stays still however the wearer looks around">Desktop</button>
        </div>
+        <select id="panelPick" class="small" hidden aria-label="Which panel to show"></select>
        <div class="seg" id="eyeSeg" aria-label="Crop">
          <button data-eye="left" class="on" title="One eye, like pointing a camera into one lens">Left eye</button>
          <button data-eye="both" title="Both eyes side by side">Both eyes</button>
        </div>
        <div class="spacer"></div>
-        <button class="action" id="shotBtn">Capture</button>
-        <button id="liveBtn" title="Keep updating: video of the headset view, or repeated captures of the desktop panel">Live</button>
-        <button id="saveBtn" disabled>Save</button>
+        <span class="view-actions">
+          <button class="action" id="shotBtn">Capture</button>
+          <button id="liveBtn" title="Keep updating, as video">Live</button>
+          <button id="ctrlBtn" title="Control the Frame by tapping or clicking on the view (C)">Control</button>
+          <button id="saveBtn" disabled>Save</button>
+        </span>
      </div>
      <div class="viewer" id="viewer">
        <div class="empty" id="viewerEmpty">
@@ -454,6 +473,25 @@
        <div class="asleep" id="asleep" hidden>The headset is showing a blank frame. It's probably asleep or not being worn. Put it on and capture again.</div>
        <div class="loading"><div class="spinner"></div></div>
      </div>
+      <div class="ctrl-status" id="ctrlBadge" role="status" hidden></div>
+      <div class="pad-keys ctrl-keys" id="ctrlKeys" hidden aria-label="Keys and clicks for the Frame">
+        <input id="ctrlType" type="text" placeholder="Type here to type on the Frame" autocomplete="off" autocapitalize="off"
+               autocorrect="off" spellcheck="false" enterkeyhint="send">
+        <span class="seg pad-clicks">
+          <button data-ctrl-click="left" title="Left click where the pointer is">Click</button>
+          <button data-ctrl-click="right" title="Right click where the pointer is">Right-click</button>
+        </span>
+        <span class="seg pad-special">
+          <button data-ctrl-key="1" title="Escape">Esc</button>
+          <button data-ctrl-key="15" title="Tab">Tab</button>
+          <button data-ctrl-key="105" title="Left arrow" aria-label="Left arrow">←</button>
+          <button data-ctrl-key="103" title="Up arrow" aria-label="Up arrow">↑</button>
+          <button data-ctrl-key="108" title="Down arrow" aria-label="Down arrow">↓</button>
+          <button data-ctrl-key="106" title="Right arrow" aria-label="Right arrow">→</button>
+          <button data-ctrl-key="14" title="Backspace" aria-label="Backspace">⌫</button>
+          <button data-ctrl-key="28" title="Enter">Enter</button>
+        </span>
+      </div>
      <div class="hint" id="viewHint"></div>
    </section>

@@ -803,9 +841,9 @@ const QUICK = [["Remmina", "org.remmina.Remmina"], ["Moonlight", "com.moonlight_
 const CDN = "https://cdn.cloudflare.steamstatic.com/steam/apps";
 const HINTS = {
  headset: "What the lenses show, composited by SteamVR: the room, floating panels, dashboard and controllers. Live streams it as video (one eye, about 30 fps); Capture takes a still of both eyes. Captures show everything on screen, including anything private.",
-  flat: "gamescope's 2D layer: the desktop panel and Steam's flat UI, without the room or VR scene.",
+  flat: "The app panel in use in the headset, flat and still: it doesn't move as the wearer looks around. Pick another panel to watch it. Control lets you tap or click right where you want on it.",
 };
-const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", shot: "Screenshot" };
+const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", panel: "Desktop", shot: "Screenshot" };
 let state = null, view = "headset", eye = "left", live = false, liveTimer = null, volTimer = null;
 let lastImg = null, lastSource = null, lastShot = null, viewGen = 0;

@@ -1065,8 +1103,12 @@ function setView(v) {
  view = v;
  document.querySelectorAll("[data-view]").forEach(b => b.classList.toggle("on", b.dataset.view === v));
  $("eyeSeg").style.visibility = v === "headset" && !video.ctl ? "visible" : "hidden";
+  $("eyeSeg").style.display = v === "flat" ? "none" : "";  // the Desktop view has no eyes to pick
  $("viewHint").textContent = HINTS[v];
-  if (live && changed) { toggleLive(false); toggleLive(true); }  // video for the headset, captures for the panel
+  $("panelPick").hidden = v !== "flat";
+  if (v === "flat") watchPanels(); else clearTimeout(desk.timer);
+  if (live && changed) { toggleLive(false); toggleLive(true); }
+  ctrlShow();
 }
 function setEye(e) {
  eye = e;
@@ -1093,20 +1135,27 @@ function setZoom(z, cx, cy) {
 $("zoomIn").onclick = () => setZoom(zoom * 1.5);
 $("zoomOut").onclick = () => setZoom(zoom / 1.5);
 $("zoomFit").onclick = () => setZoom(1);
-$("fullBtn").onclick = () => document.fullscreenElement ? document.exitFullscreen() : $("viewer").requestFullscreen();
+// Full screen, or on phones (no element full screen there) the viewer filling the window.
+$("fullBtn").onclick = () => {
+  if (document.fullscreenElement) return document.exitFullscreen();
+  if ($("viewer").requestFullscreen && !TOUCH) return $("viewer").requestFullscreen().catch(() => $("viewer").classList.toggle("immersive"));
+  $("viewer").classList.toggle("immersive");
+  setZoom(1);
+};
 document.addEventListener("fullscreenchange", () => setZoom(1));
 $("viewer").addEventListener("wheel", e => {
-  if ($("canvas").hidden) return;
+  if ($("canvas").hidden || ctrl.on) return;
  e.preventDefault();
  const v = $("viewer").getBoundingClientRect();
  setZoom(zoom * Math.exp(-e.deltaY * 0.002), e.clientX - v.left, e.clientY - v.top);
 }, { passive: false });
 $("canvas").addEventListener("dblclick", e => {
+  if (ctrl.on) return;
  const v = $("viewer").getBoundingClientRect();
  setZoom(zoom > 1 ? 1 : 2.5, e.clientX - v.left, e.clientY - v.top);
 });
 $("canvas").addEventListener("pointerdown", e => {
-  if (zoom === 1) return;
+  if (zoom === 1 || ctrl.on) return;
  const start = { x: e.clientX - panX, y: e.clientY - panY };
  $("viewer").classList.add("panning");
  $("canvas").setPointerCapture(e.pointerId);
@@ -1116,7 +1165,10 @@ $("canvas").addEventListener("pointerdown", e => {
  $("canvas").addEventListener("pointerup", up, { once: true });
 });
 document.addEventListener("keydown", e => {
-  if (page !== "home" || $("canvas").hidden || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
+  if (page !== "home" || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
+  if (ctrl.on && document.activeElement === $("viewer")) return;  // keys are going to the Frame
+  if (e.key === "c") return $("ctrlBtn").click();
+  if ($("canvas").hidden) return;
  if (e.key === "+" || e.key === "=") setZoom(zoom * 1.5);
  else if (e.key === "-") setZoom(zoom / 1.5);
  else if (e.key === "0") setZoom(1);
@@ -1158,8 +1210,13 @@ async function capture() {
  if (!live) $("viewer").classList.add("busy");  // no spinner flashing over a live stream
  let url = null;
  try {
-    const r = await fetch(view === "headset" ? "/api/screenshot?view=headset" : "/api/screenshot",
-                          { headers: {"X-Frame-UI": UI_KEY} });
+    let url_ = "/api/screenshot?view=headset";
+    if (view === "flat") {
+      const p = deskPanel() || (await loadPanels(), deskPanel());
+      if (!p) throw new Error("no app panel is open in the headset");
+      url_ = `/api/screenshot?view=panel&window=${p.window}&display=${encodeURIComponent(p.display)}`;
+    }
+    const r = await fetch(url_, { headers: {"X-Frame-UI": UI_KEY} });
    if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
    const source = r.headers.get("X-Capture-Source") || "gamescope";
    url = URL.createObjectURL(await r.blob());
@@ -1168,6 +1225,7 @@ async function capture() {
    if (gen !== viewGen) return true;
    lastImg = img; lastSource = source; lastShot = null;
    draw();
+    if (source === "panel") $("srcBadge").textContent = `Desktop · ${deskPanel()?.name || "panel"}`;
    $("stamp").hidden = false; $("stamp").textContent = new Date().toLocaleTimeString();
    $("saveBtn").disabled = false;
    return true;
@@ -1186,7 +1244,7 @@ async function liveLoop() {
  liveFailures = (await capture()) ? 0 : liveFailures + 1;
  if (liveFailures >= 5) {
    toggleLive(false);
-    toast("Live stopped after 5 failed captures. Is the headset on?", true);
+    toast(view === "flat" ? "Live stopped: couldn't capture the panel." : "Live stopped after 5 failed captures. Is the headset on?", true);
    return;
  }
  if (live) liveTimer = setTimeout(liveLoop, view === "headset" ? 300 : 700);
@@ -1199,7 +1257,7 @@ function toggleLive(on) {
  clearTimeout(liveTimer);
  stopVideo();
  if (!live) return;
-  if (view === "headset" && "VideoDecoder" in window) {
+  if ("VideoDecoder" in window && (view === "headset" || deskPanel())) {
    const started = startVideo();
    const ctl = video.ctl;
    started.catch(e => {
@@ -1246,7 +1304,7 @@ async function startVideo() {
        lastImg = null; lastSource = "video"; lastShot = null;
        $("viewer").classList.remove("busy");
        c.hidden = false; $("viewerEmpty").hidden = true; $("zoombar").hidden = false; $("asleep").hidden = true;
-        $("srcBadge").hidden = false; $("srcBadge").textContent = "Headset view · video";
+        $("srcBadge").hidden = false; $("srcBadge").textContent = `${video.label} · video`;
        $("stamp").hidden = false; $("saveBtn").disabled = false;
      }
      frames++;
@@ -1258,7 +1316,10 @@ async function startVideo() {
    },
    error: e => log("Video decoder: " + e.message, "e"),
  });
-  const r = await fetch(`/api/stream?${STREAM_QUERY}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
+  const panel = view === "flat" ? deskPanel() : null;
+  video.label = panel ? `Desktop · ${panel.name || "panel"}` : "Headset view";
+  const query = panel ? `src=panel&window=${panel.window}&display=${encodeURIComponent(panel.display)}&${STREAM_QUERY}` : STREAM_QUERY;
+  const r = await fetch(`/api/stream?${query}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
  if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
  const reader = r.body.getReader();
  let buf = new Uint8Array(0), scan = 0, auStart = -1;
@@ -1303,7 +1364,7 @@ function stopVideo() {
  if (video.dec && video.dec.state !== "closed") video.dec.close();
  video.ctl = video.dec = null;
  $("viewer").classList.remove("busy");
-  if (lastSource === "video") { $("srcBadge").textContent = "Headset view · video (stopped)"; $("stamp").hidden = true; }
+  if (lastSource === "video") { $("srcBadge").textContent = `${video.label || "Headset view"} · video (stopped)`; $("stamp").hidden = true; }
  $("eyeSeg").style.visibility = view === "headset" ? "visible" : "hidden";
 }
 $("shotBtn").onclick = () => capture();
@@ -1323,6 +1384,330 @@ function download(blob, name) {
 document.querySelectorAll("[data-view]").forEach(b => b.onclick = () => setView(b.dataset.view));
 document.querySelectorAll("[data-eye]").forEach(b => b.onclick = () => setEye(b.dataset.eye));

+// ---- Desktop view: the app panel in use in the headset (or one picked to watch) ----
+// gamescope says which panel has focus: the one the wearer last used. That's the one
+// Control reaches, so the view follows it unless you pick another to watch.
+const desk = { panels: [], focus: null, pick: "", timer: null, loaded: false };
+function deskPanel() {
+  const want = desk.pick ? +desk.pick : desk.focus;
+  // Nothing in use and nothing picked: show the first panel rather than nothing.
+  return desk.panels.find(p => p.window === want) || (!desk.pick && !desk.focus && desk.panels[0]) || null;
+}
+async function loadPanels() {
+  try {
+    const before = deskPanel()?.window;
+    const r = await api("/api/panels");
+    desk.panels = r.panels || []; desk.focus = r.focus || null; desk.loaded = true;
+    if (desk.pick && !desk.panels.some(p => p.window === +desk.pick)) desk.pick = "";
+    renderPanelPick();
+    const now = deskPanel()?.window;
+    if (view === "flat" && live && now !== before) { toggleLive(false); toggleLive(true); }
+    ctrlShow();
+  } catch (e) {
+    log("Couldn't list the headset's panels: " + e.message, "e");
+  }
+}
+function watchPanels() {
+  clearTimeout(desk.timer);
+  if (view !== "flat") return;
+  loadPanels().finally(() => { if (view === "flat") desk.timer = setTimeout(watchPanels, document.hidden ? 10000 : 3000); });
+}
+function renderPanelPick() {
+  const sel = $("panelPick");
+  sel.innerHTML = `<option value="">${desk.focus ? "Panel in use" : "Panel in use (none yet)"}</option>` +
+    desk.panels.map(p => `<option value="${p.window}">${esc(p.name || "Untitled")}${p.focused ? " (in use)" : ""}</option>`).join("");
+  sel.value = desk.pick;
+}
+$("panelPick").onchange = () => {
+  desk.pick = $("panelPick").value;
+  ctrlShow();
+  if (live) { toggleLive(false); toggleLive(true); } else if (lastImg || lastSource) capture();
+};
+
+// ---- Control: tap or click on the view to use the Frame ----
+// Through gamescope's own input (frame_touch.py). On the Desktop view a tap lands
+// exactly where you tapped; on the headset view, which moves with the wearer's head,
+// the view is a trackpad for the pointer instead.
+const ctrl = { on: false, queue: [], sending: false, state: "off", message: "", move: null, rel: [0, 0], raf: 0,
+               held: new Set(), pointers: new Map(), g: null, retry: null };
+const EVDEV = { Escape: 1, Minus: 12, Equal: 13, Backspace: 14, Tab: 15, BracketLeft: 26, BracketRight: 27, Enter: 28,
+  ControlLeft: 29, Semicolon: 39, Quote: 40, Backquote: 41, ShiftLeft: 42, Backslash: 43, Comma: 51, Period: 52,
+  Slash: 53, ShiftRight: 54, NumpadMultiply: 55, AltLeft: 56, Space: 57, CapsLock: 58, F11: 87, F12: 88,
+  IntlBackslash: 86, NumpadEnter: 96, ControlRight: 97, NumpadDivide: 98, AltRight: 100, Home: 102, ArrowUp: 103,
+  PageUp: 104, ArrowLeft: 105, ArrowRight: 106, End: 107, ArrowDown: 108, PageDown: 109, Insert: 110, Delete: 111,
+  MetaLeft: 125, MetaRight: 126, ContextMenu: 127, NumpadSubtract: 74, NumpadAdd: 78, NumpadDecimal: 83 };
+[..."qwertyuiop"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 16 + i);
+[..."asdfghjkl"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 30 + i);
+[..."zxcvbnm"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 44 + i);
+[..."1234567890"].forEach((c, i) => EVDEV["Digit" + c] = 2 + i);
+for (let i = 1; i <= 10; i++) EVDEV["F" + i] = 58 + i;
+[[7, 71], [8, 72], [9, 73], [4, 75], [5, 76], [6, 77], [1, 79], [2, 80], [3, 81], [0, 82]].forEach(([n, k]) => EVDEV["Numpad" + n] = k);
+const CTRL_BUTTONS = ["left", "middle", "right"];  // MouseEvent.button 0, 1, 2
+
+function ctrlMode() { return view === "flat" ? "abs" : "rel"; }
+// Where taps go on the Desktop view, or why they can't.
+function ctrlTarget() {
+  if (ctrlMode() === "rel") return { ok: true };
+  const p = deskPanel();
+  if (!desk.loaded) return { why: "Finding the headset's panels…" };
+  if (!p) return { why: "No app panel is open in the headset." };
+  if (!desk.focus) return { why: "Nobody's using a panel in the headset yet. Taps work once one is in use." };
+  if (p.window !== desk.focus) return { why: "Only the panel in use takes taps. Pick “Panel in use” to control it." };
+  return { ok: true, panel: p };
+}
+function ctrlShow() {
+  const v = $("viewer"), b = $("ctrlBadge");
+  $("ctrlBtn").classList.toggle("on", ctrl.on);
+  v.classList.toggle("control", ctrl.on);
+  v.classList.toggle("rel", ctrl.on && ctrlMode() === "rel");
+  $("ctrlKeys").hidden = !ctrl.on;
+  b.hidden = !ctrl.on;
+  if (!ctrl.on) return;
+  const t = ctrlTarget();
+  let text = ctrlMode() === "abs" ? `Tap or click to use ${t.panel ? t.panel.name || "the panel" : "the panel"}`
+                                  : "Drag to move the pointer, tap to click, two fingers to scroll";
+  let warn = false;
+  if (!t.ok) { text = t.why; warn = true; }
+  else if (ctrl.state === "error") { text = ctrl.message || "Control isn't working"; warn = true; }
+  else if (ctrl.state === "starting") text = "Connecting…";
+  b.textContent = text;
+  b.classList.toggle("warn", warn);
+}
+function ctrlToggle(on) {
+  if (!on) ctrlRelease();
+  ctrl.on = on;
+  if (on) {
+    setZoom(1);
+    ctrlPoll(0);
+    if (view === "flat" && !desk.loaded) watchPanels();
+    if (!TOUCH) $("viewer").focus({ preventScroll: true });
+  }
+  if (window.frameApp && window.frameApp.captureKeys) window.frameApp.captureKeys(on && document.activeElement === $("viewer"));
+  ctrlShow();
+}
+$("ctrlBtn").onclick = () => ctrlToggle(!ctrl.on);
+// Until it's ready, check on the Frame side every half second (for up to 15 s).
+async function ctrlPoll(n) {
+  if (!ctrl.on) return;
+  try {
+    const r = await api(n ? "/api/touch" : "/api/touch?start=1");
+    ctrl.state = r.state; ctrl.message = r.message || "";
+  } catch (e) { ctrl.state = "error"; ctrl.message = e.message; }
+  ctrlShow();
+  if (ctrl.state !== "ready" && ctrl.state !== "error" && n < 30) setTimeout(() => ctrlPoll(n + 1), 500);
+  else if (ctrl.state === "ready") ctrlFlush();
+}
+$("viewer").tabIndex = 0;
+$("viewer").addEventListener("focus", () => { if (ctrl.on && window.frameApp?.captureKeys) window.frameApp.captureKeys(true); });
+$("viewer").addEventListener("blur", () => { ctrlRelease(); if (window.frameApp?.captureKeys) window.frameApp.captureKeys(false); });
+document.addEventListener("visibilitychange", () => { if (document.hidden) ctrlRelease(); });
+
+const isMoveEvent = e => ("fx" in e || "dx" in e) && !("button" in e) && !("key" in e);
+function ctrlSend(events) {
+  ctrl.queue.push(...events);
+  // Keep order; if the Frame is slow to answer, drop pointer moves first, never a release.
+  if (ctrl.queue.length > 300) ctrl.queue = ctrl.queue.filter(e => !isMoveEvent(e)).slice(-200);
+  ctrlFlush();
+}
+async function ctrlFlush() {
+  if (ctrl.sending || !ctrl.queue.length) return;
+  ctrl.sending = true;
+  const batch = ctrl.queue.splice(0, 200);
+  let again = 0;
+  try {
+    const r = await api("/api/touch", { events: batch });
+    ctrl.state = r.state; ctrl.message = r.message || "";
+    if (!r.sent) {
+      // Not connected yet: keep clicks and keys for when it is (moves are stale by then).
+      ctrl.queue.unshift(...batch.filter(e => !isMoveEvent(e)));
+      if (r.state !== "error") again = 300;
+    }
+  } catch (e) {
+    ctrl.state = "error"; ctrl.message = e.message;
+    ctrl.queue.unshift(...batch.filter(e => "button" in e && e.down === false || "key" in e && e.down === false));
+    again = 1000;  // try the releases again in a moment, not in a tight loop
+  } finally {
+    ctrl.sending = false;
+    ctrlShow();
+    clearTimeout(ctrl.retry);
+    if (again) ctrl.retry = setTimeout(ctrlFlush, again);
+    else if (ctrl.queue.length) ctrlFlush();
+  }
+}
+// Pointer moves: at most one per animation frame.
+function ctrlMoveTo(fx, fy) {
+  const p = ctrlTarget().panel;
+  if (!p) return;
+  ctrl.move = { fx, fy, window: p.window };
+  ctrlSchedule();
+}
+function ctrlMoveBy(dx, dy) { ctrl.rel[0] += dx; ctrl.rel[1] += dy; ctrlSchedule(); }
+function ctrlSchedule() { if (!ctrl.raf) ctrl.raf = requestAnimationFrame(ctrlFlushMoves); }
+function ctrlFlushMoves() {
+  ctrl.raf = 0;
+  const out = [];
+  if (ctrl.move) { out.push(ctrl.move); ctrl.move = null; }
+  if (ctrl.rel[0] || ctrl.rel[1]) { out.push({ dx: Math.round(ctrl.rel[0]), dy: Math.round(ctrl.rel[1]) }); ctrl.rel = [0, 0]; }
+  if (out.length) ctrlSend(out);
+}
+function ctrlButton(name, down) {
+  ctrlFlushMoves();  // the click lands where the pointer was sent
+  if (down) ctrl.held.add(name); else ctrl.held.delete(name);
+  ctrlSend([{ button: name, down }]);
+}
+function ctrlClick(name = "left") { ctrlButton(name, true); ctrlButton(name, false); }
+function ctrlRelease() {
+  for (const b of [...ctrl.held]) ctrlButton(b, false);
+  if (ctrl.g && ctrl.g.timer) clearTimeout(ctrl.g.timer);
+  ctrl.pointers.clear(); ctrl.g = null;
+}
+// Where a pointer event falls on the panel, as fractions (the canvas box includes any letterboxing).
+function ctrlFraction(e) {
+  const c = $("canvas"), r = c.getBoundingClientRect();
+  const scale = Math.min(r.width / c.width, r.height / c.height);
+  const w = c.width * scale, h = c.height * scale;
+  const x = (e.clientX - r.left - (r.width - w) / 2) / w, y = (e.clientY - r.top - (r.height - h) / 2) / h;
+  return x >= 0 && x <= 1 && y >= 0 && y <= 1 ? [x, y] : null;
+}
+
+// Mouse (and pen) on a computer: the view is the Frame's screen.
+function ctrlMouse(e) {
+  if (ctrlMode() === "abs") {
+    const f = ctrlFraction(e);
+    if (f) ctrlMoveTo(...f);
+  } else if (e.type === "pointermove" && (e.movementX || e.movementY)) {
+    ctrlMoveBy(e.movementX * 1.5, e.movementY * 1.5);
+  }
+}
+// Touch: tap to click, press and hold to right-click, two fingers to scroll. Dragging is a
+// mouse drag on the Desktop view (sliders, selecting text) and moves the pointer on the headset view.
+const TAP_MOVE = 8, HOLD_MS = 550, TOUCH_SPEED = 1.6, SCROLL_SPEED = 2;
+function ctrlTouchDown(e) {
+  ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
+  if (ctrl.pointers.size === 1) {
+    const g = ctrl.g = { kind: "maybe", x: e.clientX, y: e.clientY, e };
+    g.timer = setTimeout(() => {
+      if (ctrl.g !== g || g.kind !== "maybe") return;
+      g.kind = "held";
+      if (ctrlMode() === "abs") { const f = ctrlFraction(g.e); if (f) ctrlMoveTo(...f); }
+      ctrlClick("right");
+      navigator.vibrate?.(10);
+    }, HOLD_MS);
+  } else if (ctrl.pointers.size === 2) {
+    if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
+    if (ctrl.held.has("left")) ctrlButton("left", false);
+    ctrl.g = { kind: "scroll", ...centroid() };
+  }
+}
+function centroid() {
+  const ps = [...ctrl.pointers.values()];
+  return { x: ps.reduce((a, p) => a + p.x, 0) / ps.length, y: ps.reduce((a, p) => a + p.y, 0) / ps.length };
+}
+function ctrlTouchMove(e) {
+  const prev = ctrl.pointers.get(e.pointerId);
+  if (!prev || !ctrl.g) return;
+  const g = ctrl.g, dx = e.clientX - prev.x, dy = e.clientY - prev.y;
+  ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
+  if (g.kind === "scroll") {
+    const c = centroid();
+    // Content follows the fingers, as on the phone itself.
+    ctrlSend([{ scroll: [-(c.x - g.x) * SCROLL_SPEED, -(c.y - g.y) * SCROLL_SPEED] }]);
+    g.x = c.x; g.y = c.y;
+    return;
+  }
+  if (g.kind === "maybe" && Math.hypot(e.clientX - g.x, e.clientY - g.y) > TAP_MOVE) {
+    clearTimeout(g.timer);
+    if (ctrlMode() === "abs") {
+      const f = ctrlFraction(g.e);
+      if (f) { ctrlMoveTo(...f); ctrlButton("left", true); }
+      g.kind = "drag";
+    } else g.kind = "move";
+  }
+  if (g.kind === "drag") { const f = ctrlFraction(e); if (f) ctrlMoveTo(...f); }
+  else if (g.kind === "move") ctrlMoveBy(dx * TOUCH_SPEED, dy * TOUCH_SPEED);
+}
+function ctrlTouchUp(e) {
+  const g = ctrl.g;
+  ctrl.pointers.delete(e.pointerId);
+  if (!g) return;
+  if (g.kind === "maybe" && ctrl.pointers.size === 0) {
+    clearTimeout(g.timer);
+    if (ctrlMode() === "abs") { const f = ctrlFraction(g.e); if (!f) { ctrl.g = null; return; } ctrlMoveTo(...f); }
+    ctrlClick("left");
+  } else if (g.kind === "drag") ctrlButton("left", false);
+  if (ctrl.pointers.size === 0) ctrl.g = null;
+}
+const canvasEl = $("canvas");
+canvasEl.addEventListener("pointerdown", e => {
+  if (!ctrl.on) return;
+  e.preventDefault();
+  $("viewer").focus({ preventScroll: true });
+  try { canvasEl.setPointerCapture(e.pointerId); } catch { /* a pointer that's already gone */ }
+  if (!ctrlTarget().ok) return ctrlShow();
+  if (e.pointerType === "touch") return ctrlTouchDown(e);
+  ctrlMouse(e);
+  ctrlButton(CTRL_BUTTONS[e.button] || "left", true);
+});
+canvasEl.addEventListener("pointermove", e => {
+  if (!ctrl.on || !ctrlTarget().ok) return;
+  if (e.pointerType === "touch") return ctrlTouchMove(e);
+  ctrlMouse(e);
+});
+for (const type of ["pointerup", "pointercancel"]) canvasEl.addEventListener(type, e => {
+  if (!ctrl.on) return;
+  if (e.pointerType === "touch") return ctrlTouchUp(e);
+  const name = CTRL_BUTTONS[e.button] || "left";
+  if (ctrl.held.has(name)) ctrlButton(name, false);
+});
+canvasEl.addEventListener("contextmenu", e => { if (ctrl.on) e.preventDefault(); });
+$("viewer").addEventListener("wheel", e => {
+  if (!ctrl.on || !ctrlTarget().ok) return;
+  e.preventDefault();
+  const unit = e.deltaMode === 1 ? 40 : e.deltaMode === 2 ? 800 : 1;
+  ctrlSend([{ scroll: [e.deltaX * unit, e.deltaY * unit] }]);
+}, { passive: false });
+// Keys while the view has focus go to the Frame (⌘ as Ctrl on a Mac); click elsewhere to stop.
+function ctrlKey(e, down) {
+  if (!ctrl.on || document.activeElement !== $("viewer") || !ctrlTarget().ok) return;
+  let code = EVDEV[e.code];
+  if (IS_MAC && (e.code === "MetaLeft" || e.code === "MetaRight")) code = 29;
+  if (!code) return;
+  e.preventDefault();
+  if (down && e.repeat && code >= 29 && [29, 42, 54, 56, 97, 100, 125, 126].includes(code)) return;
+  ctrlSend([{ key: code, down }]);
+}
+$("viewer").addEventListener("keydown", e => ctrlKey(e, true));
+$("viewer").addEventListener("keyup", e => ctrlKey(e, false));
+
+// Phone keyboard: the field types plain text; the buttons send single keys and clicks.
+function ctrlTap(code) { ctrlSend([{ key: code, down: true }, { key: code, down: false }]); }
+let ctrlWarned = false;
+function ctrlText(text) {
+  const plain = text.replace(/[^\x20-\x7e\n\t]/g, "");
+  if (plain !== text && !ctrlWarned) {
+    ctrlWarned = true;
+    toast("Control types plain letters, numbers and symbols. For accents and emoji, use Keyboard and trackpad below.");
+  }
+  for (const part of plain.match(/[\s\S]{1,500}/g) || []) ctrlSend([{ text: part }]);
+}
+$("ctrlType").addEventListener("beforeinput", e => {
+  if (e.isComposing) return;
+  e.preventDefault();
+  if (!ctrlTarget().ok) return ctrlShow();
+  if (e.inputType === "insertLineBreak" || e.inputType === "insertParagraph") return ctrlTap(28);
+  if (e.inputType === "deleteContentBackward") return ctrlTap(14);
+  if (e.inputType === "deleteContentForward") return ctrlTap(111);
+  const text = e.data ?? e.dataTransfer?.getData("text/plain") ?? "";
+  if (text) ctrlText(text);
+});
+$("ctrlType").addEventListener("compositionend", e => { if (e.data) ctrlText(e.data); $("ctrlType").value = ""; });
+$("ctrlKeys").addEventListener("click", e => {
+  const k = e.target.closest("[data-ctrl-key]"), c = e.target.closest("[data-ctrl-click]");
+  if (!ctrlTarget().ok) return ctrlShow();
+  if (k) ctrlTap(+k.dataset.ctrlKey);
+  if (c) ctrlClick(c.dataset.ctrlClick);
+});
+
 // ---- controls ----
 $("refreshAll").onclick = () => { refresh(); loadTitles(); };  // titles too: the Frame may have been asleep at start
 document.addEventListener("keydown", e => {
@@ -386,6 +386,7 @@ _stream_proc = None


 def stream_command(query):
+    """The ffmpeg that streams H.264: the headset view, or one panel's own window (src=panel)."""
    q = parse_qs(query)
    try:
        height = int((q.get("h") or ["720"])[0])
@@ -395,6 +396,16 @@ def stream_command(query):
    if height not in STREAM_HEIGHTS or fps not in STREAM_FPS:
        raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400)
    rate = 3 if height == 720 else 6  # Mbit/s
+    if q.get("src") == ["panel"]:
+        # The panel's own pixels (x11grab of its window: gamescope keeps them), fitted
+        # to the height asked for. It stays still however the wearer moves their head.
+        window, display = panel_target(q)
+        return (f"DISPLAY={display} ffmpeg -hide_banner -loglevel error -nostdin -f x11grab -framerate {fps} "
+                f"-window_id {window} -i {display} "
+                f"-vf \"scale=-2:'trunc(min({height},ih)/2)*2',format=yuv420p\" -c:v libx264 -preset ultrafast "
+                f"-tune zerolatency -g {fps * 2} -bf 0 -b:v {rate}M -maxrate {rate}M -bufsize {rate // 2 or 1}M "
+                f"-x264-params aud=1:repeat-headers=1 -f h264 - & p=$!; "
+                f"exec >&-; cat >/dev/null; kill $p 2>/dev/null; wait $p")
    return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; "
            f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} "
            f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency "
@@ -819,6 +830,109 @@ def remote_input(body):
    return _input.send([input_event(e) for e in events])


+# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
+
+PANEL_WINDOW = re.compile(r"^\d{1,10}$")
+PANEL_DISPLAY = re.compile(r"^:\d{1,2}$")
+TOUCH_BUTTONS = ("left", "right", "middle")
+
+
+def panels():
+    """The headset's app panels (window, display, name, size) and which has focus."""
+    return json.loads(ssh("python3 - panels", stdin=(HERE / "frame_touch.py").read_text(), timeout=20))
+
+
+def panel_target(q):
+    window, display = (q.get("window") or [""])[0], (q.get("display") or [""])[0]
+    if not PANEL_WINDOW.match(window) or not PANEL_DISPLAY.match(display):
+        raise Failure("window must be a window id and display an X display such as :1", 400)
+    return window, display
+
+
+def panel_capture(query):
+    """One frame of a panel's own window, as PNG (its pixels, without the room around it)."""
+    window, display = panel_target(parse_qs(query))
+    return ssh(f"DISPLAY={display} timeout 10 ffmpeg -hide_banner -loglevel error -nostdin -f x11grab "
+               f"-window_id {window} -i {display} -frames:v 1 -f image2pipe -c:v png -", timeout=20, text=False)
+
+
+def touch_event(event):
+    """A frame_touch.py event with only the fields it knows, in range."""
+    if not isinstance(event, dict):
+        raise Failure("each touch event must be an object", 400)
+
+    def num(name, limit):
+        value = event.get(name)
+        if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
+            raise Failure(f"{name} must be a number", 400)
+        return max(-limit, min(limit, round(float(value), 4)))
+    out = {}
+    if "fx" in event or "fy" in event:
+        out.update(fx=num("fx", 1), fy=num("fy", 1))
+        window = event.get("window")
+        if isinstance(window, bool) or not isinstance(window, int) or window <= 0:
+            raise Failure("window must be the panel's window id", 400)
+        out["window"] = window
+    for name in ("dx", "dy"):
+        if name in event:
+            out[name] = num(name, INPUT_MOVE_LIMIT)
+    if "button" in event:
+        if event["button"] not in TOUCH_BUTTONS:
+            raise Failure(f"button must be one of {', '.join(TOUCH_BUTTONS)}", 400)
+        out["button"], out["down"] = event["button"], event.get("down") is not False
+    if "scroll" in event:
+        sc = event["scroll"]
+        if not isinstance(sc, list) or len(sc) != 2:
+            raise Failure("scroll must be [dx, dy]", 400)
+        out["scroll"] = [num_value(v, 5000) for v in sc]
+    if "key" in event:
+        key = event["key"]
+        if isinstance(key, bool) or not isinstance(key, int) or not 0 < key < 768:
+            raise Failure("key must be a Linux key code", 400)
+        out["key"], out["down"] = key, event.get("down") is not False
+    if "text" in event:
+        text = event["text"]
+        if not isinstance(text, str) or not 0 < len(text) <= INPUT_TEXT_LIMIT:
+            raise Failure(f"text must be 1 to {INPUT_TEXT_LIMIT} characters", 400)
+        out["text"] = text
+    if not out:
+        raise Failure("touch event has nothing to do", 400)
+    return out
+
+
+def num_value(value, limit):
+    if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
+        raise Failure("scroll values must be numbers", 400)
+    return max(-limit, min(limit, round(float(value), 2)))
+
+
+class TouchAgent(InputAgent):
+    """frame_touch.py on the Frame, fed events over one long-lived ssh. Nothing to install:
+    it uses gamescope's own input socket and the libei that's on the image."""
+
+    def __init__(self):
+        super().__init__(source=HERE / "frame_touch.py", packages=[])
+
+    def deliver(self, report, force=False):
+        return ""
+
+    def command(self, folder=""):
+        code = base64.b64encode(self.source.read_bytes()).decode()
+        return "python3 -u -c " + shlex.quote(
+            f"import base64;exec(compile(base64.b64decode('{code}'),'frame_touch','exec'))")
+
+
+_touch = TouchAgent()
+
+
+def remote_touch(body):
+    """{"events": [...]} points, clicks, scrolls and types into the focused panel."""
+    events = body.get("events", [])
+    if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
+        raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
+    return _touch.send([touch_event(e) for e in events])
+
+
 def flatpak(body):
    app, action = str(body.get("id", "")), body.get("action")
    if not FLATPAK_ID.match(app):
@@ -1548,7 +1662,7 @@ def _sweep_one(prefix, d):


 POST = {"/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
-        "/api/input": remote_input,
+        "/api/input": remote_input, "/api/touch": remote_touch,
        "/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
        "/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
        "/api/webinstall/cancel": webinstall_cancel}
@@ -1668,6 +1782,10 @@ class Handler(BaseHTTPRequestHandler):
                self.send_json(title_job(url.query))
            elif path == "/api/licenses":
                self.send_json({"notices": licenses()})
+            elif path == "/api/panels":
+                self.send_json(panels())
+            elif path == "/api/touch":
+                self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
            elif path == "/api/input":
                self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
            elif path == "/api/job":
@@ -1693,6 +1811,8 @@ class Handler(BaseHTTPRequestHandler):
                self.send_bytes(*shot_image(url.query))
            elif path == "/api/stream":
                self.stream_video(url.query)
+            elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["panel"]:
+                self.send_bytes(panel_capture(url.query), "image/png", headers=[("X-Capture-Source", "panel")])
            elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]:
                self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")])
            elif path == "/api/screenshot":
@@ -1761,7 +1881,7 @@ class Handler(BaseHTTPRequestHandler):
                proc.wait()
                errors.seek(0)
                err = strip_ansi(errors.read().decode(errors="replace")).strip()
-                raise Failure(err or "The headset view sent no video for 20 s")
+                raise Failure(err or "The Frame sent no video for 20 s")
            chunk = first
            try:
                self.send_response(200)