Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/how-the-frame-works.md

@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
 | Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
 | **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
 | **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
+| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
+| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
 | **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
 | Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
 | `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |

docs/streaming.md

@@ -6,6 +6,7 @@ This covers three directions, plus input:
 - **B. Mac → Frame**: use the Mac's desktop inside the headset.
 - **C. iPhone → Frame**: mirror the phone inside the headset.
 - **Input**: type and point in the Frame from the Mac or iPhone.
+- **Live view and Control**: watch a panel flat and tap on it to use it.
 
 The confidence labels are the same as in [ssh.md](ssh.md).
 
@@ -162,13 +163,68 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
   and ignores injected pointer motion; `:1` holds apps such as Chromium and
   takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
   There's also a `gamescope-0-ei` (libei) socket.
-- **Not yet tested:** typing and clicking as seen in the headset, and whether
-  it reaches the KDE desktop panel (Plasma is its own session).
+- Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
+  panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
+  Control's own panels, Steam's UI) and, since XTest positions are clamped to
+  `:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
+  the live view (below) has neither limit.
+- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
+  own session).
 - **Known limit:** keys and clicks typed while the link is reconnecting wait
   and are sent once it's back, but anything sent in the moment the Wi-Fi
   drops, before SSH notices, can be lost. Confirming every event would add a
   round trip to each pointer move.
 
+## Live view and Control: watch a panel and tap on it
+
+**Built: Home → Desktop / Headset view → Control.** The live view has two
+sources:
+
+- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
+  moves with the wearer's head, so Control makes the view a trackpad: drag to
+  move the pointer, tap to click, press and hold to right-click, two fingers to
+  scroll. With a mouse, moving over the view moves the pointer.
+- **Desktop**: the app panel in use in the headset, from its own window, so it
+  stays still however the wearer looks around. Control makes taps and clicks
+  land exactly where you put them. Dragging is a mouse drag, press and hold is a
+  right-click, two fingers scroll, and on a computer the mouse, wheel and
+  keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
+  other panel, view only.
+
+Below the view, a text field and key buttons type on the Frame from a phone.
+
+How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
+
+- **Input goes through gamescope's own injection.** gamescope serves an EIS
+  socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
+  it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
+  talks to it with `ctypes`: nothing to install. gamescope offers one device,
+  "Gamescope Virtual Input", with relative and absolute pointer, buttons,
+  scroll and keyboard (Linux key codes; no text capability, so the text field
+  types printable ASCII on a US layout). Its absolute region is unbounded; the
+  pointer uses the focused panel's display coordinates, and gamescope fits each
+  window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
+  positions at two thirds scale. Taps on a 1280×720 page landed on the exact
+  pixel.
+- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
+  root), on either X display. In the OpenVR backend focus moves only on SteamVR
+  overlay events (the controller's laser entering or clicking a panel), or to a
+  new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
+  with `gamescopectl focus_info`, which writes to the journal). Neither
+  `GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
+  gamescope command does. So Control follows the wearer: whatever they last
+  used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
+  gets a connector of its own and doesn't hold focus.
+- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
+  apart).
+- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
+  -window_id <window> -i :<display>` works on gamescope's redirected windows,
+  while grabbing the root gives black. It streams as H.264 like the headset view
+  (about 30 fps at 720p).
+- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
+  text box in the panel and the text field typed into it; a trackpad move went
+  exactly (+40, +25).
+
 Our own `uinput` keyboard and mouse would also work (`steamos` is in the
 `input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
 remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.