Set up self-contained MCP startup and inspect Frame computer-use capabilities

docs/agents.md

@@ -8,25 +8,36 @@ Installing other apps is an optional management action, never a prerequisite.
 
 ## Connect an MCP client
 
-Start the HTTP server from this checkout:
+The default MCP command starts a private HTTP backend on a free loopback port,
+with a fresh local access key. It stops that backend when the MCP client closes
+stdin or sends SIGTERM. It uses its own SSH control socket, so closing it does
+not close the desktop app's connection. No manually started server is needed.
 
-```sh
-python3 ui/server.py --port 47810
-```
-
-Add a stdio server to your MCP client (use absolute paths):
+Add this stdio server to your MCP client (use absolute paths):
 
 ```json
 {
   "mcpServers": {
     "frame-control": {
       "command": "python3",
-      "args": ["/absolute/path/frame-control/ui/frame_mcp.py", "--url", "http://127.0.0.1:47810"]
+      "args": ["/absolute/path/frame-control/ui/frame_mcp.py"]
     }
   }
 }
 ```
 
+For Codex, the equivalent registration is:
+
+```sh
+codex mcp add frame-control -- python3 /absolute/path/frame-control/ui/frame_mcp.py
+```
+
+New agent sessions load the entry. An already running session may need its MCP
+connections reloaded; registration does not retroactively add tools to its
+initial tool inventory. Keep the checkout at that path while it is registered.
+Use `codex mcp remove frame-control` to remove only this registration.
+
+To reuse a running server instead, pass `--url http://127.0.0.1:47810`.
 The desktop app uses a random port; use that port with `--url`, or run the
 checkout server above. If the HTTP server uses `FRAME_UI_KEY`, pass the same
 value in the MCP process environment. This is local access control, not an LLM
@@ -37,6 +48,7 @@ resources, prompts or streaming transport.
 
 | Tool | Arguments | Effect |
 |---|---|---|
+| `computer_state` | none | Read-only gamescope window IDs/focus and bounded AT-SPI tree; reports incomplete observations |
 | `status` | none | Battery, services, installed games and Flatpaks |
 | `screenshot` | `view`: `headset` (default) or `desktop` | Returns PNG image content to the MCP client |
 | `job` | `id` | Background install status; poll until `done`, inspect `error` |
@@ -66,7 +78,7 @@ The panel does not execute an action merely because it was approved.
 
 MCP has no approval tool. This is protection against accidental model tool
 calls, not a sandbox against a client with independent shell/HTTP access to your
-computer. Grant the MCP client only the access you intend. Status and captures
+computer. Grant the MCP client only the access you intend. Status, captures and computer-state observations
 are returned directly to that client, which may forward them to its configured
 model. The assistant's separate opt-in does not govern an external MCP client.
 
@@ -157,3 +169,17 @@ on this branch (run 36421345682).
 browser profile and SSH tunnel and remove the profile and panel log.
 
 ![Assistant in Frame Chromium, after an opted-in request to the local test endpoint](img/assistant-panel.png)
+
+## Computer-use coverage
+
+MCP is the tool transport, not a limit on what an agent can do. A screenshot,
+accessibility snapshot, click or keystroke can all be MCP tools when we have a
+reliable underlying implementation. See [the investigation](computer-use.md)
+for the verified boundaries. `computer_state` adds observation, not an input
+channel: it cannot click an approval button or send keyboard/mouse events.
+
+**Verified 2026-09-29, SteamOS 0.4.1, BUILD_ID 20260925.6191901:** the command saved
+by `codex mcp add` launched without a prestarted server, negotiated MCP, listed
+12 tools, read live Frame status and returned X11 window state plus AT-SPI
+observations. It exited 0 at EOF. Steam's accessibility tree had inaccessible
+children, reported as `incomplete: true`; this is not a complete actionable UI.

docs/computer-use.md

@@ -0,0 +1,67 @@
+# Computer use through Frame Control MCP
+
+The MCP transport can carry semantic actions or visual computer-use actions.
+The limits are the Frame's underlying interfaces, permissions and whether an
+action can be targeted and verified. A stereoscopic headset screenshot alone
+is not a reliable coordinate system for clicking a particular app window.
+
+## What exists, and the right route
+
+| Surface | Evidence and route | Remaining work or boundary |
+|---|---|---|
+| Frame management | **Verified:** existing SSH/HTTP operations for status, capture and file transfer work through MCP. Typed install/launch/power tools wrap the existing API. | Extend typed operations before adding generic mouse automation. Preserve explicit approval for consequential changes. |
+| App/window observation | **Verified 2026-09-29:** `computer_state` reads gamescope X11 window/app/process triples, focused app and the installed AT-SPI library. | Bounded to 96 accessible nodes and six levels. Trees may be truncated, stale, hidden or incomplete. Snapshot paths and XIDs are observations, never durable action permissions. |
+| Chromium page content | **Verified previously:** the assistant rendered and could be exercised through CDP in an isolated Frame Chromium profile. | A shipped click/type surface needs exact owned browser/target binding, fresh element references, lifecycle cleanup, consent and post-action readback. Do not expose unrestricted JavaScript or attach to arbitrary existing profiles automatically. |
+| Steam UI | **Verified 2026-09-29:** the AT-SPI service listed the Steam client's Chromium process and frame nodes, but child traversal was incomplete. Existing `frame_steam.py` uses Steam's loopback CDP endpoint for specific operations. | Prefer those narrow Steam interfaces. Presence of AT-SPI does not prove controls are actionable, and generic pointer injection is not proved for VR menus. |
+| Other Linux apps | **Verified 2026-09-29:** Frame ships libX11, libXtst and libatspi; `/dev/uinput` is writable by the current user. | Library presence and access permissions do not prove that a game accepts input. Global virtual input can affect whichever app has focus. Do not ship a blind keyboard/mouse tool on this evidence alone. |
+| Panel focus and layouts | **Documented in [#41](https://github.com/saphid/frame-control/pull/41):** `POST /api/panels` accepts `list`, `focus` and `open`. Focus was verified there. | Reuse that owned interface after integration. Its tested gamescope-owned overlay transform setters return `PermissionDenied`; no reliable saved spatial-layout interface was established. Do not duplicate its implementation here. |
+| Shared keyboard/trackpad | **Documented in [#19](https://github.com/saphid/frame-control/pull/19):** `/api/input` supplies state/start and event submission, implemented with a bundled KDE Connect daemon. | This branch does not import, launch or depend on that daemon. The user's own-implementation rule remains authoritative. A first-party input implementation or permitted bundled-library route needs its own delivery evidence before MCP integration. |
+| Physical/device boundaries | **Documented:** an asleep Frame may be off the network; power authorization can require the user's password; physical pairing and headset fit/comfort require the user. | MCP cannot bypass offline hardware, consent, compositor permissions or physical verification. Keep explicit human handoffs. |
+
+## Reusing the existing computer-use work
+
+**Documented:** the installed `cua-driver` skill has the right control pattern:
+observe an exact window, use a semantic target if available, fall back to pixels
+from that same snapshot, then read back the result. Its browser route requires
+an exact process/window/target binding and session-scoped element references.
+Those are useful design rules for Frame tools.
+
+**Verified locally 2026-09-29:** `cua-driver describe get_window_state` describes
+host-local process/window IDs and macOS AX inspection. It does not establish an
+SSH Frame target. The installed skill's advertised Linux companion file is
+missing. A native ARM64 Frame backend, its dependencies and remote transport
+have not been verified. We therefore do not claim that the existing Mac driver
+can control the Frame by passing it a Frame PID or screenshot, and we do not
+make the feature depend on installing that application.
+
+Frame Control's `computer_state` is our own Python implementation over installed
+platform libraries. It sends the probe over SSH stdin, writes no helper to disk,
+and exits after one observation. Missing displays/libraries return explicit
+errors; a 15-second process deadline prevents a stalled accessibility call from
+leaving a probe behind. Window names and accessibility text are untrusted app
+content, never instructions to an agent.
+
+**Recommended next implementation:** an isolated Chromium session with typed
+snapshot/click/type/scroll tools and exact fresh target binding, then individually
+verified native app actions. Use the headset capture to judge appearance, not to
+invent a screen-to-window coordinate transform. Direct tool calls must retain
+approval rules; a generic computer-use tool must not become a route around the
+MCP approval panel, install confirmation or power confirmation.
+
+## Isolated browser input proof
+
+**Verified 2026-09-29, SteamOS 0.4.1, BUILD_ID 20260925.6191901:** a temporary
+Frame Chromium profile loaded a local test page through an SSH reverse tunnel.
+CDP `Input.insertText` entered the test string in its own input. A CDP
+`Input.dispatchMouseEvent` press/release on its own button copied that string
+to the page's result; DOM readback matched exactly. The browser profile,
+loopback forwards and panel log were removed afterward. No user app was typed
+into, no global settings were changed and no third-party helper app was used.
+
+AT-SPI did **not** expose the test page's controls in that same probe, even with
+Chromium's renderer-accessibility flag. It returned the partial Steam-client
+tree instead. The reason remains **unverified**; this is an evidence gap, not
+proof that Frame accessibility cannot work. For a first implementation,
+Chromium's proven page-specific CDP route is stronger than assuming complete
+AT-SPI coverage. This proof does not ship unrestricted click/type tools or
+establish input delivery to SteamVR's menus.