Compare commits

..
Author SHA1 Message Date
saphidandClaude Opus 5.5 0f8aaa393b Store: show owned itch.io games under either VR filter
itch.io's game records don't say which games are VR, so owned purchases
came back with VR unknown and the "VR experiences" filter hid them (found
in a live test: an owned Quest game vanished). Unknown still counts as flat
for every other source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:08:19 +11:00
saphidandClaude Opus 5.5 382fb0f906 itch.io purchases: find overlapping key matches so a partial match can't hide the real start
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:44:38 +11:00
saphidandClaude Opus 5.5 76e90af5b7 itch.io purchases: redact the key when it's percent-encoded twice
_clean (and so _carries_key) now also matches each key character as %25XX,
hex in either case and for either letter case of the character, keeping
the 8+ character starts. Tests: a fully double-encoded key, recased forms,
a start of it, a mix of literal, single and double encoding, and in a
storage URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:20:54 +11:00
saphidandClaude Opus 5.5 5456f2f6f6 itch.io purchases: redact the key when it's percent-encoded after a letter-case change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:48:54 +11:00
saphidandClaude Opus 5.5 2cfb61a208 itch.io purchases: round-4 review fixes for per-hop key checks, redaction, IPv4 and the connect watchdog
- public_apk takes a check(url) callback run on every hop, before any lookup
  or connection; the itch.io download passes its key refusal, so a later
  redirect carrying the key (or ?api_key=) is refused too.
- One redaction function removes the key and its 8+ character starts in any
  letter case, with each character literal or percent-encoded (either hex
  case); used for every itch.io message, keeping error type and retry_after.
- IPv4 is an allow-list too: the whole special-purpose blocks (incl.
  192.0.0.0/24, which Python 3.9 partly calls global) are refused besides
  requiring is_global.
- No watchdog thread: connect and TLS handshake each get what is left of the
  shared deadline as their socket timeout, then the normal read timeout, so
  nothing can shut a connection after it has been handed over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:40:54 +11:00
saphidandClaude Opus 5.5 c6e4362c66 itch.io purchases: round-3 review fixes for IPv6, keys in storage URLs, HTTP errors and connect time
- public_address treats IPv6 as an allow-list: global unicast (2000::/3)
  only, minus 6to4, Teredo, documentation, ORCHID and benchmarking blocks.
  Site-local fec0::/10 (is_global on Python 3.9) and everything else outside
  2000::/3 is refused.
- A storage URL from itch.io (Location or JSON url) that contains the API
  key, or 8+ characters of it in any case or encoding, or an api_key
  parameter, is refused before any connection. Storage-side errors lose the
  key and keep their type and retry_after.
- http.client errors (BadStatusLine, IncompleteRead, RemoteDisconnected)
  around every api.itch.io open and read become readable, key-free
  SourceErrors raised from None.
- Connecting shares one monotonic deadline (20 s, or the caller's timeout if
  shorter) across all addresses and both connect and TLS handshake, with a
  watchdog for a stalled handshake; reads keep their own timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:28:15 +11:00
saphidandClaude Opus 5.5 88d61cb3b6 itch.io purchases: round-2 review fixes for addresses, revocation races and key errors
- public_address tests each property explicitly (reserved, private,
  loopback, link-local, multicast, unspecified) as Python 3.9's is_global
  passes 4000::1 and 64:ff9b:1::/48. IPv6 forms carrying IPv4 (mapped,
  compatible, NAT64 incl. local-use, 6to4, Teredo) are refused outright.
- Every checked address is tried in turn (10 s each to connect and
  handshake), each pinned to its own address with the same TLS host name.
- _use reads the key and its connection generation together, and deletes
  only if both still match: an older refusal can't undo a reconnect, even
  with the same key.
- Key errors are judged on itch.io's own words before redaction, and any
  "invalid key" error outside /profile is confirmed with /profile before
  the key is deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:11:57 +11:00
saphidandClaude Opus 5.5 2d63dacdf9 itch.io purchases: review fixes for redirects, key redaction, races and revoked keys
- Storage downloads no longer go through a "any host name" redirect policy.
  _web.public_apk resolves each hop, refuses loopback, private, link-local,
  shared, reserved and multicast addresses (odd IPv4 spellings and IPv4
  mapped into IPv6 included), connects to the checked address, and checks
  again on every redirect. No credentials are sent there. _web.ANY and the
  extra open_url/apk parameters are gone; _web.apk's streaming is shared as
  save_apk.
- api.itch.io redirects are no longer followed with the key: the download
  step reads a 30x Location, a JSON body's url, or takes the file itself,
  since the live response is unverified.
- Error text loses the key, and any start of it from 8 characters, before
  joining and truncating.
- Saving and deleting the key are serialised; a connect whose /profile check
  finishes after a disconnect (or another connect) does not save. The key
  file is written through a unique mkstemp file.
- 401/403 from api.itch.io deletes the key only once /profile confirms the
  key itself is refused, including at download time with cached metadata.
  Storage-host 403s are download errors, not rate limits.
- Docs list which parts are verified against the public reference and which
  are assumed or unknown without a live account.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:59:15 +11:00
saphidandClaude Opus 5.5 adefdf4398 Store: install the Android builds of games you own on itch.io
Opt-in: paste an itch.io API key under My itch.io purchases in the store's
sources dialog. It is checked with GET /profile, saved 0600 in the app data
dir (itch-account.json), and only ever sent to api.itch.io as a Bearer
header. A new "itch.io (my purchases)" source lists owned Android games
(/profile/owned-keys, all pages), labelled "Owned on itch.io"; installs
pick the best Android upload (/games/ID/uploads) and download it through
/uploads/ID/download into the normal APK install path. A refused key is
never saved, or is deleted if itch.io later revokes it; rate limits and
network errors stay per-source and readable. The key is kept out of URLs,
errors, logs, API responses and the page.

Tests use recorded JSON and a fake HTTPS layer under urllib's real opener.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:50:30 +11:00
saphidandClaude Opus 5.5 2c3f8b3e0c APK sources: optional request headers and a redirect host policy for downloads
_web.open_url/apk take headers and the hosts a download may redirect to;
_web.ANY allows any HTTPS host name (no IP literals, credentials or other
ports). Authorization is still dropped on every redirect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 12:50:23 +11:00
Alex Southwell 5f3798b79b Merge pull request #88 from saphid/fix/tests-real-ssh-dir
Tests: keep every test (and the servers they start) off the real ~/.ssh
2026-10-12 16:34:34 +11:00
saphidandClaude Opus 5.5 6e0b140efc e2e: run only inside scripts/e2e.sh's container, and never start a server otherwise
test_agents had no harness.require() and FrameTestCase started ui/server.py
unconditionally, so plain discovery over tests/e2e started a server on this
computer's real app data, ~/.ssh and cache. FRAME_E2E=1 alone outside the
container did the same with the real cache.

require() now also needs FRAME_E2E_CONTAINER=1 (set in compose.yaml) and the
container runtime's marker file, and says why it refuses; FrameTestCase and
Server.start call it before starting anything; test_agents calls it at
module level. test_sandbox checks discovery over tests/e2e, with and
without FRAME_E2E=1, skips everything and starts no server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:21:09 +11:00
saphidandClaude Opus 5.5 226ec40720 Tests: keep the e2e container's real home; sandbox the per-user cache folders too
tests/e2e/test_android imported its APK helpers from the sandboxed
test_frame_apk, so the HOME move reached the whole e2e process: frame_connect
wrote keys and config into the temp home while OpenSSH, which finds ~/.ssh
from the password database, kept reading the container user's. The helpers
now live in the sandbox-free tests/apk_fixtures.py, and the e2e harness sets
the data dir, telemetry and compat DB guards itself (it used to get them
through that import).

sandbox.py also points LOCALAPPDATA, APPDATA and the XDG folders into the
throwaway home, so test servers on Windows and Linux no longer warm indexes
or sweep staging folders in the real app cache. test_sandbox checks both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:56:26 +11:00
saphidandClaude Opus 5.5 f1aaed6279 Tests: keep every test (and the servers they start) off the real ~/.ssh
Servers started by test_server's OneServer and LocalMode tests got no
FRAME_CONTROL_SSH_DIR, so each one imported the real ~/.ssh/config into a
fresh throwaway devices.json and seeded a pin into the real
~/.ssh/frame-control-hosts: about a dozen files per suite run.

tests/sandbox.py now gives the whole run a throwaway HOME/USERPROFILE and
FRAME_CONTROL_SSH_DIR (inherited by child processes), every test module
imports it first, and frame_connect.SSH_DIR honours FRAME_CONTROL_SSH_DIR
like frame_devices.ssh_dir(). test_sandbox checks the paths and reruns the
tests that leaked from the original environment, asserting nothing under
the real ~/.ssh changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:40:01 +11:00
saphidandClaude Opus 5.5 b89538f506 Release 0.4.4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:36:27 +11:00
Alex Southwell 311fb5f6c7 Merge pull request #85 from saphid/fix/host-key-recovery
Recover from changed, missing and refused SSH keys inside the app
2026-10-12 15:24:25 +11:00
saphidandClaude Opus 5.5 93ae2b1bd1 Host key recovery: quote the jump hop's ProxyCommand; IPv6 jumps (review pr85-review-r3)
- Every word of the hardened ProxyCommand is quoted for the shell ssh runs
  it with (shlex.quote; list2cmdline on Windows) and its % doubled for
  ssh's token expansion. The ssh path is always the absolute one, spaces
  and all; an ssh at a path with $(), backticks, quotes or ; no longer runs
  shell code.
- The forward is -W [%h]:%p (an IPv6 headset), and an IPv6 jump host is
  passed bare with -p.
- The stand-in ssh now runs ProxyCommand as OpenSSH does (tokens, then
  /bin/sh -c exec), with regressions for hostile paths and IPv6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:15:44 +11:00
saphidandClaude Opus 5.5 12bd10ee42 Host key recovery: harden jump hops, keep identity history (review pr85-review-r2)
- Key check through a ProxyJump host: build the hop as our own ProxyCommand
  (no agent/X11/port forwarding, no LocalCommand or RemoteCommand, no tty,
  no further proxy), with -F none for the device; ProxyJump's own ssh would
  inherit ForwardAgent and LocalCommand from ~/.ssh/config. Refuse
  ProxyCommand and multi-hop routes, which can't be hardened.
- Identity history is judged on the current registry (named_route used a
  frozen copy and fell back to accept-new).
- A key ssh saves on first contact is recorded, made private and the route
  tightened as soon as it's saved, even if the login is then refused, and
  for keys commands save (the loop settles them every tick).
- identity_established is a device-level flag that never clears (migrated
  from last_ok), so editing or removing the address that worked can't make
  a headset new again.
- The handshake waits for its reader's end marker before taking ssh's exit
  as the end, so a late "Permission denied" isn't lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 14:56:02 +11:00
saphidandClaude Opus 5.5 8aeeaef666 Host key recovery: fail closed on a lost pin, read keys only (review pr85-review-r1)
- Key check: ssh writes the key during the key exchange and is stopped at once;
  no authentication, agent, forwarding, commands (-N), terminal or local
  command, all forced ahead of ~/.ssh/config, and -F none unless a jump host
  needs the config. A device accepting "none" gets nothing.
- accept-new only for a headset that never had an identity: the saved key's
  fingerprint is recorded in devices.json, and a recorded identity, an
  existing pin file or a connection that ever worked makes a deleted, empty,
  unreadable or unparseable pin fail closed ("missing or can't be read").
- forget-identity is refused (it let the next contact save any key unseen).
- A key read before the headset's port, user or that address changed can't be
  trusted.
- Without a ControlMaster (Windows) the handshake's stderr goes to a file read
  as it grows, not a pipe.
- Pin files and the check's throwaway known_hosts get make_private.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 14:32:08 +11:00
saphidandClaude Opus 5.5 5fb02770db Recover from changed, missing and refused SSH keys inside the app
0.4.2 sent $exception reports from the status poll (POST /api/comfort status,
frame_auth) on Linux and Windows for three states the person could not fix from
the app:

- A changed host key (REMOTE HOST IDENTIFICATION HAS CHANGED): the advice was
  "Run Set Up Connection again", which checks ~/.ssh/known_hosts and never
  replaces the headset's pinned key; Forget identity trusted whatever answered
  next without showing it.
- No pinned key ("No ED25519 host key is known for frame-control-<id> and you
  have requested strict checking"): every command said
  StrictHostKeyChecking=yes, so any command that ran before (or without) the
  connector's first handshake saving the key failed.
- Permission denied (publickey,password): not something a retry fixes.

Commands now use accept-new until a key is saved and yes after (accept-new
never replaces a saved key). A changed key fails at the identity stage with its
fingerprint; the banner and Devices tab offer "Check the new key", which reads
the key with a throwaway known_hosts file and no login, shows it beside the
saved one, and pins exactly that key only when the person confirms. Refused
keys point to Set Up Connection. ssh's own host key and login refusals are
marked as link failures and frame_auth joins the expected categories, so each
is reported at most once a session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 14:10:50 +11:00
55 changed files with 3282 additions and 196 deletions

No files matched your search

+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "frame-control",
"version": "0.4.3",
"version": "0.4.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "frame-control",
"version": "0.4.3",
"version": "0.4.4",
"license": "MIT",
"devDependencies": {
"electron": "^44.4.5",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "frame-control",
"productName": "Frame Control",
"version": "0.4.3",
"version": "0.4.4",
"description": "Desktop app for managing a Valve Steam Frame over SSH",
"private": true,
"main": "main.js",
+82
View File
@@ -82,6 +82,88 @@ is an error rather than a reason to fall back to `index-v1.jar`. `entry.jar`
must be signed with SHA-2 (SHA-1 is still accepted for legacy `index-v1.jar`).
Removing a repository clears its state.
## My itch.io purchases
The store can also install the Android (Quest) builds of games you **own** on
itch.io: bought, claimed free, from a bundle or a gift. It is opt-in. In the
store, open the sources dialog and use **My itch.io purchases**: create an API
key on itch.io (account settings › API keys) and paste it there. Frame Control
checks it with itch.io (`GET /profile`) before saving it; a wrong or revoked key
is reported and nothing is saved. Once connected, a source called
**itch.io (my purchases)** appears, and its games are marked **Owned on
itch.io** instead of Free. **Disconnect itch.io** deletes the key file.
How it works, using the documented
[server-side API](https://itch.io/docs/api/serverside) with
`Authorization: Bearer <key>`:
- `GET https://api.itch.io/profile/owned-keys` lists your download keys, page by
page until an empty page (500 per page, at most 40 pages). Games whose
`traits` don't include `p_android` are left out; one entry per game, with the
newest key.
- Opening a game asks `GET /games/<game_id>/uploads?download_key_id=<key id>`
for its files. Android uploads are those marked `p_android` (in `traits`, or
as a `p_android`/`platforms.android` flag) or named `.apk`. `.apk` files come
first, then full games before demos, newest first. Soundtracks, pre-orders and
externally hosted files are skipped.
- Installing asks `GET /uploads/<upload_id>/download?download_key_id=<key id>`.
The answer accepted is a redirect to itch.io's storage host, a JSON body
with a `url` field naming it, or the file itself. Frame Control follows the
first two without the key, to HTTPS on port 443 only, and only to host names
whose every address is public. For IPv4 that means outside every special
block (`0/8`, `10/8`, `100.64/10`, `127/8`, `169.254/16`, `172.16/12`,
`192.0.0/24`, `192.0.2/24`, `192.88.99/24`, `192.168/16`, `198.18/15`,
`198.51.100/24`, `203.0.113/24`, `224/4`, `240/4`), however written. IPv6 is
an allow-list: global unicast (`2000::/3`) only, minus 6to4, Teredo,
documentation, ORCHID and benchmarking blocks, so mapped or NAT64 IPv4,
site-local, unique-local and the rest are all refused. It connects only to
the addresses it checked, trying each in turn within one 20-second budget
for connecting and the TLS handshake, and checks again on each further
redirect. A storage URL, first or after any redirect, that contains the API
key (or 8+ characters of it, in any case, plain or percent-encoded) or an
`api_key` parameter is refused before it is looked up, so the key never
reaches the storage server. The APK then takes
the normal install path, with the same ABI, OpenXR layer and Meta-services
checks and warnings as any other store download.
The key is stored in `frame_host.data_dir('itch-account.json')`
(`~/Library/Application Support/Frame Control/itch-account.json` on macOS),
written through a new 0600 temporary file and moved into place. It is only
sent to `https://api.itch.io`, in the `Authorization` header, never in a URL;
Frame Control does not follow api.itch.io's redirects with it. The key, or any
part of it from 8 characters, never appears in the page, API responses, errors
or logs; the page clears the field as soon as it is sent. A disconnect made
while a key is still being checked wins: that key is not saved.
Owned-game lists are kept in memory for ten minutes and never written to the
disk cache.
When api.itch.io answers 401 or 403, Frame Control asks `GET /profile` with the
same key. If that is refused too (or itch.io says "invalid key"), the key is
deleted and the store says to connect again; otherwise only that game or
download is refused, and the key stays. Rate limits (429) and network errors
only fail that source's results, with a readable message. A storage host's
refusal is reported as a download error, not as a rate limit.
### What is verified and what isn't
Not tested against a live itch.io account at all: everything below was tested
with recorded and hand-made responses (`tests/test_itch_purchases.py`) only.
| Part | Status |
|---|---|
| Bearer key, `GET /profile`, `/profile/owned-keys` paging and its fields, the `errors` array | Documented in the public reference (checked 2026-10-13) |
| `GET /games/<id>/uploads` and `download_key_id` | Not in the public reference; paths and parameter from itch.io's app client, [go-itchio](https://github.com/itchio/go-itchio) |
| Upload fields (`filename`, `display_name`, `size`, `storage`, `type`, `traits`, `updated_at`, `demo`, `preorder`) | Assumed from go-itchio and the API's snake_case convention |
| The `p_android` trait, on games and uploads | Assumed from itch.io's `p_windows`/`p_linux`/`p_osx` naming |
| What `/uploads/<id>/download` returns | Unknown: go-itchio only builds the URL and hands it to a downloader, so a redirect, a JSON `url` and the file are all accepted |
| Whether that endpoint accepts the key as a Bearer header | Unknown: go-itchio puts it in the query string, which Frame Control refuses to do |
| itch.io's storage host names and addresses | Unknown; any public HTTPS host is allowed |
Limits: a game whose page doesn't mark Android, or whose Android build is
inside a `.zip`, won't install. Android builds have no version codes on
itch.io, so only the newest upload can be installed. Owning a game doesn't make
it run on the Frame: Quest builds that need Meta's platform services won't.
## Publish your own repository
Only publish free APKs you own or have the developer's permission to distribute.
+2
View File
@@ -60,6 +60,8 @@ does not suppress the other at the aggregator level. Queries filter the current
feed window locally: this is not an exhaustive historical itch search. Only
explicit zero-price Android entries are returned. Covers are exposed in
`images`; absent screenshots, APK version, ABI and minimum SDK stay unknown.
Games the user owns on itch.io are a separate, opt-in source that downloads
through their own API key: see [My itch.io purchases](apk-repos.md#my-itchio-purchases).
Curated GitHub entries include publisher artwork and plain-language summaries.
Repository image URLs are pinned to inspected commits. Open Brush screenshots
come from its README-linked Steam listing; SuperTux uses the upstream gameplay
+56 -6
View File
@@ -104,7 +104,9 @@ connection is an SSH ControlMaster that every command shares; when it dies (the
headset slept or left the network) the connector notices and starts again. On
Windows, where OpenSSH can't share a connection, the same handshake runs once
and each command then connects on its own; a command that can't reach the
headset makes the connector start again.
headset makes the connector start again. There ssh's stderr goes to a file that
the connector reads as it grows, not a pipe: Windows' OpenSSH can block writing
to a redirected pipe.
Once connected, every `ssh`, `scp` and `rsync` the app runs gets
`-o HostName=<address> -o HostKeyAlias=frame-control-<id>
@@ -121,9 +123,57 @@ lease that moved) would look like a new host there. The app keeps one known_host
file per headset instead, so saving or forgetting one headset's key never touches
another's: a different device answering at one of its
addresses is refused, and the pill says so. A headset's first connection trusts
the key it shows, as Set Up Connection does. After reinstalling SteamOS the
headset has a new key; **Forget identity** on the Devices tab lets the next
connection save the new one.
the key it shows, as Set Up Connection does: while a headset has never had an
identity here, every ssh command gets `StrictHostKeyChecking=accept-new` (the
first contact saves it, whichever command makes it), and from then on
`StrictHostKeyChecking=yes`. `accept-new` never replaces a saved key, so it
can't hide a change. The saved key's fingerprint is also recorded in
`devices.json` (`identity`), along with a flag that never clears
(`identity_established`), as soon as ssh saves the key: even if the login is then
refused. A headset counts as having had an identity if the flag is set, a
fingerprint is recorded, its pin file exists at all, or a connection to it ever
worked (older files get the flag from that when loaded, so editing or removing
the address that worked doesn't undo it); then
a pin file that's deleted, empty, unreadable or not parseable **fails closed**
(*Checking the headset's identity* fails, "missing or can't be read") rather than
trusting whatever answers, and only **Check its key…** saves a key again. It
says when the key it reads matches the recorded fingerprint.
**When the key changes** (SteamOS reinstalled or reset, a new headset at the
same address, or a different device answering there), nothing is accepted by
itself. The connection fails at *Checking the headset's identity*; the banner
and the headset on the Devices tab say so, with **Check the new key…** (also
**Check its key…** on the Devices tab at any time). That reads the key the
headset shows now, and nothing more: ssh writes it to a throwaway known_hosts
file during the key exchange, before any login, and is stopped as soon as it
has. Every other kind of contact is ruled out on the command line, ahead of
anything `~/.ssh/config` says: no authentication of any kind, no agent
(`IdentityAgent=none`, `ForwardAgent=no`), no X11 or port forwarding, no local
or remote command (`-N`), no terminal, and `-F none`. Through a ProxyJump host
it builds that hop itself, as a ProxyCommand whose ssh forwards nothing and runs
no local command (ProxyJump's own ssh would take `ForwardAgent` and
`LocalCommand` from `~/.ssh/config`); it still logs in to the jump host as
configured. Every word of that ProxyCommand is quoted for the shell that runs it
(and `%` doubled for ssh's own tokens), the forward is `[%h]:%p` so an IPv6
headset works, and an IPv6 jump host is passed bare with its port. Through a ProxyCommand from `~/.ssh/config`, or several jump hosts,
it refuses, since those can't be made safe from here: add an address it can
reach directly, or compare the key by hand. It shows the key's SHA256 fingerprint next to
the saved one, with the command that prints it on the headset
(`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`, in Konsole on its desktop).
**This is my headset: trust its new key** then saves exactly that key, replacing
the old one, and reconnects. The server keeps the key it showed for 10 minutes
and saves it only if the page sends back the same fingerprint, and only while
the headset's port, user and that address are unchanged; it never saves whatever
answers next. The API's old `forget-identity`, which deleted the pin so that the
next contact saved any key unseen, is refused. Set Up Connection can't fix a changed key: it checks your
own `known_hosts`, not the headset's file.
**When the headset refuses this computer's key** (`Permission denied
(publickey,…)`), the connection stops at *Logging in*, and the banner and the
Devices tab offer **Set Up Connection…**, which pairs this computer again.
Neither is reported as a fault: the diagnostics report each kind at most once
a session ([privacy.md](privacy.md)).
## One server at a time
@@ -144,7 +194,7 @@ All under the usual `/api/` guards (loopback `Host`, `X-Frame-UI` header).
| `GET /api/devices` | Headsets, the current network, known networks, the next free alias |
| `GET /api/devices/tailscale?id=` | Tailscale peers, likely headsets first |
| `GET /api/devices/mdns?id=` | Headsets found on this network |
| `POST /api/devices` | `{"action": ...}`: `use`, `update` (name, user, port), `remove`, `address-add`, `address-update`, `address-remove`, `address-move`, `test`, `forget-identity`, `name-network`, `setup` (alias, optional host), `retry` |
| `POST /api/devices` | `{"action": ...}`: `use`, `update` (name, user, port), `remove`, `address-add`, `address-update`, `address-remove`, `address-move`, `test`, `identity-scan` (read the key the headset shows: `scan` with `type`, `fingerprint`, `host`, `saved`, `recorded`, `matches`, `recorded_match`), `identity-trust` (`fingerprint`: save the key `identity-scan` showed), `forget-identity` (refused: use the two before), `name-network`, `setup` (alias, optional host), `retry` |
Every host, alias and user is checked against strict patterns before it's
stored, because they end up in ssh arguments and `~/.ssh/config`; nothing goes
@@ -161,7 +211,7 @@ share the format later (it still connects to one host; see
```json
{"version": 1, "active": "f67f8b7e",
"devices": [{"id": "f67f8b7e", "name": "Steam Frame", "alias": "frame", "user": "steamos", "port": 22,
"identity_files": ["~/.ssh/id_ed25519_frame"],
"identity_files": ["~/.ssh/id_ed25519_frame"], "identity": ["SHA256:…"], "identity_established": true,
"addresses": [{"host": "frame.local", "kind": "mdns", "label": "",
"networks": ["n-e0998baa61"], "last_ok": 1790593550.4, "last_rtt_ms": 0.9}]}],
"networks": {"n-e0998baa61": {"name": "Home Wi-Fi", "ssid": null, "gateway": "192.168.1.1",
+4 -2
View File
@@ -101,8 +101,10 @@ scrubbed:
- `token=`, `key=`, `password=` and similar values are replaced
The same error is sent at most once every 10 minutes. When ssh reports that
it couldn't reach the Frame (asleep, away, or not set up yet), that is sent
at most once per kind each time Frame Control runs.
it couldn't reach the Frame (asleep, away, or not set up yet), or that ssh and
the Frame refused each other's keys (a changed host key, or this computer's
key not accepted), that is sent at most once per kind each time Frame Control
runs.
## Report a problem
+87
View File
@@ -0,0 +1,87 @@
"""Small APKs built in memory (binary manifest plus resource table), for test_frame_apk
and the end-to-end tests. Deliberately doesn't import sandbox: tests/e2e runs in a
throwaway container and keeps its real home, which sandbox would move."""
import io
import os
import struct
import sys
import zipfile
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ui'))
import frame_apk # noqa: E402
def pool(strings, utf8=False):
"""A ResStringPool chunk."""
data, offsets = b'', []
for s in strings:
offsets.append(len(data))
if utf8:
b = s.encode()
data += bytes([len(s), len(b)]) + b + b'\0'
else:
data += struct.pack('<H', len(s)) + s.encode('utf-16-le') + b'\0\0'
data += b'\0' * (-len(data) % 4)
start = 28 + 4 * len(strings)
body = struct.pack(f'<{len(strings)}I', *offsets) + data
return struct.pack('<HHIIIIII', 1, 28, 28 + len(body), len(strings), 0, 0x100 if utf8 else 0, start, 0) + body
def manifest(package, label_ref, version_ref, min_sdk, package_raw=True, foreign_label=False):
"""<manifest package versionName><uses-sdk minSdkVersion/><application label icon/></manifest>.
package_raw=False drops the package's raw string (as some repackers do);
foreign_label adds a non-android `label` attribute after android:label.
"""
strings = ['label', 'icon', 'versionName', 'minSdkVersion', 'package', 'manifest', 'uses-sdk',
'application', package, 'junk', 'label', 'versionCode'] # the second 'label' has no android id
resmap = struct.pack('<4I', 0x01010001, 0x01010002, 0x0101021c, 0x0101020c)
resmap = struct.pack('<HHI', 0x0180, 8, 8 + len(resmap)) + resmap
def element(name, attrs):
body = struct.pack('<IIHHHHHH', 0xffffffff, name, 20, 20, len(attrs), 0, 0, 0)
for aname, raw, dtype, value in attrs:
body += struct.pack('<IIIHBBI', 0xffffffff, aname, raw, 8, 0, dtype, value)
return struct.pack('<HHIII', 0x0102, 16, 16 + len(body), 1, 0xffffffff) + body
none = 0xffffffff
chunks = (pool(strings) + resmap
+ element(5, [(4, 8 if package_raw else none, frame_apk.T_STRING, 8),
(2, none, frame_apk.T_REF, version_ref),
(11, none, frame_apk.T_INT_DEC, 210)])
+ element(6, [(3, none, frame_apk.T_INT_DEC, min_sdk)])
+ element(7, [(0, none, frame_apk.T_REF, label_ref), (1, none, frame_apk.T_REF, 0x7f020000)]
+ ([(10, 9, frame_apk.T_STRING, 9)] if foreign_label else [])))
return struct.pack('<HHI', 3, 8, 8 + len(chunks)) + chunks
def resources(values):
"""resources.arsc with package 0x7f; values: {(type id, entry, language, density): global string index}."""
strings = ['French label', 'App label', '2.1', 'res/icon_lo.png', 'res/icon_hi.png', 'res/icon.xml']
pkg_body = b''
for (tid, lang, density), entries in values.items():
cfg = struct.pack('<I4x2s4xH', 64, lang.encode().ljust(2, b'\0'), density).ljust(64, b'\0')
count = max(entries) + 1
offsets, data = [], b''
for i in range(count):
if i in entries:
offsets.append(len(data))
data += struct.pack('<HHI', 8, 0, 0) + struct.pack('<HBBI', 8, 0, frame_apk.T_STRING, entries[i])
else:
offsets.append(0xffffffff)
header = 20 + 64
estart = header + 4 * count
body = struct.pack(f'<{count}I', *offsets) + data
pkg_body += struct.pack('<HHIBBHII', 0x0201, header, header + len(body), tid, 0, 0, count, estart) + cfg + body
pkg_header = struct.pack('<HHII', 0x0200, 288, 288 + len(pkg_body), 0x7f).ljust(288, b'\0')
pkg = pkg_header + pkg_body
table = pool(strings, utf8=True) + pkg
return struct.pack('<HHII', 2, 12, 12 + len(table), 1) + table
def apk(files):
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as z:
for name, data in files.items():
z.writestr(name, data)
return buf.getvalue()
+27 -4
View File
@@ -10,6 +10,7 @@ import atexit
import http.client
import json
import os
import shutil
import socket
import subprocess
import sys
@@ -22,7 +23,22 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path[:0] = [str(ROOT / 'ui'), str(ROOT / 'tests'), str(ROOT / 'tests' / 'smoke')]
ENABLED = os.environ.get('FRAME_E2E') == '1'
# Only inside the compose `host` container (FRAME_E2E_CONTAINER from compose.yaml, plus
# the container runtime's marker file): these tests drive ui/server.py with the real
# home, ~/.ssh and cache, which is only safe in a throwaway container.
REQUESTED = os.environ.get('FRAME_E2E') == '1'
IN_CONTAINER = os.environ.get('FRAME_E2E_CONTAINER') == '1' and (
os.path.exists('/.dockerenv') or os.path.exists('/run/.containerenv'))
ENABLED = REQUESTED and IN_CONTAINER
if ENABLED:
# What tests/sandbox.py does, minus moving HOME: the container's real ~/.ssh is what
# points `ssh frame` at the fake Frame, and OpenSSH reads ~/.ssh from the password
# database, not $HOME. (These tests used to get this through test_frame_apk.)
_data = tempfile.mkdtemp(prefix='frame-control-e2e-')
atexit.register(shutil.rmtree, _data, ignore_errors=True)
os.environ['FRAME_CONTROL_DATA_DIR'] = _data
os.environ['FRAME_CONTROL_TELEMETRY'] = '0'
os.environ['FRAME_COMPAT_DB_URL'] = 'http://127.0.0.1:9'
CTL = os.environ.get('FAKEFRAME_CTL', 'http://fakeframe:9999').rstrip('/')
FAKE_HOST = os.environ.get('FAKEFRAME_HOST', 'fakeframe')
HOME = '/home/steamos'
@@ -30,9 +46,14 @@ SERVER_LOG = os.path.join(tempfile.gettempdir(), 'fakeframe-e2e-server.log')
def require():
"""Call at module level: skips the module unless the fake Frame is up."""
if not ENABLED:
raise unittest.SkipTest('needs the fake Frame: run scripts/e2e.sh (sets FRAME_E2E=1)')
"""Skips (the module, at module level; the class, in setUpClass) unless this is
scripts/e2e.sh's container with the fake Frame up."""
if ENABLED:
return
if REQUESTED:
raise unittest.SkipTest('FRAME_E2E=1 outside the e2e container: these tests use the real home, ~/.ssh '
'and app cache, so they only run in scripts/e2e.sh\'s throwaway container')
raise unittest.SkipTest('needs the fake Frame: run scripts/e2e.sh (sets FRAME_E2E=1)')
# ---- the fake Frame's control port --------------------------------------------
@@ -97,6 +118,7 @@ class Server:
@classmethod
def start(cls):
require() # never a server on this computer's real data
if cls.proc and cls.proc.poll() is None:
return
with socket.socket() as s:
@@ -193,6 +215,7 @@ class FrameTestCase(unittest.TestCase):
@classmethod
def setUpClass(cls):
require()
Server.start()
def setUp(self):
+2
View File
@@ -6,6 +6,8 @@ import sys
import harness
from harness import api, ok, finished, ssh
harness.require()
sys.path.insert(0, str(harness.ROOT / 'ui'))
import frame_mcp
+1 -1
View File
@@ -5,7 +5,7 @@ import unittest
import harness
from harness import HOME, exists, ok, state, upload, wait_for
from test_frame_apk import apk, manifest, resources
from apk_fixtures import apk, manifest, resources
harness.require()
+1
View File
@@ -43,6 +43,7 @@ services:
FAKEFRAME_CTL: http://fakeframe:9999
FAKEFRAME_HOST: fakeframe
FRAME_E2E: "1"
FRAME_E2E_CONTAINER: "1" # tests/e2e/harness.py refuses to run anywhere else
healthcheck:
test: ["CMD", "test", "-f", "/home/tester/.ready"]
interval: 1s
+96 -3
View File
@@ -3,10 +3,20 @@
each step of a connection, and plays a ControlMaster. What each HostName does comes
from $FAKESSH_HOSTS (JSON: host -> "ok", "wrong" (a different host key), "denied" or
"slow" (hangs after connecting);
every call is appended to $FAKESSH_LOG as a JSON line. POSIX only."""
every call is appended to $FAKESSH_LOG as a JSON line. POSIX only.
With $FAKESSH_PINS set, host keys are checked as OpenSSH does against UserKnownHostsFile under
HostKeyAlias: "wrong" is then just a different key from the others' (a reinstalled headset),
StrictHostKeyChecking=yes refuses a key not saved, accept-new saves a first one, and a saved key
that differs is refused. "evil" is a device with its own key that lets anyone in without
authentication ("none") and asks for the agent: it records in $FAKESSH_DIR what it got
(agent-forwarded, command-ran). Like ssh, the first value of an option counts."""
import base64
import hashlib
import json
import os
import signal
import subprocess
import sys
import time
@@ -17,12 +27,12 @@ hosts = json.loads(os.environ.get("FAKESSH_HOSTS", "{}"))
opts = {}
i = 0
while i < len(args) and args[i].startswith("-"):
if args[i] in ("-o", "-O", "-p", "-l"):
if args[i] in ("-o", "-O", "-p", "-l", "-F", "-W"):
key = args[i]
val = args[i + 1]
if key == "-o":
k, _, v = val.partition("=")
opts[k.lower()] = v
opts.setdefault(k.lower(), v)
else:
opts[key] = val
i += 2
@@ -30,6 +40,12 @@ while i < len(args) and args[i].startswith("-"):
opts[args[i]] = True
i += 1
alias = args[i] if i < len(args) else ""
command = args[i + 1:]
if opts.get("-F") != "none":
# $FAKESSH_CONFIG: what ~/.ssh/config says (JSON, lower-case option -> value), below the
# command line's options, as ssh has it; `-F none` reads no config.
for k, v in json.loads(os.environ.get("FAKESSH_CONFIG", "{}")).items():
opts.setdefault(k, v)
host = opts.get("hostname", alias).replace("%%", "%")
marker = os.path.join(os.environ["FAKESSH_DIR"], "master-" + host.replace("/", "_"))
say = lambda s: (sys.stderr.write(s + "\n"), sys.stderr.flush())
@@ -44,6 +60,23 @@ if opts.get("-O") == "exit":
os.unlink(marker)
sys.exit(0)
if "-W" in opts: # a jump host's ssh, forwarding to the headset: the call is logged, that's all
sys.exit(0)
if opts.get("proxycommand", "none") != "none":
# As OpenSSH: expand %-tokens, then `$SHELL -c "exec <line>"` (here /bin/sh).
line, out, n = opts["proxycommand"], "", 0
while n < len(line):
if line[n] == "%" and n + 1 < len(line):
out += {"%": "%", "h": host, "p": opts.get("port", "22"), "r": opts.get("user", "")}.get(
line[n + 1], line[n:n + 2])
n += 2
else:
out += line[n]
n += 1
if subprocess.run(["/bin/sh", "-c", "exec " + out], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL).returncode:
say("kex_exchange_identification: Connection closed by remote host")
sys.exit(255)
what = hosts.get(host)
if what is None:
say(f"ssh: Could not resolve hostname {host}: nodename nor servname provided, or not known")
@@ -53,6 +86,66 @@ say("debug1: Connection established.")
if what == "slow":
time.sleep(30)
say(f"debug1: Authenticating to {host}:22 as '{opts.get('user', 'tester')}'")
if os.environ.get("FAKESSH_PINS") and "hostkeyalias" in opts and "userknownhostsfile" in opts:
name, path = opts["hostkeyalias"], os.path.expanduser(opts["userknownhostsfile"])
blob = base64.b64encode(b"fake-" + {"wrong": b"other", "evil": b"evil"}.get(what, b"headset") + b"-key").decode()
fp = base64.b64encode(hashlib.sha256(base64.b64decode(blob)).digest()).decode().rstrip("=")
say(f"debug1: Server host key: ssh-ed25519 SHA256:{fp}")
try:
saved = [ln.split() for ln in open(path, errors="replace").read().splitlines() if ln.strip()]
except OSError:
saved = []
saved = [f[2] for f in saved if len(f) >= 3 and name in f[0].split(",")]
strict = opts.get("stricthostkeychecking", "ask")
if saved and blob not in saved:
say("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@")
say("@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @")
say("Host key verification failed.")
sys.exit(255)
if not saved and strict != "accept-new":
say(f"No ED25519 host key is known for {name} and you have requested strict checking.")
say("Host key verification failed.")
sys.exit(255)
if not saved:
try:
with open(path, "a") as f:
f.write(f"{name} ssh-ed25519 {blob}\n")
say(f"Warning: Permanently added '{name}' (ED25519) to the list of known hosts.")
except OSError: # ssh doesn't create the folder: it goes on, saving nothing
say(f"Failed to add the host to the list of known hosts ({path}).")
else:
say(f"debug1: Host '{name}' is known and matches the ED25519 host key.")
say("debug1: Next authentication method: publickey")
if what == "evil":
say(f'Authenticated to {host} ([127.0.0.1]:22) using "none".')
got = os.path.join(os.environ["FAKESSH_DIR"], "evil-")
if opts.get("forwardagent", "no") != "no" and opts.get("identityagent", "") != "none":
open(got + "agent-forwarded", "w").close()
if "-N" not in opts and (command or opts.get("remotecommand", "none") != "none"):
open(got + "command-ran", "w").close()
sys.exit(0)
time.sleep(30) # -N: holds the session open
sys.exit(0)
if opts.get("preferredauthentications") == "none":
say(f"tester@{host}: Permission denied (publickey,password).")
sys.exit(255)
if what == "denied":
say(f"tester@{host}: Permission denied (publickey).")
sys.exit(255)
what = "ok"
say(f'Authenticated to {host} ([127.0.0.1]:22) using "publickey".')
if opts.get("controlmaster") != "yes":
sys.exit(0)
open(marker, "w").close()
def bye(*_):
if os.path.exists(marker):
os.unlink(marker)
sys.exit(0)
signal.signal(signal.SIGTERM, bye)
while True:
time.sleep(0.2)
if not os.path.exists(marker):
sys.exit(0)
say("debug1: Server host key: ssh-ed25519 SHA256:fakefakefakefakefakefakefakefakefakefakefak")
if what == "wrong":
say("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@")
+5
View File
@@ -0,0 +1,5 @@
Hand-made itch.io API responses for tests/test_itch_purchases.py; no real account or
network was used. owned-keys-*.json follow the documented /profile/owned-keys sample
(https://itch.io/docs/api/serverside). uploads.json follows the shape the itch.io app's
client library (github.com/itchio/go-itchio) reads from /games/ID/uploads, with the
`traits` array it handles; `p_android` there is assumed, not documented.
+22
View File
@@ -0,0 +1,22 @@
{
"page": 1,
"per_page": 500,
"owned_keys": [
{"id": 9001, "game_id": 341190, "purchase_id": 3923754, "downloads": 6,
"created_at": "2026-03-10T23:46:36Z", "updated_at": "2026-03-10T23:46:36Z",
"game": {"id": 341190, "title": "Orbit Garden VR", "url": "https://maker.itch.io/orbit-garden",
"cover_url": "https://img.itch.zone/aW1nLzE2OTA3NTIuanBn/315x250%23c/Ihj7Wa.jpg",
"short_text": "Grow plants in zero gravity.", "type": "default", "classification": "game",
"min_price": 1299, "traits": ["p_windows", "p_android", "can_be_bought"],
"created_at": "2025-12-04T16:18:04Z", "published_at": "2026-01-03T17:28:47Z",
"user": {"id": 46531, "username": "maker", "display_name": "Maker Studio",
"url": "https://maker.itch.io"}}},
{"id": 9002, "game_id": 3, "purchase_id": null, "downloads": 0,
"created_at": "2026-02-01T10:00:00Z", "updated_at": "2026-02-01T10:00:00Z",
"game": {"id": 3, "title": "Desk Only", "url": "https://leafo.itch.io/x-moon",
"cover_url": "https://img.itch.zone/aW1hZ2UvMy8xODM3LnBuZw==/315x250%23/y2uYQI.png",
"type": "default", "classification": "game", "min_price": 0,
"traits": ["p_windows", "p_linux", "p_osx"], "published_at": "2013-03-03T23:02:14Z",
"user": {"id": 1, "username": "leafo", "display_name": "leafo", "url": "https://leafo.itch.io"}}}
]
}
+19
View File
@@ -0,0 +1,19 @@
{
"page": 2,
"per_page": 500,
"owned_keys": [
{"id": 8001, "game_id": 341190, "purchase_id": null, "downloads": 1,
"created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z",
"game": {"id": 341190, "title": "Orbit Garden VR", "url": "https://maker.itch.io/orbit-garden",
"type": "default", "classification": "game", "min_price": 1299,
"traits": ["p_windows", "p_android", "can_be_bought"],
"user": {"id": 46531, "username": "maker", "display_name": "Maker Studio"}}},
{"id": 8002, "game_id": 77, "purchase_id": null, "downloads": 2,
"created_at": "2025-05-01T10:00:00Z", "updated_at": "2025-05-01T10:00:00Z",
"game": {"id": 77, "title": "Free Quest Toy", "url": "https://toys.itch.io/quest-toy",
"cover_url": "http://img.example/not-itch.png",
"type": "default", "classification": "game", "min_price": 0,
"traits": ["p_android"], "published_at": "2025-04-01T00:00:00Z",
"user": {"id": 5, "username": "toys"}}}
]
}
+28
View File
@@ -0,0 +1,28 @@
{
"uploads": [
{"id": 501, "game_id": 341190, "filename": "OrbitGarden-win64.zip", "display_name": "Windows",
"size": 734003200, "storage": "hosted", "type": "default", "traits": ["p_windows"],
"created_at": "2026-04-01T10:00:00Z", "updated_at": "2026-04-02T10:00:00Z"},
{"id": 502, "game_id": 341190, "filename": "OrbitGarden-Quest-old.apk", "display_name": "Quest 1.0",
"size": 104857600, "storage": "hosted", "type": "default", "traits": ["p_android"],
"created_at": "2026-01-01T10:00:00Z", "updated_at": "2026-01-05T10:00:00Z"},
{"id": 503, "game_id": 341190, "filename": "OrbitGarden-Quest.apk", "display_name": "Quest 1.2",
"size": 115343360, "storage": "hosted", "type": "default", "traits": ["p_android"],
"created_at": "2026-04-01T10:00:00Z", "updated_at": "2026-04-03T10:00:00Z"},
{"id": 504, "game_id": 341190, "filename": "OrbitGarden-demo.apk", "display_name": "Quest demo",
"size": 52428800, "storage": "hosted", "type": "default", "traits": ["p_android", "demo"],
"created_at": "2026-05-01T10:00:00Z", "updated_at": "2026-05-01T10:00:00Z"},
{"id": 505, "game_id": 341190, "filename": "OrbitGarden-android-bundle.zip", "display_name": "Android (zip)",
"size": 120000000, "storage": "hosted", "type": "default", "traits": ["p_android"],
"created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z"},
{"id": 506, "game_id": 341190, "filename": "Soundtrack.apk", "display_name": "Soundtrack",
"size": 1000, "storage": "hosted", "type": "soundtrack", "traits": [],
"created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z"},
{"id": 507, "game_id": 341190, "filename": "mirror.apk", "display_name": "Mirror",
"size": 1000, "storage": "external", "host": "example.com", "type": "default", "traits": ["p_android"],
"created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z"},
{"id": 508, "game_id": 341190, "filename": "OrbitGarden-mac.dmg", "display_name": "macOS",
"size": 1000, "storage": "hosted", "type": "default", "traits": ["p_osx"],
"created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z"}
]
}
+12 -1
View File
@@ -19,7 +19,7 @@ function grab(name) {
fail(`unbalanced ${name}`);
}
const names = ["esc", "storeName", "storeHue", "storeInitial", "storeText", "storeOpenSource", "sourcePage", "storeImage",
"storeArt", "storeIcon", "sourceMark", "currentOffer", "detailBadges", "renderAppDetail", "renderAppDetailBody",
"storeArt", "storeIcon", "sourceMark", "storePrice", "currentOffer", "detailBadges", "renderAppDetail", "renderAppDetailBody",
"updateInstall", "openStoreApp", "loadOfferDetails"];
const code = names.map(grab).join("");
@@ -54,6 +54,17 @@ const offer = { source: "fdroid", id: "org.example.brush", source_name: "F-Droid
if (html.includes('role="alert"')) fail("a good listing showed the error state");
}
// A game the user owns on itch.io says so instead of "Free".
{
const p = page();
p.sourceState.detail = { app: { name: "Orbit Garden", offers: [{ ...offer, source: "itch-owned", source_name: "itch.io (my purchases)",
owned: true, owned_label: "Owned on itch.io", free: false }] }, choice: 0 };
if (p.renderAppDetail() !== true) fail("an owned listing reported a render failure");
const html = p.els.detailBody.innerHTML;
if (!html.includes('<div class="detail-price">Owned on itch.io</div>')) fail("owned listing has no Owned on itch.io label");
if (html.includes("No purchase needed")) fail("owned listing claims no purchase is needed");
}
// A listing that can't render (no offer to show) gets a clear message, not an empty dialog.
{
const p = page();
+31 -1
View File
@@ -1,12 +1,20 @@
"""Imported first by every test module: nothing a test does reaches this person's
app data, their telemetry, or the shared compatibility database.
app data, their ~/.ssh, their telemetry, or the shared compatibility database.
Must run before any ui module is imported, since those read these at import time.
The variables are set in os.environ, so servers and scripts the tests start
inherit them too (they pass {**os.environ, ...}).
"""
import atexit
import os
import shutil
import tempfile
from pathlib import Path
# The environment and real ~/.ssh from before the changes below: test_sandbox
# reruns tests from that environment and checks ~/.ssh is left alone.
ORIGINAL_ENV = dict(os.environ)
REAL_SSH_DIR = Path.home() / ".ssh"
_dir = tempfile.mkdtemp(prefix="frame-control-tests-")
atexit.register(shutil.rmtree, _dir, ignore_errors=True)
@@ -14,3 +22,25 @@ os.environ["FRAME_CONTROL_DATA_DIR"] = _dir
os.environ["FRAME_CONTROL_TELEMETRY"] = "0"
# A maintainer's machine holds the database key; send anything that slips through nowhere.
os.environ["FRAME_COMPAT_DB_URL"] = "http://127.0.0.1:9"
# A throwaway home, so Path.home() / "~" land here: ~/.ssh (config, host-key pins,
# its lock file) and the other per-user folders some modules use. A server started
# without FRAME_CONTROL_SSH_DIR used to import the real ~/.ssh/config and seed a pin
# into the real ~/.ssh/frame-control-hosts on every run.
#
# tests/e2e must not import this (directly or through a test_* module): they run in
# a throwaway container whose real ~/.ssh points `ssh frame` at the fake Frame, and
# OpenSSH finds ~/.ssh from the password database, not $HOME. Shared helpers they
# need live in sandbox-free modules such as apk_fixtures.py.
HOME = os.path.join(_dir, "home")
SSH_DIR = os.path.join(HOME, ".ssh")
os.makedirs(SSH_DIR, mode=0o700)
os.environ["HOME"] = HOME
os.environ["USERPROFILE"] = HOME # Path.home() on Windows
os.environ["FRAME_CONTROL_SSH_DIR"] = SSH_DIR # frame_devices.ssh_dir(), frame_connect.SSH_DIR
# The per-user folders that don't follow HOME when these are set: frame_host.cache_dir()
# (servers warm indexes there and sweep its staging folders) and data_dir()'s fallbacks.
for _name, _sub in (("LOCALAPPDATA", "AppData/Local"), ("APPDATA", "AppData/Roaming"),
("XDG_CACHE_HOME", ".cache"), ("XDG_DATA_HOME", ".local/share"),
("XDG_STATE_HOME", ".local/state"), ("XDG_CONFIG_HOME", ".config")):
os.environ[_name] = os.path.join(HOME, *_sub.split("/"))
+1
View File
@@ -1,4 +1,5 @@
"""MCP protocol, exact-action approvals and explicit assistant data sharing."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import io
import json
import os
+1
View File
@@ -1,3 +1,4 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import http.client
import json
from pathlib import Path
+1
View File
@@ -1,3 +1,4 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import hashlib, io, json, os, sys, tempfile, time, unittest, urllib.error, urllib.request, zipfile
from pathlib import Path
from unittest.mock import patch
+1
View File
@@ -1,3 +1,4 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import http.client
import json
from pathlib import Path
+1
View File
@@ -1,4 +1,5 @@
"""Fake-Frame session clock/actions plus real helper serialization and sensor probes."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import os
import shutil
import json
+1
View File
@@ -1,4 +1,5 @@
"""Run the actual shared page's comfort renderer against a minimal DOM/bridge."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import pathlib
import shutil
import subprocess
+24
View File
@@ -264,6 +264,30 @@ class Pins(Base):
self.assertFalse(fd.pinned("d1"))
self.assertFalse(fd.forget_pin("d1"))
def test_a_trusted_key_replaces_the_saved_one_and_fingerprints_match_ssh_keygen(self):
key = self.dir / "hostkey"
frame_host.run_ssh(["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(key)], capture_output=True,
stdin=subprocess.DEVNULL, check=True, timeout=20)
kind, blob = key.with_suffix(".pub").read_text().split()[:2]
want = frame_host.run_ssh(["ssh-keygen", "-l", "-E", "sha256", "-f", str(key.with_suffix(".pub"))],
capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=10).stdout.split()[1]
self.assertEqual(fd.fingerprint(blob), want)
self.assertIsNone(fd.fingerprint("not base64!"))
(self.ssh / "known_hosts").write_text(f"frame.local {KEY}\n")
fd.seed_pin("d1", ["frame.local"])
old = fd.pinned_keys("d1")
fd.pin_key("d1", kind, blob)
self.assertEqual(fd.known_hosts("d1").read_text(), f"frame-control-d1 {kind} {blob}\n") # old key gone
self.assertEqual(fd.pinned_keys("d1"), [(kind, want)])
self.assertNotEqual(old, fd.pinned_keys("d1"))
self.assertTrue(fd.pinned("d1"))
for bad in (("ssh-ed25519\nHost *", blob), ("ssh-ed25519", "AAAA AAAA"), ("evil", blob)):
with self.assertRaises(fd.DeviceError):
fd.pin_key("d1", *bad)
# Only plain entries for this headset's alias count.
self.assertIsNone(fd.key_entry(f"frame-control-d2 {kind} {blob}", "frame-control-d1"))
self.assertIsNone(fd.key_entry(f"# frame-control-d1 {kind} {blob}", "frame-control-d1"))
def test_each_headset_has_its_own_file(self):
(self.ssh / "known_hosts").write_text(f"a.local {KEY}\nb.local {KEY}\n")
fd.seed_pin("da", ["a.local"])
+1
View File
@@ -1,4 +1,5 @@
"""Offline authenticated repository fixtures; no tests contact a server."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import io
import json
import os
+1
View File
@@ -1,3 +1,4 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import io
import json
import os
+1
View File
@@ -1,4 +1,5 @@
"""Offline artwork, Steam API and launcher supervision regressions."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import importlib.util
import json
import os
+1 -76
View File
@@ -11,82 +11,7 @@ import zipfile
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ui'))
import frame_apk # noqa: E402
def pool(strings, utf8=False):
"""A ResStringPool chunk."""
data, offsets = b'', []
for s in strings:
offsets.append(len(data))
if utf8:
b = s.encode()
data += bytes([len(s), len(b)]) + b + b'\0'
else:
data += struct.pack('<H', len(s)) + s.encode('utf-16-le') + b'\0\0'
data += b'\0' * (-len(data) % 4)
start = 28 + 4 * len(strings)
body = struct.pack(f'<{len(strings)}I', *offsets) + data
return struct.pack('<HHIIIIII', 1, 28, 28 + len(body), len(strings), 0, 0x100 if utf8 else 0, start, 0) + body
def manifest(package, label_ref, version_ref, min_sdk, package_raw=True, foreign_label=False):
"""<manifest package versionName><uses-sdk minSdkVersion/><application label icon/></manifest>.
package_raw=False drops the package's raw string (as some repackers do);
foreign_label adds a non-android `label` attribute after android:label.
"""
strings = ['label', 'icon', 'versionName', 'minSdkVersion', 'package', 'manifest', 'uses-sdk',
'application', package, 'junk', 'label', 'versionCode'] # the second 'label' has no android id
resmap = struct.pack('<4I', 0x01010001, 0x01010002, 0x0101021c, 0x0101020c)
resmap = struct.pack('<HHI', 0x0180, 8, 8 + len(resmap)) + resmap
def element(name, attrs):
body = struct.pack('<IIHHHHHH', 0xffffffff, name, 20, 20, len(attrs), 0, 0, 0)
for aname, raw, dtype, value in attrs:
body += struct.pack('<IIIHBBI', 0xffffffff, aname, raw, 8, 0, dtype, value)
return struct.pack('<HHIII', 0x0102, 16, 16 + len(body), 1, 0xffffffff) + body
none = 0xffffffff
chunks = (pool(strings) + resmap
+ element(5, [(4, 8 if package_raw else none, frame_apk.T_STRING, 8),
(2, none, frame_apk.T_REF, version_ref),
(11, none, frame_apk.T_INT_DEC, 210)])
+ element(6, [(3, none, frame_apk.T_INT_DEC, min_sdk)])
+ element(7, [(0, none, frame_apk.T_REF, label_ref), (1, none, frame_apk.T_REF, 0x7f020000)]
+ ([(10, 9, frame_apk.T_STRING, 9)] if foreign_label else [])))
return struct.pack('<HHI', 3, 8, 8 + len(chunks)) + chunks
def resources(values):
"""resources.arsc with package 0x7f; values: {(type id, entry, language, density): global string index}."""
strings = ['French label', 'App label', '2.1', 'res/icon_lo.png', 'res/icon_hi.png', 'res/icon.xml']
pkg_body = b''
for (tid, lang, density), entries in values.items():
cfg = struct.pack('<I4x2s4xH', 64, lang.encode().ljust(2, b'\0'), density).ljust(64, b'\0')
count = max(entries) + 1
offsets, data = [], b''
for i in range(count):
if i in entries:
offsets.append(len(data))
data += struct.pack('<HHI', 8, 0, 0) + struct.pack('<HBBI', 8, 0, frame_apk.T_STRING, entries[i])
else:
offsets.append(0xffffffff)
header = 20 + 64
estart = header + 4 * count
body = struct.pack(f'<{count}I', *offsets) + data
pkg_body += struct.pack('<HHIBBHII', 0x0201, header, header + len(body), tid, 0, 0, count, estart) + cfg + body
pkg_header = struct.pack('<HHII', 0x0200, 288, 288 + len(pkg_body), 0x7f).ljust(288, b'\0')
pkg = pkg_header + pkg_body
table = pool(strings, utf8=True) + pkg
return struct.pack('<HHII', 2, 12, 12 + len(table), 1) + table
def apk(files):
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w') as z:
for name, data in files.items():
z.writestr(name, data)
return buf.getvalue()
from apk_fixtures import apk, manifest, pool, resources # noqa: E402,F401
class ApkInfo(unittest.TestCase):
+2 -1
View File
@@ -1,4 +1,5 @@
"""Local-only VR manifest, raw ZIP and independent signature checks."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import io
import os
from pathlib import Path
@@ -15,7 +16,7 @@ import frame_apk
import frame_apk_vr as vr
import frame_apk_sign as signing
import frame_android
from test_frame_apk import pool
from apk_fixtures import pool
DEFAULT = 'android.intent.category.DEFAULT'
+1
View File
@@ -6,6 +6,7 @@ identity line, then starts TLS as the *client*.
Run: python3 -m unittest discover -s tests
"""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import json
import os
import shutil
+862
View File
@@ -0,0 +1,862 @@
"""My itch.io purchases: the account key, owned games and their Android downloads.
Offline: a fake HTTPS layer under urllib's real opener (so redirects and header
handling are the real ones) serves the recorded JSON in fixtures/itch_account.
"""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import contextlib, email.message, http.client, io, ipaddress, json, os, socket, sys, tempfile, threading, time
import unittest
import urllib.error, urllib.parse, urllib.request, zipfile
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
from apk_sources import SourceError, SourceLimited, _itch_account as account, _web, itch, search
import server
FIX = Path(__file__).parent / 'fixtures' / 'itch_account'
KEY = 'TESTKEYabcdefghijklmnopqrstuvwxyz0123456789'
OTHER = 'OTHERKEY0123456789abcdefghijklmnopqrstuv'
def apk_bytes():
out = io.BytesIO()
with zipfile.ZipFile(out, 'w') as z:
z.writestr('AndroidManifest.xml', b'\x03\x00\x08\x00')
z.writestr('classes.dex', b'dex\n035\x00')
return out.getvalue()
class Reply(io.BytesIO):
"""Just enough of http.client.HTTPResponse for urllib's handlers."""
def __init__(self, url, code=200, body=b'', headers=None):
super().__init__(body if isinstance(body, bytes) else json.dumps(body).encode())
self.url, self.code, self.status, self.msg, self.reason = url, code, code, 'fake', 'fake'
self.headers = email.message.Message()
for k, v in (headers or {}).items():
self.headers[k] = v
def info(self):
return self.headers
def geturl(self):
return self.url
def getcode(self):
return self.code
def getheader(self, name, default=None):
return self.headers.get(name, default)
class FakeConnection:
"""_web._connect's http.client connection, answered from the same routes."""
def __init__(self, web, host):
self.web, self.host, self.reply = web, host, None
def request(self, method, path, headers=None):
req = urllib.request.Request('https://' + self.host + path, headers=headers or {})
self.reply = self.web(None, req)
def getresponse(self):
return self.reply
def close(self):
pass
# What the fake resolver answers: the storage host is public; the rest are traps.
ADDRESSES = {'storage.itch-cdn.example': '151.101.1.1', 'other-cdn.example': '151.101.2.2',
'localhost.': '127.0.0.1', 'frame.local': '192.168.1.20', '127.0.0x0.1': '127.0.0.1',
'intranet.example': '10.0.0.5', 'mapped.example': '::ffff:127.0.0.1', 'cgnat.example': '100.64.0.1',
'mixed.example': ('151.101.1.1', '10.0.0.5'), 'dual.example': ('2a04:4e42::81', '151.101.1.1'),
'nat64.example': '64:ff9b:1::a00:1', 'sitelocal.example': 'fec0::1',
'many.example': tuple('151.101.%d.1' % n for n in range(1, 9))}
REAL_CONNECT = _web._connect
class FakeItch:
"""Routes https requests by host and path; records every request with its headers."""
def __init__(self):
self.calls, self.routes = [], {}
def route(self, host_path, reply):
self.routes[host_path] = reply
def __call__(self, handler, req):
url = urllib.parse.urlsplit(req.full_url)
headers = {k.lower(): v for k, v in req.header_items()}
self.calls.append((req.full_url, headers))
reply = self.routes.get(url.hostname + url.path)
if reply is None:
return Reply(req.full_url, 404, {'errors': ['not found']})
if callable(reply):
reply = reply(req, urllib.parse.parse_qs(url.query))
if isinstance(reply, Exception):
raise reply
if isinstance(reply, Reply):
return reply
return Reply(req.full_url, 200, reply)
def urls(self, host_path):
return [u for u, _ in self.calls if urllib.parse.urlsplit(u).hostname + urllib.parse.urlsplit(u).path == host_path]
def owned_pages(req, query):
page = query['page'][0]
return {'1': json.loads((FIX / 'owned-keys-1.json').read_text()),
'2': json.loads((FIX / 'owned-keys-2.json').read_text())}.get(page, {'page': int(page), 'owned_keys': []})
class ItchPurchases(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
root = Path(self.tmp.name)
for target, value in [(account.frame_host, 'data_dir'), (search.frame_host, 'data_dir')]:
p = patch.object(target, value, lambda *parts: root.joinpath('data', *parts))
p.start()
self.addCleanup(p.stop)
(root / 'cache').mkdir()
for p in (patch.object(_web, 'cache', return_value=str(root / 'cache')),
patch.object(_web.frame_host, 'cache_dir', lambda *parts: root.joinpath('cache', *parts))):
p.start()
self.addCleanup(p.stop)
self.web = FakeItch()
self.web.route('api.itch.io/profile', {'user': {'username': 'player', 'id': 1}})
self.web.route('api.itch.io/profile/owned-keys', owned_pages)
self.web.route('api.itch.io/games/341190/uploads', json.loads((FIX / 'uploads.json').read_text()))
p = patch('urllib.request.HTTPSHandler.https_open', autospec=True, side_effect=self.web)
p.start()
self.addCleanup(p.stop)
p = patch('urllib.request.HTTPHandler.http_open', side_effect=AssertionError('plain HTTP in test'))
p.start()
self.addCleanup(p.stop)
real_lookup = _web.socket.getaddrinfo
def lookup(host, port, *args, **kwargs):
if host not in ADDRESSES:
try:
ipaddress.ip_address(host) # IP literals (and the in-process test server) need no DNS
return real_lookup(host, port, *args, **kwargs)
except ValueError:
pass
if host == 'localhost':
return real_lookup(host, port, *args, **kwargs)
raise _web.socket.gaierror('unknown test host ' + host)
found = ADDRESSES[host]
return [(10 if ':' in a else 2, 1, 6, '', (a, port)) for a in ([found] if isinstance(found, str) else found)]
self.connected, self.unreachable = [], set()
def connect(host, address, timeout, deadline):
self.connected.append((host, address))
if address[0] in self.unreachable:
raise OSError('Network is unreachable')
self.assertLessEqual(deadline - time.monotonic(), _web.CONNECT_BUDGET)
return FakeConnection(self.web, host)
for p in (patch.object(_web.socket, 'getaddrinfo', lookup), patch.object(_web, '_connect', connect)):
p.start()
self.addCleanup(p.stop)
account._forget()
self.addCleanup(account._forget)
_web._limited.clear()
self.addCleanup(_web._limited.clear)
search._status.clear()
def owned_source(self):
return next(s for s in itch.sources() if s['id'] == itch.OWNED)
def connect(self):
self.assertTrue(account.connect(KEY)['itch']['connected'])
# -- the key ---------------------------------------------------------------
def test_connect_checks_the_key_then_saves_it_0600(self):
result = account.connect(' ' + KEY + '\n')
self.assertEqual(result['message'], 'Connected to itch.io as player')
self.assertEqual(result['itch'], {'connected': True, 'username': 'player'})
self.assertNotIn(KEY, json.dumps(result))
url, headers = self.web.calls[0]
self.assertEqual(url, 'https://api.itch.io/profile')
self.assertEqual(headers['authorization'], 'Bearer ' + KEY)
self.assertEqual(json.loads(account.path().read_text())['key'], KEY)
if os.name == 'posix':
self.assertEqual(os.stat(account.path()).st_mode & 0o777, 0o600)
self.assertEqual([p.name for p in account.path().parent.iterdir()], ['itch-account.json']) # no .tmp left
self.assertNotIn(KEY, json.dumps(account.status()))
def test_reconnecting_rewrites_the_file_0600_even_if_it_was_wider(self):
self.connect()
os.chmod(account.path(), 0o644)
account.connect(OTHER)
self.assertEqual(json.loads(account.path().read_text())['key'], OTHER)
if os.name == 'posix':
self.assertEqual(os.stat(account.path()).st_mode & 0o777, 0o600)
def test_bad_or_revoked_key_stays_disconnected(self):
for reply in (Reply('https://api.itch.io/profile', 401, {'errors': ['invalid key']}),
Reply('https://api.itch.io/profile', 403, b'Forbidden'),
{'errors': ['invalid key']}):
self.web.route('api.itch.io/profile', reply)
with self.assertRaisesRegex(account.BadKey, 'did not accept the API key') as caught:
account.connect(KEY)
self.assertNotIn(KEY, str(caught.exception))
self.assertFalse(account.path().exists())
self.assertEqual(account.status(), {'connected': False, 'username': None})
self.assertEqual([s['id'] for s in itch.sources()], ['itch', 'itch-quest'])
with self.assertRaisesRegex(account.BadKey, 'does not look like'):
account.connect('short')
with self.assertRaisesRegex(account.BadKey, 'does not look like'):
account.connect(KEY + '\r\nX-Injected: 1')
self.assertEqual(len(self.web.calls), 3) # malformed keys never reach itch.io
def test_a_refused_request_with_a_working_key_keeps_the_key(self):
self.connect()
self.web.route('api.itch.io/games/341190/uploads', Reply('https://api.itch.io/games/341190/uploads', 403, b''))
with self.assertRaisesRegex(SourceError, r'refused this request \(HTTP 403\)') as caught:
itch.details(self.owned_source(), '341190')
self.assertNotIsInstance(caught.exception, account.BadKey)
self.assertTrue(account.connected())
def test_a_key_revoked_later_is_removed_with_a_clear_message(self):
self.connect()
self.web.route('api.itch.io/profile/owned-keys', Reply('https://api.itch.io/profile/owned-keys', 401, b''))
self.web.route('api.itch.io/profile', Reply('https://api.itch.io/profile', 401, b''))
with self.assertRaisesRegex(account.BadKey, 'no longer accepts the saved API key'):
itch.search(self.owned_source(), '')
self.assertFalse(account.path().exists())
self.assertNotIn(itch.OWNED, [s['id'] for s in itch.sources()])
def test_disconnect_deletes_the_key_file(self):
self.connect()
self.assertIn(itch.OWNED, [s['id'] for s in itch.sources()])
result = account.disconnect()
self.assertFalse(account.path().exists())
self.assertEqual(result['itch'], {'connected': False, 'username': None})
self.assertNotIn(itch.OWNED, [s['id'] for s in itch.sources()])
with self.assertRaisesRegex(SourceError, 'Connect your itch.io account first'):
account.owned_keys()
account.disconnect() # already gone: fine
# -- owned games -------------------------------------------------------------
def test_owned_keys_follow_pages_until_an_empty_one(self):
self.connect()
keys = account.owned_keys()
self.assertEqual([k['id'] for k in keys], [9001, 9002, 8001, 8002])
pages = self.web.urls('api.itch.io/profile/owned-keys')
self.assertEqual([urllib.parse.parse_qs(urllib.parse.urlsplit(u).query) for u in pages],
[{'page': [str(n)], 'per_page': ['500']} for n in (1, 2, 3)])
self.assertTrue(all('Bearer ' + KEY == h['authorization'] for _, h in self.web.calls))
self.assertTrue(all(KEY not in u for u, _ in self.web.calls))
account.owned_keys() # cached in memory, not on disk
self.assertEqual(len(self.web.urls('api.itch.io/profile/owned-keys')), 3)
self.assertFalse(any(KEY in p.read_text(errors='ignore') or b'Orbit' in p.read_bytes()
for p in Path(_web.cache()).iterdir()))
def test_page_cap_stops_a_server_that_never_ends(self):
self.connect()
self.web.route('api.itch.io/profile/owned-keys', lambda req, q: {'owned_keys': [{'id': int(q['page'][0])}]})
with patch.object(account, 'PAGES', 3):
self.assertEqual(len(account.owned_keys()), 3)
def test_search_lists_owned_android_games_labelled(self):
self.connect()
entries = itch.search(self.owned_source(), '')
self.assertEqual([(e['id'], e['name']) for e in entries], [('341190', 'Orbit Garden VR'), ('77', 'Free Quest Toy')])
orbit, toy = entries
self.assertEqual(orbit['download_key_id'], 9001) # the newest key for a game owned twice
self.assertTrue(orbit['owned'])
self.assertEqual(orbit['owned_label'], 'Owned on itch.io')
self.assertFalse(orbit['free'])
self.assertTrue(toy['free'])
self.assertIsNone(toy['icon']) # not an itch.io image host
self.assertEqual(orbit['page'], 'https://maker.itch.io/orbit-garden')
self.assertEqual(itch.search(self.owned_source(), 'zero gravity')[0]['id'], '341190')
# Through the store's search, owned (paid) games pass the free-only filter.
result = search.search('orbit', source=itch.OWNED)
self.assertEqual(result['sources'][0]['status'], 'ok')
offer = result['apps'][0]['offers'][0]
self.assertEqual((offer['source'], offer['source_name'], offer['owned_label']),
(itch.OWNED, 'itch.io (my purchases)', 'Owned on itch.io'))
self.assertNotIn(KEY, json.dumps(result))
def test_owned_games_show_under_either_vr_choice(self):
# itch.io doesn't say which purchases are VR, so neither filter hides them; other
# games of unknown kind still count as flat.
self.connect()
both = ['Free Quest Toy', 'Orbit Garden VR']
for vr in (True, False):
names = sorted(a['name'] for a in search.search('', vr=vr, source=itch.OWNED)['apps'])
self.assertEqual(names, both)
flat = {'source': 'x', 'id': '1', 'name': 'Flat', 'summary': '', 'vr': None}
self.assertEqual(search.group([flat], vr=True), [])
self.assertEqual(len(search.group([flat], vr=False)), 1)
self.assertEqual(search.group([dict(flat, owned=True, vr=False)], vr=True), [])
def test_only_the_account_source_may_mark_paid_games_owned(self):
entry = {'free': False, 'owned': True}
self.assertFalse(search.obtainable({'id': 'fdroid'}, entry))
self.assertTrue(search.obtainable({'id': itch.OWNED, 'owned': True}, entry))
self.assertTrue(search.obtainable({'id': 'fdroid'}, {'free': True}))
def test_android_upload_selection(self):
uploads = json.loads((FIX / 'uploads.json').read_text())['uploads']
# .apk files before other Android files, full games before demos, newest first; never
# soundtracks, externally hosted files or other platforms.
self.assertEqual([u['id'] for u in account.android_uploads(uploads)], [503, 502, 504, 505])
self.assertTrue(account.is_android({'filename': 'Game.APK'}))
self.assertTrue(account.is_android({'filename': 'game.zip', 'p_android': True}))
self.assertTrue(account.is_android({'filename': 'game.zip', 'platforms': {'android': 'all'}}))
self.assertFalse(account.is_android({'filename': 'game.zip', 'traits': ['p_windows']}))
self.assertEqual(account.android_uploads({'not': 'a list'}), [])
def test_details_offer_the_android_uploads(self):
self.connect()
entry = itch.details(self.owned_source(), '341190')
self.assertTrue(entry['downloadable'])
self.assertEqual([v['upload_id'] for v in entry['versions']], [503, 502, 504, 505])
self.assertEqual((entry['version'], entry['size'], entry['updated']), ('Quest 1.2', 115343360, '2026-04-03'))
self.assertTrue(entry['versions'][2]['demo'])
url = self.web.urls('api.itch.io/games/341190/uploads')[0]
self.assertEqual(url, 'https://api.itch.io/games/341190/uploads?download_key_id=9001')
with self.assertRaisesRegex(SourceError, 'not among'):
itch.details(self.owned_source(), '3') # owned, but not for Android
self.web.route('api.itch.io/games/77/uploads', {'uploads': []})
self.assertFalse(itch.details(self.owned_source(), '77')['downloadable'])
with self.assertRaisesRegex(SourceError, 'no Android download'):
itch.download(self.owned_source(), '77')
def test_download_url_building(self):
self.assertEqual(account.download_url(503, 9001),
'https://api.itch.io/uploads/503/download?download_key_id=9001')
for bad in [(0, 1), (1, None), ('5', 1), (True, 1), (-2, 1)]:
with self.assertRaises(SourceError):
account.download_url(*bad)
# -- downloads -----------------------------------------------------------------
def storage(self, location='https://storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk?sig=abc'):
self.web.route('api.itch.io/uploads/503/download',
Reply('https://api.itch.io/uploads/503/download', 302, b'', {'Location': location}))
self.web.route('storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk', Reply(location, 200, apk_bytes()))
def test_download_sends_the_key_to_itch_only_and_strips_it_on_redirect(self):
self.connect()
self.storage()
result = itch.download(self.owned_source(), '341190')
self.assertTrue(os.path.isfile(result['apk']))
self.assertFalse(result['verified'])
api, cdn = self.web.calls[-2:]
self.assertEqual(api[0], 'https://api.itch.io/uploads/503/download?download_key_id=9001')
self.assertEqual(api[1]['authorization'], 'Bearer ' + KEY)
self.assertTrue(cdn[0].startswith('https://storage.itch-cdn.example/'))
self.assertNotIn('authorization', cdn[1])
# Connected to the address that was checked, not to a fresh lookup.
self.assertEqual(self.connected, [('storage.itch-cdn.example', ('151.101.1.1', 443))])
self.assertTrue(all(KEY not in u for u, _ in self.web.calls))
with self.assertRaisesRegex(SourceError, 'version codes'):
itch.download(self.owned_source(), '341190', version_code=3)
def test_download_also_accepts_a_json_url_or_the_file_itself(self):
self.connect()
self.storage()
location = 'https://storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk?sig=abc'
self.web.route('api.itch.io/uploads/503/download',
Reply('https://api.itch.io/uploads/503/download', 200, {'url': location},
{'Content-Type': 'application/json'}))
self.assertTrue(os.path.isfile(itch.download(self.owned_source(), '341190')['apk']))
self.assertNotIn('authorization', self.web.calls[-1][1])
self.web.route('api.itch.io/uploads/503/download',
Reply('https://api.itch.io/uploads/503/download', 200, apk_bytes(),
{'Content-Type': 'application/octet-stream'}))
self.connected.clear()
self.assertTrue(os.path.isfile(itch.download(self.owned_source(), '341190')['apk']))
self.assertEqual(self.connected, []) # api.itch.io served it; no storage host involved
self.web.route('api.itch.io/uploads/503/download',
Reply('https://api.itch.io/uploads/503/download', 200, {'nothing': 1},
{'Content-Type': 'application/json'}))
with self.assertRaisesRegex(SourceError, 'did not say where'):
itch.download(self.owned_source(), '341190')
def test_storage_redirects_are_checked_on_every_hop(self):
self.connect()
self.storage('https://other-cdn.example/hop')
self.web.route('other-cdn.example/hop', Reply('https://other-cdn.example/hop', 302, b'',
{'Location': 'https://storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk'}))
self.assertTrue(os.path.isfile(itch.download(self.owned_source(), '341190')['apk']))
self.assertEqual([h for h, _ in self.connected], ['other-cdn.example', 'storage.itch-cdn.example'])
self.web.route('other-cdn.example/hop', Reply('https://other-cdn.example/hop', 302, b'',
{'Location': 'https://intranet.example/a.apk'}))
with self.assertRaisesRegex(SourceError, 'private network address'):
itch.download(self.owned_source(), '341190')
def test_redirects_to_plain_http_or_private_addresses_are_refused(self):
self.connect()
for location in ('http://storage.itch-cdn.example/a.apk', 'https://user:pw@cdn.example/a.apk',
'https://cdn.example:8443/a.apk'):
self.storage(location)
with self.assertRaisesRegex(SourceError, 'unexpected download URL'):
itch.download(self.owned_source(), '341190')
for location in ('https://localhost./a.apk', 'https://frame.local/a.apk', 'https://127.0.0x0.1/a.apk',
'https://intranet.example/a.apk', 'https://mapped.example/a.apk',
'https://cgnat.example/a.apk', 'https://mixed.example/a.apk',
'https://127.0.0.1/a.apk', 'https://[::1]/a.apk', 'https://169.254.169.254/a.apk',
'https://nat64.example/a.apk', 'https://[4000::1]/a.apk', 'https://[64:ff9b:1::a00:1]/a.apk',
'https://[fec0::1]/a.apk', 'https://sitelocal.example/a.apk'):
self.storage(location)
with self.assertRaisesRegex(SourceError, 'private network address'):
itch.download(self.owned_source(), '341190')
self.assertEqual(self.connected, [])
self.assertEqual({urllib.parse.urlsplit(u).hostname for u, _ in self.web.calls}, {'api.itch.io'})
def test_public_address_rules(self):
for ip in ('151.101.1.1', '8.8.8.8', '2a04:4e42::81', '2001:4860:4860::8888'):
self.assertTrue(_web.public_address(ip), ip)
for ip in ('127.0.0.1', '10.1.2.3', '172.16.0.1', '192.168.0.1', '169.254.169.254', '100.64.0.1',
'0.0.0.0', '224.0.0.1', '255.255.255.255', '240.0.0.1', '::1', '::', 'fe80::1%en0', 'fc00::1',
'ff02::1', 'not an ip',
'192.0.0.8', '192.0.0.11', '192.0.0.255', '192.0.0.170', '192.0.0.9', '192.0.0.1', # 3.9: global
'192.0.2.1', '192.88.99.1', '198.18.0.1', '198.19.255.255', '198.51.100.7', '203.0.113.9',
'0.1.2.3', '10.255.255.255', '172.31.0.1', '239.1.1.1', '254.0.0.1',
'4000::1', '100::1', 'f000::1', # reserved: Python 3.9's is_global says True for 4000::1
'::ffff:10.0.0.1', '::ffff:127.0.0.1', '::ffff:8.8.8.8', # IPv4-mapped
'::a00:1', '::8.8.8.8', # IPv4-compatible
'64:ff9b::a00:1', '64:ff9b::808:808', # NAT64 well-known prefix
'64:ff9b:1::a00:1', '64:ff9b:1::808:808', # NAT64 local-use: 3.9's is_global says True
'2002:a00:1::1', '2002:808:808::1', # 6to4
'2001:0:4136:e378:8000:63bf:3fff:fdd2', # Teredo
'fec0::1', 'feff::1', # site-local: Python 3.9's is_global says True
'fc00::1', 'fd12:3456::1', 'fe80::1', 'ff00::1', 'ff0e::1', '8000::1', 'c000::1',
'2001:db8::1', '3fff::1', # documentation
'2001:10::1', '2001:20::1', '2001:2::1'): # ORCHID, ORCHIDv2, benchmarking
self.assertFalse(_web.public_address(ip), ip)
self.assertEqual(_web.resolve_public('storage.itch-cdn.example'), [('151.101.1.1', 443)])
self.assertEqual(_web.resolve_public('dual.example'), [('2a04:4e42::81', 443), ('151.101.1.1', 443)])
def test_a_storage_url_carrying_the_key_is_refused(self):
self.connect()
for location in ('https://storage.itch-cdn.example/a.apk?api_key=' + KEY,
'https://storage.itch-cdn.example/a.apk?API_KEY=someone-elses',
'https://storage.itch-cdn.example/a.apk?apikey=x',
'https://storage.itch-cdn.example/' + KEY + '/a.apk',
'https://storage.itch-cdn.example/a.apk#' + KEY[:10],
'https://storage.itch-cdn.example/a.apk?x=' + urllib.parse.quote('%' + KEY),
'https://' + KEY.lower() + '.itch-cdn.example/a.apk'):
self.storage(location)
with self.assertRaisesRegex(SourceError, 'carrying your API key; refused') as caught:
itch.download(self.owned_source(), '341190')
self.assertNotIn(KEY.lower(), str(caught.exception).lower())
self.web.route('api.itch.io/uploads/503/download',
Reply('https://api.itch.io/uploads/503/download', 200,
{'url': 'https://storage.itch-cdn.example/a.apk?api_key=' + KEY},
{'Content-Type': 'application/json'}))
with self.assertRaisesRegex(SourceError, 'carrying your API key; refused'):
itch.download(self.owned_source(), '341190')
self.assertEqual(self.connected, [])
# A storage URL's own signature or token parameters are fine.
self.storage('https://storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk?token=abc&Signature=x')
self.assertTrue(os.path.isfile(itch.download(self.owned_source(), '341190')['apk']))
def test_a_later_redirect_carrying_the_key_is_refused_before_connecting(self):
self.connect()
bad = 'https://storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk?api_key=' + KEY
for hops in (['https://other-cdn.example/hop1'], # 2nd hop carries the key
['https://other-cdn.example/hop1', 'https://other-cdn.example/hop2']): # 3rd hop
self.connected.clear()
self.storage(hops[0])
for here, there in zip(hops, hops[1:] + [bad]):
path = urllib.parse.urlsplit(here).path
self.web.route('other-cdn.example' + path, Reply(here, 302, b'', {'Location': there}))
with self.assertRaisesRegex(SourceError, 'carrying your API key; refused') as caught:
itch.download(self.owned_source(), '341190')
self.assertNotIn(KEY, str(caught.exception))
self.assertEqual([h for h, _ in self.connected], ['other-cdn.example'] * len(hops))
self.assertFalse(any(KEY in u for u, _ in self.web.calls))
# The check runs on every hop before the name is even looked up.
seen = []
with patch.object(_web, 'resolve_public', side_effect=AssertionError('looked up')):
with self.assertRaisesRegex(SourceError, 'no thanks'):
_web.public_apk('https://storage.itch-cdn.example/a.apk', 'itch.io',
check=lambda u: seen.append(u) or (_ for _ in ()).throw(SourceError('no thanks')))
self.assertEqual(seen, ['https://storage.itch-cdn.example/a.apk'])
def test_a_partial_match_before_the_key_cannot_hide_its_start(self):
key = 'abcdefghABCDEFGHi1234567890'
for text in (key[:8].upper() + key.upper(), key[:9] + key, 'xx' + key[:8] + key + 'yy'):
out = account._clean(text, key)
self.assertNotIn('1234567890', out, (text, out))
import time
t0 = time.monotonic(); account._clean(('abcdefgh' * 12500)[:100000], key)
self.assertLess(time.monotonic() - t0, 2.0)
def test_redaction_catches_every_spelling_of_the_key(self):
hexed = ''.join('%%%02X' % ord(c) for c in KEY)
for spelling in (KEY, KEY.lower(), KEY.upper(), hexed, hexed.lower(),
KEY[:5] + hexed[15:30] + KEY[10:], # partly encoded
KEY[:9], hexed[:30].lower(), # starts of it, 8+ characters
''.join('%%%02x' % ord(c) for c in KEY.lower()), # encoded after a case change
''.join('%%%02X' % ord(c) for c in KEY.swapcase()),
'%%%02x' % ord(KEY[0].swapcase()) + KEY[1:]): # one encoded, recased letter
cleaned = account._clean('before ' + spelling + ' after', KEY)
self.assertEqual(cleaned, 'before [API key] after', spelling)
double = ''.join('%%25%02X' % ord(c) for c in KEY) # percent-encoded twice
for spelling in (double, double.lower(), double.upper(), double[:5 * 9], # full, hex recased, 9-char start
''.join('%%25%02x' % ord(c) for c in KEY.lower()), # the key recased, then double-encoded
KEY[:4] + double[20:45] + hexed[27:33] + KEY[11:]): # literal, double and single mixed
self.assertEqual(account._clean('before ' + spelling + ' after', KEY), 'before [API key] after', spelling)
self.assertEqual(account._clean(double[:5 * 7], KEY), double[:5 * 7]) # 7 characters: kept
self.assertTrue(account._carries_key('https://cdn.example/a?x=' + double, KEY))
self.assertTrue(account._carries_key('https://cdn.example/' + double[:5 * 8].lower() + '/a', KEY))
self.assertEqual(account._clean('TESTKEY is short', KEY), 'TESTKEY is short') # 7 characters: kept
self.assertEqual(account._clean(KEY + KEY.lower(), KEY), '[API key][API key]')
for url in ('https://cdn.example/a?x=' + hexed, 'https://cdn.example/a?x=' + hexed.lower(),
'https://cdn.example/' + KEY.upper() + '/a', 'https://cdn.example/a?x=' + urllib.parse.quote_plus(KEY)):
self.assertTrue(account._carries_key(url, KEY), url)
self.assertFalse(account._carries_key('https://cdn.example/a?token=abc&Signature=x', KEY))
# Storage errors carrying an encoded or lowercased key lose it, keeping their kind.
self.connect()
self.storage()
with patch.object(_web, 'public_apk', side_effect=SourceLimited('slow ' + hexed.lower(), 42)):
with self.assertRaises(SourceLimited) as caught:
itch.download(self.owned_source(), '341190')
self.assertEqual((str(caught.exception), caught.exception.retry_after), ('slow [API key]', 42))
with patch.object(_web, 'public_apk', side_effect=SourceError('host ' + KEY.lower() + '.cdn')):
with self.assertRaisesRegex(SourceError, r'^host \[API key\]\.cdn$'):
itch.download(self.owned_source(), '341190')
def test_storage_errors_lose_the_key_but_keep_their_kind(self):
self.connect()
self.storage()
with patch.object(_web, 'public_apk', side_effect=SourceLimited('slow down ' + KEY, 90)):
with self.assertRaises(SourceLimited) as caught:
itch.download(self.owned_source(), '341190')
self.assertEqual((str(caught.exception), caught.exception.retry_after), ('slow down [API key]', 90))
with patch.object(_web, 'public_apk', side_effect=SourceError('Could not find the download server ' + KEY)):
with self.assertRaisesRegex(SourceError, r'^Could not find the download server \[API key\]$') as caught:
itch.download(self.owned_source(), '341190')
self.assertIsNone(caught.exception.__cause__)
self.assertTrue(caught.exception.__suppress_context__)
def test_broken_http_responses_are_readable_and_lose_the_key(self):
self.connect()
class Truncated(Reply):
def read(self, *args):
raise http.client.IncompleteRead(b'{"owned_keys": [' + KEY.encode(), 1000)
cases = [http.client.BadStatusLine('HTTP/1.1 ' + KEY + ' oops'),
lambda req, q: Truncated(req.full_url, 200, b''),
http.client.RemoteDisconnected('closed after ' + KEY)]
for case in cases:
account._forget()
self.web.route('api.itch.io/profile/owned-keys', case)
with self.assertRaisesRegex(SourceError, '^Could not reach itch.io: ') as caught:
account.owned_keys()
self.assertNotIn(KEY, str(caught.exception))
self.assertIsNone(caught.exception.__cause__)
self.assertTrue(caught.exception.__suppress_context__)
self.assertTrue(account.connected())
# The same on the download request, for each kind of answer it reads.
for case in (http.client.BadStatusLine('HTTP/1.1 ' + KEY),
lambda req, q: Truncated(req.full_url, 200, b'', {'Content-Type': 'application/json'}),
lambda req, q: Truncated(req.full_url, 200, b'', {'Content-Type': 'application/octet-stream'})):
self.web.route('api.itch.io/uploads/503/download', case)
with self.assertRaises(SourceError) as caught:
account.download(503, 9001)
self.assertNotIn(KEY, str(caught.exception))
self.assertRegex(str(caught.exception), '^Could not (reach itch.io|download APK): ')
def test_connecting_shares_one_deadline_across_addresses(self):
clock, deadlines = [1000.0], []
def slow(host, address, timeout, deadline):
deadlines.append(deadline)
clock[0] += 7 # each address hangs for 7 s before failing
raise socket.timeout('timed out')
with patch.object(_web.time, 'monotonic', lambda: clock[0]), patch.object(_web, '_connect', slow):
with self.assertRaises(OSError):
_web._public_connection('many.example', 60)
self.assertEqual(len(set(deadlines)), 1) # one deadline for every attempt
self.assertEqual(deadlines[0], 1000.0 + _web.CONNECT_BUDGET)
self.assertEqual(len(deadlines), 3) # 8 addresses, but only 21 s of trying, not 56
self.assertLessEqual(clock[0] - 1000.0, _web.CONNECT_BUDGET + 7)
deadlines.clear()
clock[0] = 1000.0
with patch.object(_web.time, 'monotonic', lambda: clock[0]), patch.object(_web, '_connect', slow):
with self.assertRaises(OSError):
_web._public_connection('many.example', 5) # the caller's shorter timeout wins
self.assertEqual(deadlines, [1005.0])
def test_a_stalled_handshake_stops_at_the_deadline(self):
# A local server that accepts and then says nothing: the TLS handshake never completes.
server = socket.socket()
server.bind(('127.0.0.1', 0))
server.listen(1)
self.addCleanup(server.close)
timeouts = []
real_settimeout = socket.socket.settimeout
def settimeout(sock, value):
timeouts.append(value)
return real_settimeout(sock, value)
start = time.monotonic()
# No thread is left behind to shut the connection later: the deadline is only socket timeouts.
with patch.object(_web.threading, 'Timer', side_effect=AssertionError('watchdog thread')), \
patch.object(socket.socket, 'settimeout', settimeout):
with self.assertRaises(OSError):
REAL_CONNECT('storage.itch-cdn.example', server.getsockname(), 60, start + 0.5)
self.assertLess(time.monotonic() - start, 3)
self.assertTrue(timeouts and all(0 < t <= 0.5 for t in timeouts), timeouts) # what's left, each step
with patch.object(_web.socket, 'create_connection', side_effect=AssertionError('connected')):
with self.assertRaises(socket.timeout): # a deadline already gone: no connection attempt at all
REAL_CONNECT('storage.itch-cdn.example', server.getsockname(), 60, time.monotonic() - 1)
def test_an_unreachable_address_falls_back_to_the_next_checked_one(self):
self.connect()
self.storage('https://dual.example/upload/503/OrbitGarden-Quest.apk')
self.web.route('dual.example/upload/503/OrbitGarden-Quest.apk', Reply('https://dual.example/', 200, apk_bytes()))
self.unreachable.add('2a04:4e42::81')
self.assertTrue(os.path.isfile(itch.download(self.owned_source(), '341190')['apk']))
self.assertEqual(self.connected, [('dual.example', ('2a04:4e42::81', 443)), ('dual.example', ('151.101.1.1', 443))])
self.unreachable.add('151.101.1.1')
with self.assertRaisesRegex(SourceError, '^Could not download APK: Network is unreachable$'):
itch.download(self.owned_source(), '341190')
def test_storage_refusals_are_download_errors_not_rate_limits(self):
self.connect()
self.storage()
self.web.route('storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk',
Reply('https://storage.itch-cdn.example/x', 403, b'denied'))
with self.assertRaisesRegex(SourceError, r"itch.io's download server refused the file \(HTTP 403\)") as caught:
itch.download(self.owned_source(), '341190')
self.assertNotIsInstance(caught.exception, SourceLimited)
self.assertTrue(account.connected())
self.assertEqual(_web.wait_time('https://api.itch.io/'), 0)
self.web.route('storage.itch-cdn.example/upload/503/OrbitGarden-Quest.apk',
Reply('https://storage.itch-cdn.example/x', 429, b'', {'Retry-After': '60'}))
with self.assertRaises(SourceLimited):
itch.download(self.owned_source(), '341190')
def test_a_key_revoked_after_details_were_cached_is_removed_at_download(self):
self.connect()
itch.details(self.owned_source(), '341190') # uploads and owned keys now cached
for code in (401, 403):
self.connect()
self.web.route('api.itch.io/uploads/503/download', Reply('https://api.itch.io/uploads/503/download', code, b''))
self.web.route('api.itch.io/profile', Reply('https://api.itch.io/profile', code, b''))
with self.assertRaisesRegex(account.BadKey, 'no longer accepts the saved API key'):
account.download(503, 9001)
self.assertFalse(account.path().exists())
self.web.route('api.itch.io/profile', {'user': {'username': 'player'}})
# The key still works, only this download is refused: keep the key, say so.
self.connect()
self.web.route('api.itch.io/uploads/503/download', Reply('https://api.itch.io/uploads/503/download', 403, b''))
with self.assertRaisesRegex(SourceError, r'refused this request \(HTTP 403\)') as caught:
account.download(503, 9001)
self.assertNotIsInstance(caught.exception, (account.BadKey, SourceLimited))
self.assertTrue(account.connected())
def test_a_disconnect_during_the_key_check_wins(self):
checking, release = threading.Event(), threading.Event()
def slow_profile(req, query):
checking.set()
release.wait(5)
return {'user': {'username': 'player'}}
self.web.route('api.itch.io/profile', slow_profile)
outcome = {}
def run():
try:
outcome['result'] = account.connect(KEY)
except SourceError as e:
outcome['error'] = e
worker = threading.Thread(target=run)
worker.start()
self.assertTrue(checking.wait(5))
account.disconnect()
release.set()
worker.join(5)
self.assertRegex(str(outcome.get('error')), 'changed while the key was being checked; nothing was saved')
self.assertFalse(account.path().exists())
self.assertFalse(account.connected())
def test_an_old_refusal_does_not_undo_a_reconnect_with_the_same_key(self):
self.connect()
listing, release, profile_ok = threading.Event(), threading.Event(), [True]
def slow_owned(req, query):
listing.set()
release.wait(5)
return Reply(req.full_url, 401, b'')
self.web.route('api.itch.io/profile/owned-keys', slow_owned)
self.web.route('api.itch.io/profile', lambda req, q: {'user': {'username': 'player'}} if profile_ok[0]
else Reply(req.full_url, 401, b''))
outcome = {}
def run():
try:
account.owned_keys()
except SourceError as e:
outcome['error'] = e
worker = threading.Thread(target=run)
worker.start()
self.assertTrue(listing.wait(5))
account.disconnect()
account.connect(KEY) # same key, connected again while the old listing waits
profile_ok[0] = False # the old listing's confirmation now sees a refusal
release.set()
worker.join(5)
self.assertIsInstance(outcome.get('error'), account.BadKey)
self.assertTrue(account.path().exists()) # the fresh connection survives
self.assertEqual(json.loads(account.path().read_text())['key'], KEY)
def test_redacted_text_is_not_mistaken_for_a_key_error(self):
self.connect()
before = len(self.web.urls('api.itch.io/profile'))
self.web.route('api.itch.io/profile/owned-keys', {'errors': ['Invalid request: ' + KEY]})
with self.assertRaisesRegex(SourceError, r'^itch.io: Invalid request: \[API key\]$') as caught:
account.owned_keys()
self.assertNotIsInstance(caught.exception, account.BadKey)
self.assertTrue(account.connected())
self.assertEqual(len(self.web.urls('api.itch.io/profile')), before) # not even a key check needed
def test_an_invalid_key_error_elsewhere_is_confirmed_before_deleting(self):
self.connect()
self.web.route('api.itch.io/profile/owned-keys', {'errors': ['invalid key']})
with self.assertRaisesRegex(SourceError, '^itch.io: invalid key$') as caught:
account.owned_keys() # /profile still accepts the key: keep it
self.assertNotIsInstance(caught.exception, account.BadKey)
self.assertTrue(account.connected())
account._forget()
self.web.route('api.itch.io/profile', {'errors': ['invalid key']})
with self.assertRaisesRegex(account.BadKey, 'no longer accepts'):
account.owned_keys()
self.assertFalse(account.connected())
def test_concurrent_connects_never_share_a_temporary_file(self):
made = []
real = account.tempfile.mkstemp
def mkstemp(*args, **kwargs):
fd, name = real(*args, **kwargs)
made.append(name)
if os.name == 'posix':
self.assertEqual(os.stat(name).st_mode & 0o777, 0o600)
return fd, name
with patch.object(account.tempfile, 'mkstemp', mkstemp):
workers = [threading.Thread(target=lambda k=k: self.swallow(account.connect, k)) for k in (KEY, OTHER) * 3]
for w in workers:
w.start()
for w in workers:
w.join(5)
self.assertEqual(len(made), len(set(made)))
self.assertIn(json.loads(account.path().read_text())['key'], (KEY, OTHER))
self.assertEqual([p.name for p in account.path().parent.iterdir()], ['itch-account.json'])
@staticmethod
def swallow(fn, *args):
try:
fn(*args)
except SourceError:
pass # a connect overtaken by another one refuses to save: fine here
def test_existing_hosts_still_strip_authorization_on_redirect(self):
req = urllib.request.Request('https://api.github.com/a', headers={'Authorization': 'Bearer x'})
nxt = _web.Redirect(('api.github.com',)).redirect_request(req, None, 302, 'Found', {}, 'https://api.github.com/b')
self.assertFalse(nxt.has_header('Authorization'))
with self.assertRaises(SourceError):
_web.Redirect(('api.github.com',)).redirect_request(req, None, 302, 'Found', {}, 'https://evil.example/b')
def test_install_uses_the_normal_apk_install_path(self):
import frame_android
self.connect()
self.storage()
calls = []
def install(apk_path, flatscreen=None, name=None, source=None, icon_png=None, xr_compat=None, artwork=None):
with zipfile.ZipFile(apk_path) as z:
self.assertIn('AndroidManifest.xml', z.namelist())
calls.append((name, source))
return {'package': 'com.maker.orbit', 'message': 'Installed'}
with patch.object(frame_android, 'install', install):
result = search.install(itch.OWNED, '341190')
self.assertEqual(calls, [('Orbit Garden VR', 'itch.io (my purchases)')])
self.assertEqual(result['package'], 'com.maker.orbit')
# -- errors never carry the key ------------------------------------------------
def test_errors_are_readable_and_never_contain_the_key(self):
self.connect()
cases = [
(Reply('https://api.itch.io/profile/owned-keys', 429, b'', {'Retry-After': '120'}),
SourceLimited, '^itch.io is limiting requests; try again in 2 minutes$'),
(Reply('https://api.itch.io/profile/owned-keys', 503, b''), SourceError, 'HTTP error 503; try again later'),
(urllib.error.URLError('connect to api.itch.io with ' + KEY + ' failed'), SourceError,
r'^Could not reach itch.io: connect to api.itch.io with \[API key\] failed$'),
(OSError('timed out'), SourceError, '^Could not reach itch.io: timed out$'),
(Reply('https://api.itch.io/profile/owned-keys', 200, b'<html>'), SourceError, 'invalid response'),
({'errors': ['echo ' + KEY]}, SourceError, r'^itch.io: echo \[API key\]$'),
# The key straddles the 200-character cut: no piece of it may survive.
({'errors': ['x' * 180, KEY]}, SourceError, r'^itch.io: x{180}; \[API key\]$'),
({'errors': ['x' * 190 + KEY]}, SourceError, r'^itch.io: x{190}\[API key\]$'),
]
self.assertEqual(account._clean('cut at ' + KEY[:9], KEY), 'cut at [API key]') # a truncated key's start
stderr = io.StringIO()
with contextlib.redirect_stderr(stderr):
for reply, kind, message in cases:
account._forget()
_web._limited.clear()
self.web.route('api.itch.io/profile/owned-keys', reply)
with self.assertRaisesRegex(kind, message) as caught:
account.owned_keys()
self.assertNotIn(KEY, str(caught.exception))
self.assertIsNone(caught.exception.__cause__)
self.assertTrue(account.connected()) # not the key's fault: stays connected
# The store's search reports the failure as a non-fatal source status.
account._forget()
result = search.search('', source=itch.OWNED)
self.assertEqual(result['sources'][0]['status'], 'error')
self.assertNotIn(KEY, json.dumps(result) + stderr.getvalue())
def test_server_api_connects_and_never_echoes_the_key(self):
httpd = server.ThreadingHTTPServer(('127.0.0.1', 0), server.Handler)
threading.Thread(target=httpd.serve_forever, daemon=True).start()
self.addCleanup(httpd.server_close)
self.addCleanup(httpd.shutdown)
def request(method, path, body=None):
c = http.client.HTTPConnection('127.0.0.1', httpd.server_port)
c.request(method, path, json.dumps(body) if body is not None else None,
{'X-Frame-UI': '1', 'Content-Type': 'application/json'})
r = c.getresponse()
out = r.status, r.read().decode()
c.close()
return out
stderr = io.StringIO()
with contextlib.redirect_stderr(stderr):
self.web.route('api.itch.io/profile', Reply('https://api.itch.io/profile', 401, b''))
bad = request('POST', '/api/sources', {'action': 'itch-connect', 'key': KEY})
self.web.route('api.itch.io/profile', {'user': {'username': 'player'}})
good = request('POST', '/api/sources', {'action': 'itch-connect', 'key': KEY})
listed = request('GET', '/api/sources')
empty = request('POST', '/api/sources', {'action': 'itch-connect', 'key': ' '})
gone = request('POST', '/api/sources', {'action': 'itch-disconnect'})
self.assertEqual(bad[0], 400)
self.assertIn('did not accept the API key', json.loads(bad[1])['error'])
self.assertEqual(good[0], 200)
self.assertEqual(json.loads(good[1])['itch'], {'connected': True, 'username': 'player'})
self.assertEqual(json.loads(listed[1])['itch'], {'connected': True, 'username': 'player'})
self.assertIn(itch.OWNED, [s['id'] for s in json.loads(listed[1])['sources']])
self.assertEqual(empty[0], 400)
self.assertEqual(json.loads(gone[1])['itch'], {'connected': False, 'username': None})
self.assertFalse(account.path().exists())
for text in (bad[1], good[1], listed[1], empty[1], gone[1], stderr.getvalue()):
self.assertNotIn(KEY, text)
if __name__ == '__main__':
unittest.main()
+1
View File
@@ -1,4 +1,5 @@
"""Every public sideload entry point reaches the mandatory five-slot writer."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import contextlib
import io
import json
+428 -5
View File
@@ -9,6 +9,7 @@ import http.client
import io
import json
import os
import shlex
import shutil
import socket
import subprocess
@@ -199,7 +200,7 @@ class Connecting(unittest.TestCase):
rows = {p["host"]: p for p in s["probes"]}
self.assertEqual(rows["nothing.invalid"]["state"], "unresolved")
self.assertEqual(rows["::1"]["state"], "sshfailed")
self.assertIn("different headset", rows["::1"]["detail"])
self.assertIn("different SSH identity", rows["::1"]["detail"])
# Every ssh command was pointed at the winner, with the host key pinned per device.
alias, opts = self.routes[-1]
self.assertEqual(alias, "frame-t")
@@ -208,9 +209,10 @@ class Connecting(unittest.TestCase):
self.assertIn(f"Port={self.port}", opts)
master = [c for c in self.calls() if "ControlMaster=yes" in c][-1]
self.assertIn("StrictHostKeyChecking=accept-new", master) # first connection: nothing pinned yet
# ssh takes an option's first value: accept-new must come before the commands' own "yes".
self.assertLess(master.index("StrictHostKeyChecking=accept-new"), master.index("StrictHostKeyChecking=yes"))
self.assertIn("StrictHostKeyChecking=yes", opts) # every other command checks the pinned key
self.assertNotIn("StrictHostKeyChecking=yes", master)
# This stand-in ssh saved no key, so commands may still save the first one (accept-new);
# test_a_first_connection_pins_and_then_checks_strictly has real pins.
self.assertIn("StrictHostKeyChecking=accept-new", opts)
self.assertTrue(fd.known_hosts(d["id"]).parent.is_dir()) # where ssh saves the key it accepts
# It learned: 127.0.0.1 works on this network.
learned = {a["host"]: a for a in self.reg.get(d["id"])["addresses"]}
@@ -257,6 +259,7 @@ class Connecting(unittest.TestCase):
s = self.link.snapshot()
self.assertEqual((s["phase"], s["error"]["stage"]), ("failed", "login"))
self.assertIn("SSH key", s["error"]["message"])
self.assertEqual(s["error"]["auth"], "denied") # the page offers Set Up Connection
def test_pinned_identity_is_checked_strictly(self):
d = self.device("localhost")
@@ -266,6 +269,426 @@ class Connecting(unittest.TestCase):
master = [c for c in self.calls() if "ControlMaster=yes" in c][-1]
self.assertIn("StrictHostKeyChecking=yes", master)
def pins(self):
"""The stand-in ssh checks host keys as OpenSSH does (tests/fakessh/ssh)."""
p = mock.patch.dict(os.environ, {"FAKESSH_PINS": "1"})
p.start()
self.addCleanup(p.stop)
def command(self, device, host="127.0.0.1"):
"""One ssh command as the server runs it, with the route the connector applied."""
return subprocess.run(["ssh", *self.link.host_opts(device, host), device["alias"], "true"],
capture_output=True, text=True, timeout=20)
def test_a_first_connection_pins_and_then_checks_strictly(self):
# 0.4.2 on Windows: "No ED25519 host key is known for frame-control-<id> and you have
# requested strict checking." Every command said "yes" before anything was saved, so one
# that ran before the connector's handshake saved the key (or without it) failed.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
r = self.command(d)
self.assertEqual(r.returncode, 0, r.stderr) # the first contact saves the key...
self.assertIn("Permanently added", r.stderr)
self.assertTrue(fd.pinned(d["id"]))
self.assertIn("StrictHostKeyChecking=yes", self.link.host_opts(d, "127.0.0.1")) # ...then it's checked
self.assertEqual(self.reg.get(d["id"])["identity"], [k[1] for k in fd.pinned_keys(d["id"])]) # and recorded
# The connector's first handshake, for another new headset.
self.reg.remove_device(d["id"])
d = self.device("localhost")
self.link.connect(["start"])
s = self.link.snapshot()
self.assertEqual(s["phase"], "connected", s["error"])
self.assertTrue(fd.pinned(d["id"]))
self.assertTrue(self.reg.get(d["id"])["identity"])
alias, opts = self.routes[-1]
self.assertIn("StrictHostKeyChecking=yes", opts) # every command after it checks strictly
self.assertNotIn("StrictHostKeyChecking=accept-new", opts)
def lose_pin(self, device_id, how):
f = fd.known_hosts(device_id)
if how == "deleted":
f.unlink()
elif how == "empty":
f.write_text("")
elif how == "unreadable":
f.chmod(0)
self.addCleanup(lambda: f.exists() and f.chmod(0o600))
elif how == "another alias":
f.write_text(f.read_text().replace(f"frame-control-{device_id}", "frame-control-someone-else"))
elif how == "not utf-8":
f.write_bytes(b"\xff\xfe" + f.read_bytes())
def test_a_lost_or_unusable_pin_fails_closed(self):
# Review pr85-review-r1: accept-new can't tell a new headset from one whose saved key
# went missing; real OpenSSH then trusted a replacement key. Once a headset had an
# identity, nothing is trusted again until the person checks it.
self.pins()
cases = ["deleted", "empty", "another alias", "not utf-8"]
if os.name == "posix" and os.geteuid() != 0:
cases.append("unreadable") # root reads anything
for n, how in enumerate(cases):
with self.subTest(how=how):
self.hosts({"localhost": "ok"})
d = self.reg.add_device(f"frame-l{n}", port=self.port, hosts=[])
self.reg.add_address(d["id"], "localhost", kind="lan")
self.link.override = None
self.reg.set_active(d["id"])
self.link.connect(["switch"])
self.assertEqual(self.link.snapshot()["phase"], "connected")
self.link.close_master()
d = self.reg.get(d["id"])
self.lose_pin(d["id"], how)
self.hosts({"localhost": "wrong"}) # another device, or a reinstalled headset
r = self.command(d)
self.assertEqual(r.returncode, 255, r.stderr)
self.assertIn("Host key verification failed", r.stderr)
self.assertNotIn("Permanently added", r.stderr)
self.link.connect(["dropped"])
s = self.link.snapshot()
self.assertEqual((s["phase"], s["error"]["stage"]), ("failed", "identity"))
self.assertEqual(s["error"]["hostkey"]["state"], "missing")
self.assertIn("missing or can't be read", s["error"]["message"])
view = next(x for x in fl.devices_view(self.link)["devices"] if x["id"] == d["id"])
self.assertTrue(view["identity_lost"])
scan = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)["scan"]
self.assertFalse(scan["matches"] or scan["recorded_match"]) # not the key it had
self.hosts({"localhost": "ok"})
scan = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)["scan"]
self.assertTrue(scan["recorded_match"]) # its own key, as recorded
self.assertFalse(scan["matches"]) # but not usable from the file: trusting restores it
fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"],
"fingerprint": scan["fingerprint"]}, None)
self.link.connect(["switch"])
self.assertEqual(self.link.snapshot()["phase"], "connected", self.link.snapshot()["error"])
self.link.close_master()
def test_a_key_check_only_reads_the_key(self):
# Review pr85-review-r1: with ForwardAgent yes in ~/.ssh/config, a device that accepts
# "none" got the agent and ran a command before anyone trusted it.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "evil"})
os.environ["FAKESSH_CONFIG"] = json.dumps({"forwardagent": "yes", "remotecommand": "touch /tmp/x",
"permitlocalcommand": "yes", "requesttty": "force"})
self.addCleanup(os.environ.pop, "FAKESSH_CONFIG", None)
for proxied in (False, True):
route = ("jump", "me@jumpy:2222") if proxied else (None, None)
with self.subTest(proxied=proxied), mock.patch.object(fl, "proxy_route", return_value=route):
t0 = time.monotonic()
scan = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)["scan"]
self.assertLess(time.monotonic() - t0, 10) # stopped once it had the key, not left open
self.assertTrue(scan["fingerprint"].startswith("SHA256:"))
self.assertEqual([p.name for p in self.dir.iterdir() if p.name.startswith("evil-")], [])
call = [c for c in self.calls() if "PreferredAuthentications=none" in c][-1]
self.assertEqual(call[call.index("-F") + 1], "none") # never ~/.ssh/config for the device
self.assertIn("-N", call)
hop = [c for c in call if c.startswith("ProxyCommand=")]
if proxied:
# Review pr85-review-r2: ProxyJump's own ssh inherits the config (ForwardAgent,
# LocalCommand); the scan builds the jump host's ssh itself, hardened.
self.assertEqual(len(hop), 1)
words = shlex.split(hop[0][len("ProxyCommand="):])
self.assertEqual(words[-7:], ["-l", "me", "-p", "2222", "-W", "[%h]:%p", "jumpy"])
for o in fl.Link.HOP_OPTS:
self.assertIn(o, words)
self.assertLess(call.index(hop[0]), call.index("HostKeyAlias=frame-control-" + d["id"]))
else:
self.assertEqual(hop, [])
for o in ("ForwardAgent=no", "IdentityAgent=none", "ClearAllForwardings=yes", "RemoteCommand=none",
"PermitLocalCommand=no", "RequestTTY=no", "PubkeyAuthentication=no"):
self.assertIn(o, call)
self.assertLess(call.index("StrictHostKeyChecking=accept-new"), call.index("HostKeyAlias=frame-control-"
+ d["id"]))
self.assertFalse(fd.pinned(d["id"]))
self.assertEqual(sorted(p.name for p in fd.known_hosts(d["id"]).parent.iterdir()), []) # no scan files
@unittest.skipIf(os.name != "posix", "the stand-in ssh runs ProxyCommand with /bin/sh")
def test_the_jump_hosts_command_line_is_quoted_whatever_the_names(self):
# Review pr85-review-r3: the hop's words went into the ProxyCommand line unquoted, so an
# ssh at /tmp/x/bin$(:>INJECTED)/ssh ran shell code; and an IPv6 jump host broke.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
for n, name in enumerate(["with space", "bin$(touch INJECTED-0)", "back`touch INJECTED-1`tick",
"quote'it\"s", "per%cent %h %%", "semi;touch INJECTED-2"]):
with self.subTest(name=name):
bindir = self.dir / f"b{n}" / name
bindir.mkdir(parents=True)
(bindir / "ssh").symlink_to(FAKESSH / "ssh")
for spec, at in (("me@jumpy:2222", "jumpy"), ("[::1]:2200", "::1")):
before = len(self.calls())
with mock.patch.dict(os.environ, {"PATH": f"{bindir}{os.pathsep}{os.environ['PATH']}"}), \
mock.patch.object(fl, "proxy_route", return_value=("jump", spec)):
scan = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)["scan"]
self.assertTrue(scan["fingerprint"].startswith("SHA256:"))
hops = [c for c in self.calls()[before:] if "-W" in c]
self.assertEqual(len(hops), 1, self.calls()[before:])
self.assertEqual(hops[0][-1], at) # the jump host, bare
self.assertEqual(hops[0][hops[0].index("-W") + 1], "[localhost]:%d" % self.port)
outer = [c for c in self.calls()[before:] if "PreferredAuthentications=none" in c][-1]
line = next(c for c in outer if c.startswith("ProxyCommand="))
self.assertTrue(line[len("ProxyCommand="):].startswith(shlex.quote(str(bindir / "ssh")
.replace("%", "%%"))))
self.assertEqual([p.name for p in self.dir.rglob("INJECTED*")], [])
self.assertFalse(list(Path.cwd().glob("INJECTED*")))
def test_a_key_check_refuses_routes_it_cant_harden(self):
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
before = len(self.calls())
for route in (("command", "nc %h %p"), ("jump", "a,b"), ("jump", "-oProxyCommand=x"), ("jump", "ssh://a"),
("jump", "a b"), ("jump", "a%h"), ("jump", "jumpy:99999")):
with self.subTest(route=route), mock.patch.object(fl, "proxy_route", return_value=route):
with self.assertRaises(fd.DeviceError) as caught:
fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)
self.assertIn("can't make safe", str(caught.exception))
self.assertFalse(any("PreferredAuthentications=none" in c for c in self.calls()[before:]))
def test_a_key_read_before_the_address_or_port_changed_isnt_trusted(self):
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
for change in ({"action": "update", "id": d["id"], "port": self.port + 1},
{"action": "address-remove", "id": d["id"], "host": "localhost"}):
with self.subTest(change=change["action"]):
scan = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)["scan"]
fl.devices_action(self.link, change, None)
with self.assertRaises(fd.DeviceError):
fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"],
"fingerprint": scan["fingerprint"]}, None)
self.assertFalse(fd.pinned(d["id"]))
self.reg.update_device(d["id"], port=self.port)
if not self.reg.get(d["id"])["addresses"]:
self.reg.add_address(d["id"], "localhost", kind="lan")
def test_terminals_judge_identity_by_the_registry_not_a_frozen_route(self):
# Review pr85-review-r2: named_route used the device as routed (before its first key was
# recorded), so a Terminal window fell back to accept-new once the pin was deleted.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
self.link.connect(["start"])
self.assertEqual(self.link.snapshot()["phase"], "connected")
self.assertFalse(self.link.routed_device.get("identity")) # the frozen copy knows nothing of it
fd.known_hosts(d["id"]).unlink()
alias, opts = self.link.named_route()
self.assertIn("StrictHostKeyChecking=yes", opts)
self.assertNotIn("StrictHostKeyChecking=accept-new", opts)
self.hosts({"localhost": "wrong"})
r = subprocess.run(["ssh", *opts, alias, "true"], capture_output=True, text=True, timeout=20)
self.assertEqual(r.returncode, 255, r.stderr)
self.assertFalse(fd.pinned(d["id"]))
def test_a_key_saved_before_a_refused_login_is_kept_and_checked(self):
# Review pr85-review-r2: ssh saves the key during the key exchange; if the login is then
# refused, it was neither recorded nor protected, and the route stayed accept-new.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "denied"})
self.link.connect(["start"])
s = self.link.snapshot()
self.assertEqual((s["phase"], s["error"]["stage"]), ("failed", "login"))
got = self.reg.get(d["id"])
self.assertEqual(got["identity"], [f for _, f in fd.pinned_keys(d["id"])])
self.assertTrue(got["identity_established"])
if os.name == "posix":
self.assertEqual(fd.known_hosts(d["id"]).stat().st_mode & 0o777, 0o600)
self.assertIn("StrictHostKeyChecking=yes", self.routes[-1][1])
fd.known_hosts(d["id"]).unlink()
self.hosts({"localhost": "wrong"})
r = self.command(self.reg.get(d["id"]))
self.assertEqual(r.returncode, 255)
self.assertNotIn("Permanently added", r.stderr)
def test_a_key_a_command_saved_tightens_the_route(self):
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "denied"})
with self.link.route_lock:
self.link.routed_device = d
self.link.apply(d["alias"], self.link.host_opts(d, "127.0.0.1"))
opts = self.routes[-1][1]
self.assertIn("StrictHostKeyChecking=accept-new", opts)
r = subprocess.run(["ssh", *opts, d["alias"], "true"], capture_output=True, text=True, timeout=20)
self.assertIn("Permanently added", r.stderr)
self.assertEqual(r.returncode, 255) # saved the key, then refused the login
self.link.settle_pin() # what the connector's loop does every tick while accept-new is applied
self.assertIn("StrictHostKeyChecking=yes", self.routes[-1][1])
self.assertNotIn("StrictHostKeyChecking=accept-new", self.routes[-1][1])
self.assertTrue(self.reg.get(d["id"])["identity"])
def test_editing_the_address_that_worked_keeps_the_identity_history(self):
# Review pr85-review-r2: last_ok lives on the address, so editing or removing it made a
# headset whose pin was gone (and had no fingerprint recorded) new again.
path = self.dir / "legacy.json"
path.write_text(json.dumps({"version": 1, "active": "abcd1234", "networks": {}, "devices": [
{"id": "abcd1234", "name": "Old", "alias": "frame-old", "user": "steamos", "port": 22,
"identity_files": [], "addresses": [{"host": "10.0.0.5", "kind": "lan", "label": "",
"networks": [], "last_ok": 1790000000.0, "last_rtt_ms": 3}]}]}))
reg = fd.Registry(path)
self.assertTrue(reg.get("abcd1234")["identity_established"]) # migrated from last_ok
reg.update_address("abcd1234", "10.0.0.5", new_host="10.0.0.6")
reg.remove_address("abcd1234", "10.0.0.6")
reg.add_address("abcd1234", "10.0.0.7")
d = fd.Registry(path).get("abcd1234") # saved, too
self.assertFalse(any(a.get("last_ok") for a in d["addresses"]))
self.assertTrue(fd.identity_established(d))
self.assertEqual(self.link.strictness(d), "yes")
# A connection that works sets it as well, for a headset added since.
n = self.reg.add_device("frame-new", hosts=["10.0.0.8"])
self.reg.record_success(n["id"], "10.0.0.8", None, 1)
self.reg.remove_address(n["id"], "10.0.0.8")
self.assertTrue(self.reg.get(n["id"])["identity_established"])
def test_ssh_last_words_arrive_even_when_the_reader_lags(self):
# Review pr85-review-r2: ssh's exit with nothing queued yet was taken as the end, and a
# late reader lost "Permission denied" ("ssh stopped").
self.pins()
link = fl.Link(self.reg, env_alias=None, mux_base=["ssh", "-o", "BatchMode=yes"], control=None,
apply=lambda *a: None, explain=explain)
self.addCleanup(link.stop)
real = threading.Thread
class Late(real):
def __init__(self, *a, target=None, **kw):
if target is not None and getattr(target, "__name__", "") == "tail":
inner = target
def target():
time.sleep(1)
inner()
super().__init__(*a, target=target, **kw)
self.device("localhost")
self.hosts({"localhost": "denied"})
with mock.patch.object(fl.threading, "Thread", Late):
link.connect(["start"])
s = link.snapshot()
self.assertEqual((s["error"]["stage"], s["error"].get("auth")), ("login", "denied"), s["error"])
self.assertIn("SSH key", s["error"]["message"])
def test_forgetting_an_identity_is_refused(self):
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
self.link.connect(["start"])
saved = fd.known_hosts(d["id"]).read_text()
with self.assertRaises(fd.DeviceError) as caught:
fl.devices_action(self.link, {"action": "forget-identity", "id": d["id"]}, None)
self.assertIn("identity-scan", str(caught.exception))
self.assertEqual(fd.known_hosts(d["id"]).read_text(), saved)
def test_without_a_shared_connection_ssh_writes_to_a_file_not_a_pipe(self):
# Windows: no ControlMaster, and its OpenSSH can block writing to a stderr pipe.
self.pins()
tmp = self.dir / "tmp"
tmp.mkdir()
link = fl.Link(self.reg, env_alias=None, mux_base=["ssh", "-o", "BatchMode=yes"], control=None,
apply=lambda *a: None, explain=explain)
self.addCleanup(link.stop)
real = subprocess.Popen
seen = []
def popen(argv, **kw):
if "-v" in argv: # the handshake (run_ssh calls go through Popen too)
seen.append(kw.get("stderr"))
return real(argv, **kw)
with mock.patch.object(tempfile, "tempdir", str(tmp)), mock.patch.object(fl.subprocess, "Popen", popen):
d = self.device("localhost")
self.hosts({"localhost": "ok"})
link.connect(["start"])
s = link.snapshot()
self.assertEqual(s["phase"], "connected", s["error"])
self.assertEqual([st["state"] for st in s["stages"]], ["done"] * 5)
self.assertIn("saved this headset's identity", next(st for st in s["stages"] if st["id"] == "identity")["detail"])
self.reg.update_device(d["id"], user="nobody")
self.hosts({"localhost": "denied"})
link.connect(["switch"])
s = link.snapshot()
self.assertEqual((s["phase"], s["error"]["stage"], s["error"].get("auth")), ("failed", "login", "denied"))
self.assertTrue(seen and all(isinstance(x, int) for x in seen)) # a file descriptor, never PIPE
self.assertNotIn(subprocess.PIPE, seen)
for _ in range(50): # the reader removes its file once ssh has exited
if not list(tmp.iterdir()):
break
time.sleep(0.05)
self.assertEqual(list(tmp.iterdir()), [])
def test_a_changed_key_is_shown_never_accepted_and_trusted_on_request(self):
# 0.4.2 on Linux and Windows: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!", with
# "Run Set Up Connection again" as the advice, which can't replace the saved key.
self.pins()
d = self.device("localhost")
self.hosts({"localhost": "ok"})
self.link.connect(["start"])
self.assertEqual(self.link.snapshot()["phase"], "connected")
saved = fd.known_hosts(d["id"]).read_text()
old = fd.pinned_keys(d["id"])
self.link.close_master()
self.hosts({"localhost": "wrong"}) # SteamOS reinstalled: a new host key
self.link.connect(["dropped"])
s = self.link.snapshot()
self.assertEqual((s["phase"], s["error"]["stage"]), ("failed", "identity"))
shown = s["error"]["hostkey"]
self.assertEqual((shown["state"], shown["type"], shown["host"]), ("changed", "ssh-ed25519", "localhost"))
self.assertTrue(shown["fingerprint"].startswith("SHA256:"))
self.assertIn("Devices tab", s["error"]["message"])
self.assertEqual(fd.known_hosts(d["id"]).read_text(), saved) # never accepted on its own
r = self.command(d)
self.assertEqual(r.returncode, 255)
self.assertIn("REMOTE HOST IDENTIFICATION HAS CHANGED", r.stderr) # nor by any command
# Trusting needs the key read first, and exactly the fingerprint shown.
with self.assertRaises(fd.DeviceError):
fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"],
"fingerprint": shown["fingerprint"]}, None)
out = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)
scan = out["scan"]
self.assertEqual(scan["fingerprint"], shown["fingerprint"])
self.assertFalse(scan["matches"])
self.assertEqual(scan["saved"], [{"type": t, "fingerprint": f} for t, f in old])
self.assertEqual(fd.known_hosts(d["id"]).read_text(), saved) # reading it saves nothing
self.assertEqual(sorted(p.name for p in fd.known_hosts(d["id"]).parent.iterdir()), [d["id"]]) # no scan files
call = [c for c in self.calls() if "PreferredAuthentications=none" in c][-1] # no login offered
self.assertLess(call.index("StrictHostKeyChecking=accept-new"), call.index("StrictHostKeyChecking=yes"))
self.assertNotIn(f"UserKnownHostsFile={fd.known_hosts_opt(d['id'])}",
call[:call.index("StrictHostKeyChecking=yes")]) # a throwaway file, first
with self.assertRaises(fd.DeviceError):
fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"], "fingerprint": "SHA256:other"}, None)
self.link.scans[d["id"]]["at"] -= fl.SCAN_TTL + 1
with self.assertRaises(fd.DeviceError): # shown too long ago
fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"],
"fingerprint": shown["fingerprint"]}, None)
fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)
with mock.patch.object(self.link, "kick") as kick:
out = fl.devices_action(self.link, {"action": "identity-trust", "id": d["id"],
"fingerprint": shown["fingerprint"]}, None)
kick.assert_called_with("switch")
self.assertEqual(fd.pinned_keys(d["id"]), [("ssh-ed25519", shown["fingerprint"])]) # the old key is gone
self.assertEqual(next(x for x in out["devices"] if x["id"] == d["id"])["keys"],
[{"type": "ssh-ed25519", "fingerprint": shown["fingerprint"]}])
self.link.connect(["switch"])
self.assertEqual(self.link.snapshot()["phase"], "connected", self.link.snapshot()["error"])
out = fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)
self.assertTrue(out["scan"]["matches"])
def test_test_now_never_saves_a_key(self):
self.pins()
d = self.device("127.0.0.1")
self.hosts({"127.0.0.1": "ok"})
self.link.test(d["id"])
rows = self.link.snapshot()["tests"][d["id"]]["rows"]
self.assertEqual(rows[0]["ssh"], "unpinned")
self.assertFalse(fd.pinned(d["id"]))
def test_reading_a_key_from_a_headset_that_doesnt_answer_says_why(self):
self.pins()
d = self.device("nothing.invalid")
with self.assertRaises(fd.DeviceError) as caught:
fl.devices_action(self.link, {"action": "identity-scan", "id": d["id"]}, None)
self.assertIn("Can't find the Frame", str(caught.exception))
self.assertEqual([p.name for p in fd.known_hosts(d["id"]).parent.iterdir()], [])
def test_ssh_goes_to_the_ipv4_address_that_answered(self):
self.device("localhost")
self.hosts({"localhost": "ok"})
@@ -501,7 +924,7 @@ class Connecting(unittest.TestCase):
{"action": "update", "id": d["id"], "port": 2222},
{"action": "address-remove", "id": d["id"], "host": "localhost"},
{"action": "address-update", "id": d["id"], "host": "localhost", "newHost": "127.0.0.1"},
{"action": "forget-identity", "id": d["id"]}):
{"action": "identity-trust", "id": d["id"], "fingerprint": "SHA256:x"}):
with self.assertRaises(fd.DeviceError, msg=body):
fl.devices_action(self.link, body, open_setup=None, busy=lambda: 1)
# Renaming, or changing another headset, is fine.
+1
View File
@@ -7,6 +7,7 @@ input reaching the source, and closing.
Run: python3 -m unittest discover -s tests
"""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import base64
import http.client
import json
+1
View File
@@ -1,4 +1,5 @@
"""Owned media planning, eye isolation, decoder choice and fake-Frame ownership."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import argparse
import io
import json
+1
View File
@@ -1,4 +1,5 @@
"""Fake-Frame panel responses and the headset page's real HTTP guards."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import http.client
import json
from pathlib import Path
+122
View File
@@ -0,0 +1,122 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import os
import re
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "ui"))
import frame_connect # noqa: E402
import frame_devices # noqa: E402
import frame_host # noqa: E402
def inside(path, folder):
path, folder = os.path.realpath(path), os.path.realpath(folder)
return path == folder or path.startswith(folder + os.sep)
def snapshot(folder):
"""{relative path: (size, mtime)} for everything under FOLDER."""
out = {}
for base, dirs, files in os.walk(folder):
for name in dirs + files:
p = os.path.join(base, name)
try:
st = os.lstat(p)
except OSError:
continue
out[os.path.relpath(p, folder)] = (st.st_size, st.st_mtime_ns)
return out
class Sandbox(unittest.TestCase):
"""Tests once wrote a host-key pin into the real ~/.ssh/frame-control-hosts on every
run (a server started without FRAME_CONTROL_SSH_DIR imported the real ~/.ssh/config)."""
def test_home_and_ssh_dir_are_throwaway(self):
tmp = tempfile.gettempdir()
self.assertTrue(inside(Path.home(), tmp), Path.home())
self.assertTrue(inside(os.path.expanduser("~"), tmp))
for d in (frame_devices.ssh_dir(), frame_devices.pin_dir(), frame_connect.SSH_DIR,
os.environ["FRAME_CONTROL_SSH_DIR"], frame_host.cache_dir(), frame_host.data_dir()):
self.assertTrue(inside(d, tmp), d)
self.assertFalse(inside(d, sandbox.REAL_SSH_DIR), d)
def test_cache_folders_on_every_platform_are_throwaway(self):
"""cache_dir() reads LOCALAPPDATA on Windows and XDG_CACHE_HOME on Linux, which a
moved HOME alone doesn't cover: servers sweep staging folders there."""
tmp = tempfile.gettempdir()
for platform in ("MAC", "WINDOWS", None):
with mock.patch.object(frame_host, "MAC", platform == "MAC"), \
mock.patch.object(frame_host, "WINDOWS", platform == "WINDOWS"):
self.assertTrue(inside(frame_host.cache_dir("apk-sources"), tmp), platform)
for name in ("LOCALAPPDATA", "APPDATA", "XDG_CACHE_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME"):
self.assertTrue(inside(os.environ[name], tmp), name)
def test_end_to_end_tests_stay_out_of_it(self):
"""tests/e2e keeps the container's real home (OpenSSH reads ~/.ssh from the password
database, not $HOME), so nothing there may import sandbox, even through a test_* module."""
here = Path(__file__).resolve().parent
pattern = re.compile(r"^\s*(import sandbox|from sandbox |import test_|from test_)", re.M)
for f in [*(here / "e2e").glob("*.py"), *(here / "smoke").glob("*.py"), here / "apk_fixtures.py"]:
self.assertIsNone(pattern.search(f.read_text(encoding="utf-8")), f)
def test_end_to_end_tests_start_no_server_outside_their_container(self):
"""Plain discovery over tests/e2e, with or without FRAME_E2E=1, must skip everything
and never start ui/server.py: outside scripts/e2e.sh's container it would run on
this computer's real app data, ~/.ssh and cache."""
e2e = str(Path(__file__).resolve().parent / "e2e")
code = ("import io, sys, unittest; sys.path.insert(0, sys.argv[1]); import harness; calls = []\n"
"harness.Server.start = classmethod(lambda cls: calls.append(cls))\n"
"suite = unittest.defaultTestLoader.discover(sys.argv[1], top_level_dir=sys.argv[1])\n"
"r = unittest.TextTestRunner(stream=io.StringIO()).run(suite)\n"
"print(len(calls), r.testsRun, len(r.skipped), len(r.errors) + len(r.failures))")
for extra in ({}, {"FRAME_E2E": "1"}):
env = {k: v for k, v in os.environ.items() if not k.startswith("FRAME_E2E")}
env.update(extra, PYTHONDONTWRITEBYTECODE="1")
out = subprocess.run([sys.executable, "-c", code, e2e], env=env, capture_output=True, text=True,
timeout=120)
self.assertEqual(out.returncode, 0, out.stderr[-2000:])
started, ran, skipped, broken = map(int, out.stdout.split())
self.assertEqual((started, broken), (0, 0), (extra, out.stdout))
self.assertEqual(ran, skipped, (extra, out.stdout))
self.assertGreater(skipped, 0, extra)
def test_child_processes_inherit_it(self):
out = subprocess.run(
[sys.executable, "-c", "import sys; sys.path.insert(0, sys.argv[1]); import frame_devices, frame_connect;"
" from pathlib import Path; print(Path.home()); print(frame_devices.ssh_dir()); print(frame_connect.SSH_DIR)",
str(ROOT / "ui")], env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"},
capture_output=True, text=True, check=True).stdout.split("\n")
for d in filter(None, out):
self.assertFalse(inside(d, sandbox.REAL_SSH_DIR), d)
self.assertTrue(inside(d, tempfile.gettempdir()), d)
def test_real_ssh_folder_untouched_by_tests_that_start_servers(self):
"""Reruns, from the environment the suite was started in, the tests that used to
leak (servers started without FRAME_CONTROL_SSH_DIR) plus the device and connect
tests, and checks nothing under the real ~/.ssh changed."""
real = sandbox.REAL_SSH_DIR
if inside(real, tempfile.gettempdir()):
self.skipTest("this run's HOME is already a temporary folder")
before = snapshot(real)
env = {k: v for k, v in sandbox.ORIGINAL_ENV.items() if k != "FRAME_CONTROL_SSH_DIR"}
env["PYTHONDONTWRITEBYTECODE"] = "1"
r = subprocess.run([sys.executable, "-m", "unittest", "-q", "test_server.OneServer", "test_server.LocalMode",
"test_devices", "test_connect"],
cwd=str(Path(__file__).resolve().parent), env=env, capture_output=True, text=True,
timeout=300)
self.assertEqual(r.returncode, 0, r.stderr[-3000:])
after = snapshot(real)
changed = sorted(k for k in set(before) | set(after) if before.get(k) != after.get(k)
and not k.startswith("agent")) # ssh-agent sockets come and go on their own
self.assertEqual(changed, [], f"tests changed the real {real}")
if __name__ == "__main__":
unittest.main()
+5 -1
View File
@@ -535,7 +535,11 @@ class UnreachableMessages(unittest.TestCase):
"ssh: connect to host 192.168.1.9 port 22: No route to host": "isn't answering",
"ssh: connect to host 192.168.1.9 port 22: Connection refused": "refused",
"steamos@192.168.1.9: Permission denied (publickey,password).": "SSH key",
"Host key verification failed.": "identity changed",
"Host key verification failed.": "identity has changed",
"@@@@\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\nHost key verification failed.":
"Check the new key on the Devices tab",
"No ED25519 host key is known for frame-control-f7b4e9ac and you have requested strict checking.\n"
"Host key verification failed.": "isn't saved yet",
"kex_exchange_identification: read: Connection reset by peer": "dropped",
"Timed out talking to frame": "too long",
}
+1
View File
@@ -1,3 +1,4 @@
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import json
from pathlib import Path
import sys
+20
View File
@@ -335,6 +335,26 @@ class LinkFailureProvenance(Base):
tm.diagnostic("job steam", e)
self.assertEqual([p["properties"]["error_category"] for p in self.queued()], ["frame_unreachable"])
def test_host_key_and_login_refusals_are_sent_once(self):
# 0.4.2: each of these came in as a $exception from the status poll, again every 10 minutes.
for _ in range(3):
for stderr in ("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n"
"@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n"
"IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nHost key verification failed.\r\n",
"No ED25519 host key is known for frame-control-f7b4e9ac and you have requested strict "
"checking.\r\nHost key verification failed.\r\n",
"jane doe@frame-control-f7b4e9ac: Permission denied (publickey,password).\r\n",
"steamos@192.168.1.9: Permission denied (publickey,password).\n"):
e = self.ssh_fails(self.server, 255, stderr)
self.assertTrue(getattr(e, "frame_link_failed", False), stderr)
tm.diagnostic("POST /api/comfort status", e)
self.assertEqual([p["properties"]["error_category"] for p in self.queued()], ["frame_auth"])
# A command on the Frame that was refused something isn't ssh refusing the headset.
for code, stderr in ((1, "steamos@frame: Permission denied (publickey).\n"),
(255, "cat: /root/x: Permission denied\n"),
(255, "scp: /home/steamos/x: Permission denied\n")):
self.assertFalse(getattr(self.ssh_fails(self.server, code, stderr), "frame_link_failed", False), stderr)
def test_a_command_that_fails_on_the_frame_is_not_marked(self):
self.poll_fails()
for module in (self.server, self.android):
+1
View File
@@ -2,6 +2,7 @@
Run: python3 -m unittest discover -s tests
"""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import io
import json
import os
+1
View File
@@ -1,4 +1,5 @@
"""Own VR telemetry, optional catalogue and entitlement guards; no headset needed."""
import sandbox # noqa: F401 (first: keeps tests off real data and services)
import ctypes
import json
from pathlib import Path
+3 -1
View File
@@ -32,7 +32,9 @@ Entry dict (missing facts are None, never guessed)
Rules
-----
- Only sources that distribute APKs with the developer's consent: free listings,
never paid apps re-hosted, no licence or entitlement workarounds.
never paid apps re-hosted, no licence or entitlement workarounds. The one paid
exception is a store's own download of a game the user bought there: a source
marked 'owned': True whose entries carry 'owned': True (itch.io purchases).
- Honour each site's terms and robots rules; if automated download isn't allowed,
return entries with 'downloadable': False and a 'page' link instead.
- Cache indexes under frame_host.cache_dir('apk-sources'); send a clear
+394
View File
@@ -0,0 +1,394 @@
"""The user's own itch.io account: an API key, the games it owns and their Android uploads.
Opt-in (store > sources > Connect itch.io). The key is kept in
frame_host.data_dir('itch-account.json') with mode 0600 and is only ever sent to
https://api.itch.io, in an Authorization header. Downloads end on itch.io's storage
host, fetched without the key from a checked public address (_web.public_apk).
The key never goes into a URL, an error, a log line or an API response. Owned-key
listings stay in memory, never in the disk cache.
API: https://itch.io/docs/api/serverside. See docs/apk-repos.md.
"""
import hashlib, http.client, json, os, re, tempfile, threading, time, urllib.error, urllib.parse, urllib.request, zipfile
import frame_host
from . import SourceError, SourceLimited, _web
API = 'https://api.itch.io'
HOSTS = ('api.itch.io',)
NAME = 'itch.io'
TTL = 600 # owned keys and uploads are re-read after ten minutes
PER_PAGE = 500 # the documented maximum for /profile/owned-keys
PAGES = 40 # stop after 20,000 owned keys
KEY = re.compile(r'[A-Za-z0-9_.-]{16,256}')
BAD_KEY = ('itch.io did not accept the API key. Create one in itch.io settings > '
'API keys and connect again')
REVOKED = ('itch.io no longer accepts the saved API key, so it was removed. '
'Create a new one in itch.io settings > API keys and connect again')
REDIRECTS = (301, 302, 303, 307, 308)
# Everything a connection or response can raise. http.client's errors (IncompleteRead,
# BadStatusLine...) are not OSErrors, and BadStatusLine repeats what the server sent.
NETWORK = (OSError, ValueError, http.client.HTTPException)
_lock = threading.Lock() # the cache
_cache = {} # (generation, key digest, what) -> (time, value)
_account = threading.Lock() # saving and deleting the key file
_generation = [0] # bumped by every save and delete: a connect that started earlier doesn't save
class BadKey(SourceError):
"""itch.io refused the key: wrong, revoked or deleted."""
def path():
return frame_host.data_dir('itch-account.json')
def _load():
try:
data = json.loads(path().read_text())
except (OSError, ValueError):
return None
if not isinstance(data, dict) or not isinstance(data.get('key'), str) or not KEY.fullmatch(data['key']):
return None
return data
def status():
"""{'connected', 'username'}: what the page may know. Never the key; no network."""
data = _load()
return {'connected': bool(data), 'username': data.get('username') if data else None}
def connected():
return _load() is not None
def _key():
data = _load()
if not data:
raise SourceError('Connect your itch.io account first (store > sources > Connect itch.io)')
return data['key']
def _save(key, username):
"""Write the key file through a fresh 0600 temporary file (mkstemp), then move it into place."""
target = path()
target.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(target.parent), prefix='.itch-account-', suffix='.tmp')
try:
with os.fdopen(fd, 'w') as f:
os.chmod(tmp, 0o600) # mkstemp already does on POSIX; say so for every platform
json.dump({'key': key, 'username': username}, f)
os.replace(tmp, str(target))
finally:
if os.path.exists(tmp):
os.remove(tmp)
def _forget():
with _lock:
_cache.clear()
def _clean(message, key):
"""message without the key, or any start of it (8+ characters, as truncation leaves), in
any letter case, with any of its characters written literally, percent-encoded (%4B or
%4b) or percent-encoded twice (%254B, %254b). The only redaction used here: every message
that might hold the key goes through it. (The key's characters are letters, digits and
_.-, so +/space encoding can't change it.)"""
if not key:
return message
# Each character: itself, or the percent-encoding (once or twice) of either of its letter cases.
tokens = ['(?:%s)' % '|'.join([re.escape(c)] + ['%%(?:25)?%02x' % ord(v) for v in sorted({c.lower(), c.upper()})])
for c in key]
start, rest = re.compile(''.join(tokens[:8]), re.I), [re.compile(t, re.I) for t in tokens[8:]]
spans, pos = [], 0
while True:
found = start.search(message, pos)
if not found:
break
end = found.end()
for token in rest: # as much more of the key as follows
more = token.match(message, end)
if not more:
break
end = more.end()
spans.append((found.start(), end))
# Look again from the next character, not the end: a partial match just before the
# real key can overlap its start. Overlapping spans merge below.
pos = found.start() + 1
out, pos = [], 0
for a, b in spans: # in order of start; only overlapping spans merge
if out and a < pos:
pos = max(pos, b)
continue
out += [message[pos:a], '[API key]']
pos = b
return ''.join(out) + message[pos:]
class _Stay(urllib.request.HTTPRedirectHandler):
"""Don't follow: the caller decides where a redirect may go, and never sends the key there."""
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
def _open(url, key, accept='application/json'):
"""GET url on api.itch.io with the key. Redirects come back as HTTPError 30x."""
_web.checked_url(url, HOSTS)
wait = _web.wait_time(url)
if wait:
raise _web.limited_error(NAME, wait)
return urllib.request.build_opener(_Stay()).open(urllib.request.Request(
url, headers={'User-Agent': _web.UA, 'Authorization': 'Bearer ' + key, 'Accept': accept}), timeout=60)
def _refused(code, key, api_path, message=None):
"""A refusal that may be about the key (401/403, or an "invalid key" error): BadKey only
once /profile confirms the key itself is refused; otherwise it's this request (a game or
upload this purchase doesn't cover), and the key stays."""
if api_path != '/profile':
_get('/profile', key) # raises BadKey if the key is the problem
raise SourceError(message or 'itch.io refused this request (HTTP %d); your purchase may not include it' % code)
raise BadKey(BAD_KEY)
def _network_error(e, key):
reason = getattr(e, 'reason', None) or e
return SourceError(_clean('Could not reach itch.io: ' + str(reason), key))
BAD_KEY_TEXT = re.compile(r'\b(invalid|revoked|expired)\b.*\bkey\b|\bkey\b.*\b(invalid|revoked|expired)\b', re.I)
def _checked(raw, key, api_path):
"""Decoded JSON object from api.itch.io, or SourceError/BadKey from its errors array."""
if len(raw) > 8 * 1024 * 1024:
raise SourceError('itch.io response is too large')
try:
data = json.loads(raw)
except ValueError:
raise SourceError('itch.io returned an invalid response') from None
if not isinstance(data, dict):
raise SourceError('itch.io returned an invalid response')
errors = data.get('errors')
if errors: # documented: errors is an array of strings
messages = [str(m) for m in errors if isinstance(m, (str, int))]
# Judged on itch.io's own words: redacted text says "[API key]", which would look like a key error.
about_key = any(BAD_KEY_TEXT.search(m) and 'download key' not in m.lower() for m in messages)
# Each message loses the key before joining and shortening, so no part of it survives the cut.
text = _clean('; '.join(_clean(m, key) for m in messages)[:200], key)
if about_key:
_refused(None, key, api_path, 'itch.io: ' + text)
raise SourceError('itch.io: ' + (text or 'request refused'))
return data
def _get(api_path, key, params=None):
"""JSON from api.itch.io; SourceError with a readable message (never the key) on failure."""
url = API + api_path + ('?' + urllib.parse.urlencode(params) if params else '')
try:
with _open(url, key) as r:
raw = r.read(8 * 1024 * 1024 + 1)
except urllib.error.HTTPError as e:
e.close()
if e.code == 429:
raise _web.limited_error(NAME, _web.throttle(url, e.headers)) from None
if e.code in (401, 403):
_refused(e.code, key, api_path)
raise SourceError('itch.io returned HTTP error %d; try again later' % e.code) from None
except SourceError:
raise
except NETWORK as e:
raise _network_error(e, key) from None
return _checked(raw, key, api_path)
def _cached(key, what, load):
slot = (_generation[0], hashlib.sha256(key.encode()).hexdigest(), what)
with _lock:
hit = _cache.get(slot)
if hit and time.time() - hit[0] < TTL:
return hit[1]
value = load()
with _lock:
_cache[slot] = (time.time(), value)
return value
def _use(call):
"""Run call(key); a key itch.io now refuses is removed, so the account shows disconnected."""
with _account: # the key and the connection it belongs to, read together
key, generation = _key(), _generation[0]
try:
return call(key)
except BadKey:
with _account:
# Only the connection this call started with: a reconnect meanwhile (even with
# the same key) has been checked since, and is not undone by this older verdict.
if _generation[0] == generation and (_load() or {}).get('key') == key:
_delete()
raise BadKey(REVOKED) from None
def connect(key):
"""Check key with itch.io (GET /profile), then save it. A refused key is never saved, and
nor is one whose check finished after a disconnect or another connect."""
key = (key or '').strip() if isinstance(key, str) else ''
if not KEY.fullmatch(key):
raise BadKey('That does not look like an itch.io API key. Copy it from itch.io settings > API keys')
with _account:
started = _generation[0]
user = _get('/profile', key).get('user') # the network, outside the lock
if not isinstance(user, dict):
raise SourceError('itch.io returned an invalid profile')
username = user.get('username') if isinstance(user.get('username'), str) else None
with _account:
if _generation[0] != started:
raise SourceError('The itch.io connection changed while the key was being checked; '
'nothing was saved. Try again')
_generation[0] += 1
_save(key, username)
_forget()
return {'message': 'Connected to itch.io' + (' as ' + username if username else ''), 'itch': status()}
def _delete():
"""Remove the key file; call with _account held."""
_generation[0] += 1
try:
os.remove(str(path()))
except FileNotFoundError:
pass
_forget()
def disconnect():
with _account:
_delete()
return {'message': 'Disconnected from itch.io; the API key was deleted from this computer', 'itch': status()}
def owned_keys():
"""Every download key the account owns, newest first (GET /profile/owned-keys, all pages)."""
def load(key):
keys = []
for page in range(1, PAGES + 1):
batch = _get('/profile/owned-keys', key, {'page': page, 'per_page': PER_PAGE}).get('owned_keys')
# Documented: a page can hold fewer than per_page even when more follow; an empty one ends it.
if not batch or not isinstance(batch, list):
break
keys.extend(k for k in batch if isinstance(k, dict))
return keys
return _use(lambda key: _cached(key, 'owned', lambda: load(key)))
def _number(value):
return value if type(value) is int and value > 0 else None
def is_android(upload):
"""An upload marked for Android, or an .apk file. The marker's field name is not in the
public reference: itch.io's traits use p_<platform> (p_windows...), so p_android is assumed."""
traits = upload.get('traits') if isinstance(upload.get('traits'), list) else []
platforms = upload.get('platforms') if isinstance(upload.get('platforms'), dict) else {}
filename = str(upload.get('filename') or '').lower()
return 'p_android' in traits or upload.get('p_android') is True or 'android' in platforms \
or filename.endswith('.apk')
def android_uploads(uploads):
"""Installable Android uploads, best first: .apk files, then full games, then newest."""
chosen = []
for u in uploads if isinstance(uploads, list) else []:
if not isinstance(u, dict) or not _number(u.get('id')) or not is_android(u):
continue
traits = u.get('traits') if isinstance(u.get('traits'), list) else []
if u.get('type') not in (None, 'default') or u.get('preorder') is True or 'preorder' in traits:
continue # soundtracks, books, placeholders
if u.get('storage') == 'external':
continue # hosted on another site: not something itch.io can hand us
chosen.append(u)
chosen.sort(key=lambda u: str(u.get('updated_at') or u.get('created_at') or ''), reverse=True)
chosen.sort(key=lambda u: (not str(u.get('filename') or '').lower().endswith('.apk'),
u.get('demo') is True or 'demo' in (u.get('traits') or [])))
return chosen
def uploads(game_id, download_key_id):
"""The game's uploads this download key may fetch (GET /games/ID/uploads)."""
game_id, download_key_id = _number(game_id), _number(download_key_id)
if not game_id or not download_key_id:
raise SourceError('Invalid itch.io game')
return _use(lambda key: _cached(key, ('uploads', game_id, download_key_id), lambda: _get(
'/games/%d/uploads' % game_id, key, {'download_key_id': download_key_id}).get('uploads') or []))
def download_url(upload_id, download_key_id):
upload_id, download_key_id = _number(upload_id), _number(download_key_id)
if not upload_id or not download_key_id:
raise SourceError('Invalid itch.io upload')
return API + '/uploads/%d/download?' % upload_id + urllib.parse.urlencode({'download_key_id': download_key_id})
def _fetch(url, key):
"""Ask api.itch.io for the file. Not checked against a live account, so this accepts each
plausible answer: a redirect to storage, JSON naming the storage URL, or the file itself."""
try:
r = _open(url, key, 'application/vnd.android.package-archive, application/json;q=0.5, */*;q=0.1')
except urllib.error.HTTPError as e:
e.close()
if e.code in REDIRECTS:
target = e.headers.get('Location')
elif e.code == 429:
raise _web.limited_error(NAME, _web.throttle(url, e.headers)) from None
elif e.code in (401, 403):
_refused(e.code, key, '/uploads')
else:
raise SourceError('itch.io returned HTTP error %d for the download; try again later' % e.code) from None
except SourceError:
raise
except NETWORK as e:
raise _network_error(e, key) from None
else:
with r:
if 'json' not in (r.headers.get('Content-Type') or '').lower():
try:
return _web.save_apk(r) # served by api.itch.io itself
except NETWORK + (zipfile.BadZipFile,) as e:
raise SourceError(_clean('Could not download APK: ' + str(e), key)) from None
try:
raw = r.read(1024 * 1024 + 1)
except NETWORK as e:
raise _network_error(e, key) from None
target = _checked(raw, key, '/uploads').get('url')
if not isinstance(target, str) or not target:
raise SourceError('itch.io did not say where to download the file')
def refuse(hop): # on the first URL and on every redirect after it, before any lookup
if _carries_key(hop, key):
raise SourceError('itch.io pointed the download at a URL carrying your API key; refused, '
'so the key is not sent to the storage server')
try:
return _web.public_apk(urllib.parse.urljoin(url, target), NAME, check=refuse) # no key sent
except SourceLimited as e:
raise SourceLimited(_clean(str(e), key), e.retry_after) from None
except SourceError as e:
raise type(e)(_clean(str(e), key)) from None
def _carries_key(url, key):
"""True if url holds the key (or 8+ characters of its start), however cased or encoded, or
an itch.io API-key parameter: go-itchio puts the key in ?api_key=, and storage must never
get it. Other parameters (signatures, tokens) are the storage URL's own and are kept."""
for form in (url, urllib.parse.unquote(url), urllib.parse.unquote_plus(url)):
if _clean(form, key) != form:
return True
names = {name.lower() for name, _ in urllib.parse.parse_qsl(urllib.parse.urlsplit(url).query,
keep_blank_values=True)}
return bool(names & {'api_key', 'apikey', 'api-key'})
def download(upload_id, download_key_id):
"""The APK: the key goes to api.itch.io only, never to the storage host."""
url = download_url(upload_id, download_key_id)
return _use(lambda key: _fetch(url, key))
+161 -11
View File
@@ -1,5 +1,6 @@
"""Small HTTPS cache and APK downloader for public publisher sources."""
import hashlib, os, shutil, tempfile, threading, time, urllib.error, urllib.parse, urllib.request, zipfile
import hashlib, http.client, ipaddress, os, shutil, socket, ssl, tempfile, threading, time
import urllib.error, urllib.parse, urllib.request, zipfile
from email.utils import parsedate_to_datetime
import frame_host
@@ -136,6 +137,8 @@ def checked_url(url, hosts):
class Redirect(urllib.request.HTTPRedirectHandler):
"""Follow redirects only to hosts; an Authorization header never follows one."""
def __init__(self, hosts):
self.hosts = hosts
@@ -196,19 +199,16 @@ def read(url, hosts, headers=None, ttl=3600, name=None, hint=''):
raise SourceError('Could not read source: ' + str(e)) from e
def apk(url, hosts, digest=None, name=None):
def save_apk(response, digest=None):
"""Stream response (anything with read(n)) into the cache as a checked APK."""
tmp = None
name = name or _host(url)
wait = wait_time(url)
if wait:
raise limited_error(name, wait)
try:
fd, tmp = tempfile.mkstemp(dir=cache(), suffix='.part')
h = hashlib.sha256()
with os.fdopen(fd, 'wb') as f, open_url(url, hosts) as r:
with os.fdopen(fd, 'wb') as f:
size = 0
while True:
chunk = r.read(1 << 20)
chunk = response.read(1 << 20)
if not chunk:
break
size += len(chunk)
@@ -225,12 +225,162 @@ def apk(url, hosts, digest=None, name=None):
path = os.path.join(cache(), actual + '.apk')
os.replace(tmp, path)
return {'apk': path, 'obb': [], 'sha256': actual, 'verified': bool(digest)}
finally:
if tmp and os.path.exists(tmp):
os.remove(tmp)
def apk(url, hosts, digest=None, name=None):
name = name or _host(url)
wait = wait_time(url)
if wait:
raise limited_error(name, wait)
try:
with open_url(url, hosts) as r:
return save_apk(r, digest)
except urllib.error.HTTPError as e:
if e.code in (403, 429):
raise limited_error(name, throttle(url, e.headers)) from e
raise SourceError('Could not download APK: HTTP error ' + str(e.code)) from e
except (OSError, ValueError, zipfile.BadZipFile) as e:
raise SourceError('Could not download APK: ' + str(e)) from e
finally:
if tmp and os.path.exists(tmp):
os.remove(tmp)
# IPv6 is an allow-list: only global unicast (2000::/3), which rules out everything else at
# once (loopback, unspecified, mapped/compatible IPv4, NAT64 64:ff9b::, site-local fec0::,
# unique-local fc00::, link-local, multicast, reserved). Inside it, these special blocks are
# refused too: 6to4 and Teredo carry IPv4 that may be private; the rest aren't the internet.
_GLOBAL6 = ipaddress.ip_network('2000::/3')
_SPECIAL6 = tuple(ipaddress.ip_network(n) for n in (
'2001::/32', # Teredo
'2001:2::/48', # benchmarking
'2001:10::/28', # ORCHID
'2001:20::/28', # ORCHIDv2
'2001:db8::/32', # documentation
'2002::/16', # 6to4
'3fff::/20')) # documentation (2024)
# IPv4 blocks that are never the public internet, listed in full rather than trusting
# is_global (Python 3.9 calls 192.0.0.8, .11 and .255 global).
_SPECIAL4 = tuple(ipaddress.ip_network(n) for n in (
'0.0.0.0/8', '10.0.0.0/8', '100.64.0.0/10', '127.0.0.0/8', '169.254.0.0/16', '172.16.0.0/12',
'192.0.0.0/24', '192.0.2.0/24', '192.88.99.0/24', '192.168.0.0/16', '198.18.0.0/15',
'198.51.100.0/24', '203.0.113.0/24', '224.0.0.0/4', '240.0.0.0/4', '255.255.255.255/32'))
def public_address(ip):
"""True for an address on the public internet: never loopback, private, link-local,
site-local, shared (CGNAT), reserved, unspecified or multicast, nor an IPv6 wrapper
around IPv4. Python 3.9's is_global alone lets several of these through."""
try:
a = ipaddress.ip_address(ip.split('%', 1)[0])
except ValueError:
return False
if a.version == 6 and (a not in _GLOBAL6 or any(a in n for n in _SPECIAL6)):
return False
if a.version == 4 and any(a in n for n in _SPECIAL4):
return False
return a.is_global and not (a.is_reserved or a.is_private or a.is_loopback or a.is_link_local
or a.is_multicast or a.is_unspecified)
def resolve_public(host):
"""host's addresses for port 443, if every address it has is public; else SourceError."""
try:
addresses = socket.getaddrinfo(host, 443, type=socket.SOCK_STREAM)
except (OSError, UnicodeError) as e:
raise SourceError('Could not find the download server ' + host) from e
if not addresses or not all(public_address(a[4][0]) for a in addresses):
raise SourceError('The download server is on a private network address; refused')
found = []
for a in addresses:
if a[4][:2] not in found:
found.append(a[4][:2])
return found
CONNECT_BUDGET = 20 # seconds for connecting, all addresses and TLS handshakes together
def _public_connection(host, timeout, budget=CONNECT_BUDGET):
"""A verified TLS connection to host:443, made to an address checked to be public.
Each checked address is tried in turn, all within one deadline (budget, or timeout if
shorter) covering both connecting and the handshake. The one connected to is always one
that was checked, so a second lookup can't swap in another. timeout is for reads after."""
deadline = time.monotonic() + min(budget, timeout)
error = None
for address in resolve_public(host):
if time.monotonic() >= deadline:
break
try:
return _connect(host, address, timeout, deadline)
except OSError as e: # unreachable, refused, or its TLS failed: try the next one
error = e
raise error or socket.timeout('Timed out connecting to ' + host)
def _connect(host, address, timeout, deadline):
"""TLS to exactly address, verified as host, connected and handshaken before deadline
(a time.monotonic() value); reads then get timeout seconds each."""
def remaining():
left = deadline - time.monotonic()
if left <= 0:
raise socket.timeout('Timed out connecting to ' + host)
return left
sock = socket.create_connection(address, timeout=remaining())
try:
# Each step gets what is left of the deadline as its socket timeout. A handshake
# read as several receives could in theory run past it by one timeout; no thread
# is kept to cut it short, as one could fire after the connection was handed over.
sock.settimeout(remaining())
sock = ssl.create_default_context().wrap_socket(sock, server_hostname=host,
do_handshake_on_connect=False)
sock.settimeout(remaining())
sock.do_handshake()
sock.settimeout(timeout)
except BaseException:
sock.close()
raise
conn = http.client.HTTPConnection(host, 443, timeout=timeout)
conn.sock = sock
return conn
def public_apk(url, name, redirects=3, timeout=60, check=None):
"""Download an APK from a public HTTPS host that no source lists in advance (itch.io's
storage). No credentials are sent. Every hop, redirects included, is re-checked, and
check(url), if given, runs on each hop before any lookup or connection (it raises)."""
for _ in range(redirects + 1):
if check:
check(url)
try:
p = urllib.parse.urlsplit(url)
port = p.port
except (ValueError, TypeError) as e:
raise SourceError(name + ' returned an invalid download URL') from e
if p.scheme != 'https' or not p.hostname or p.username or p.password or port not in (None, 443):
raise SourceError(name + ' returned an unexpected download URL')
wait = wait_time(url)
if wait:
raise limited_error(name, wait)
try:
conn = _public_connection(p.hostname, timeout)
except OSError as e:
raise SourceError('Could not download APK: ' + str(e)) from e
try:
conn.request('GET', (p.path or '/') + ('?' + p.query if p.query else ''), headers={'User-Agent': UA})
r = conn.getresponse()
if r.status in (301, 302, 303, 307, 308):
url = urllib.parse.urljoin(url, r.getheader('Location') or '')
continue
if r.status == 429:
raise limited_error(name, throttle(url, r.headers))
if r.status != 200:
raise SourceError("%s's download server refused the file (HTTP %d); try again later"
% (name, r.status))
return save_apk(r)
except (OSError, ValueError, http.client.HTTPException, zipfile.BadZipFile) as e:
raise SourceError('Could not download APK: ' + str(e)) from e
finally:
conn.close()
raise SourceError(name + ' redirected the download too many times')
+97 -15
View File
@@ -1,17 +1,35 @@
"""Free Android VR RSS listings. Download pages stay in the publisher's UI."""
"""Free Android VR RSS listings, plus the Android builds of games the user owns.
The free feeds' download pages stay in the publisher's UI. Owned games appear only
after the user connects their itch.io account with an API key (_itch_account.py);
they download through itch.io's own API, with the download key of that purchase.
"""
import html, re, urllib.parse, xml.etree.ElementTree as ET
from email.utils import parsedate_to_datetime
from . import SourceError, _web
from . import SourceError, _itch_account, _web
KIND = 'itch'
FEEDS = ('openxr', 'oculus-quest')
OWNED = 'itch-owned'
OWNED_LABEL = 'Owned on itch.io'
def sources():
return [{'id': KIND if tag == 'openxr' else 'itch-quest', 'kind': KIND,
'name': 'itch.io (' + tag + ')', 'url': 'https://itch.io', 'tag': tag,
'builtin': True, 'enabled': True, 'trust': 'community'} for tag in FEEDS]
found = [{'id': KIND if tag == 'openxr' else 'itch-quest', 'kind': KIND,
'name': 'itch.io (' + tag + ')', 'url': 'https://itch.io', 'tag': tag,
'builtin': True, 'enabled': True, 'trust': 'community'} for tag in FEEDS]
if _itch_account.connected():
found.append({'id': OWNED, 'kind': KIND, 'name': 'itch.io (my purchases)',
'url': 'https://itch.io/my-purchases', 'owned': True,
'description': 'Android builds of games you own on itch.io',
'builtin': True, 'enabled': True, 'trust': 'user'})
return found
def _itch_page(url):
p = urllib.parse.urlsplit(url if isinstance(url, str) else '')
return p.scheme == 'https' and (p.hostname or '').endswith('.itch.io') and not p.username and ':' not in p.netloc
def _parse(source, data):
@@ -22,8 +40,7 @@ def _parse(source, data):
entries = []
for item in root.findall('./channel/item'):
page = item.findtext('link') or ''
p = urllib.parse.urlsplit(page)
if p.scheme != 'https' or not (p.hostname or '').endswith('.itch.io') or p.username or ':' in p.netloc:
if not _itch_page(page):
continue
if item.findtext('price') != '$0.00' or item.findtext('platforms/android') != 'yes':
continue
@@ -45,23 +62,81 @@ def _parse(source, data):
return entries
def _owned_entry(source, key):
"""Entry for one owned download key, or None if its game isn't for Android."""
game = key.get('game') if isinstance(key.get('game'), dict) else {}
game_id = key.get('game_id') or game.get('id')
if type(game_id) is not int or type(key.get('id')) is not int:
return None
traits = game.get('traits')
if isinstance(traits, list) and 'p_android' not in traits:
return None # the game's page offers no Android download
cover = game.get('cover_url') if isinstance(game.get('cover_url'), str) else None
if cover and not cover.startswith('https://img.itch.zone/'):
cover = None
user = game.get('user') if isinstance(game.get('user'), dict) else {}
price = game.get('min_price')
return {'source': source['id'], 'id': str(game_id), 'package': None,
'name': str(game.get('title') or 'itch.io game ' + str(game_id)),
'summary': str(game.get('short_text') or OWNED_LABEL),
'developer': user.get('display_name') or user.get('username'),
'icon': cover, 'images': {'icon': cover, 'banner': cover, 'screenshots': []},
'page': game['url'] if _itch_page(game.get('url')) else None,
'version': None, 'version_code': None, 'min_sdk': None, 'abis': None,
'vr': None, 'size': None, 'free': price == 0 if type(price) is int else None,
'owned': True, 'owned_label': OWNED_LABEL, 'license': None,
'updated': str(game.get('published_at') or '')[:10] or None,
'downloadable': None, 'download_key_id': key['id']}
def _owned(source):
entries = {}
for key in _itch_account.owned_keys():
entry = _owned_entry(source, key)
if entry:
entries.setdefault(entry['id'], entry) # keys come newest first; one entry per game
return entries
def search(source, query, limit=50):
limit = max(0, min(int(limit), 100))
if not limit:
return []
entries = {}
tag = source.get('tag', 'openxr')
if tag not in FEEDS:
raise SourceError('Unsupported itch.io feed')
url = 'https://itch.io/games/free/platform-android/tag-' + tag + '.xml'
for entry in _parse(source, _web.read(url, ('itch.io',), name='itch.io')):
entries.setdefault(entry['id'], entry)
if source.get('owned'):
entries = _owned(source)
else:
entries = {}
tag = source.get('tag', 'openxr')
if tag not in FEEDS:
raise SourceError('Unsupported itch.io feed')
url = 'https://itch.io/games/free/platform-android/tag-' + tag + '.xml'
for entry in _parse(source, _web.read(url, ('itch.io',), name='itch.io')):
entries.setdefault(entry['id'], entry)
words = query.lower().split()
return [e for e in entries.values() if all(w in (e['name'] + ' ' + e['summary']).lower()
for w in words)][:limit]
def _owned_details(source, entry_id):
entry = _owned(source).get(str(entry_id))
if not entry:
raise SourceError('This game is not among the Android games you own on itch.io')
uploads = _itch_account.uploads(int(entry['id']), entry['download_key_id'])
versions = [{'version': str(u.get('display_name') or u.get('filename') or u['id']), 'version_code': None,
'upload_id': u['id'], 'name': u.get('filename'), 'min_sdk': None,
'demo': u.get('demo') is True or 'demo' in (u.get('traits') or []),
'size': u.get('size') if type(u.get('size')) is int else None,
'updated': str(u.get('updated_at') or u.get('created_at') or '')[:10] or None}
for u in _itch_account.android_uploads(uploads)]
entry = dict(entry, versions=versions, downloadable=bool(versions))
if versions:
entry.update({k: versions[0][k] for k in ('version', 'size', 'updated')})
return entry
def details(source, entry_id):
if source.get('owned'):
return _owned_details(source, entry_id)
entry = next((e for e in search(source, '', 100) if e['id'] == entry_id), None)
if not entry:
raise SourceError('Game is not in the current free Android VR feed; open its publisher page')
@@ -69,4 +144,11 @@ def details(source, entry_id):
def download(source, entry_id, version_code=None):
raise SourceError('Open the itch.io publisher page to download; automated download pages are disallowed by robots rules')
if not source.get('owned'):
raise SourceError('Open the itch.io publisher page to download; automated download pages are disallowed by robots rules')
if version_code is not None:
raise SourceError('itch.io does not publish Android version codes; download the latest upload')
entry = _owned_details(source, entry_id)
if not entry['versions']:
raise SourceError('This game has no Android download on itch.io')
return _itch_account.download(entry['versions'][0]['upload_id'], entry['download_key_id'])
+11 -3
View File
@@ -182,7 +182,10 @@ def group(entries, query='', vr=None, installable=False):
groups = {}
for entry in entries:
entry = decorate(entry)
if vr is not None and (entry.get('vr') is True) != vr: # unknown counts as flat
# Unknown counts as flat, except the user's own purchases (itch.io doesn't say which are
# VR): those show under either choice rather than vanish from one.
if vr is not None and (entry.get('vr') is True) != vr \
and not (entry.get('owned') is True and entry.get('vr') is None):
continue
if installable and entry['fit']['installable'] is not True:
continue
@@ -215,6 +218,11 @@ def group(entries, query='', vr=None, installable=False):
return result
def obtainable(source, entry):
"""Free, or a game the user owns, listed by their own account's source (itch.io purchases)."""
return entry.get('free') is True or (source.get('owned') is True and entry.get('owned') is True)
def _launch(module, source, query, limit):
"""One search per source at a time; the newest different query runs next."""
key = source['id']
@@ -243,7 +251,7 @@ def _start(key, task):
queued = None
try:
task['entries'] = [dict(e, source=key, source_name=source['name'], trust=source.get('trust'))
for e in module.search(source, query, limit=limit) if e.get('free') is True]
for e in module.search(source, query, limit=limit) if obtainable(source, e)]
task['stale'] = bool(getattr(module, 'stale', lambda s: False)(source))
except Exception as e:
task['error'] = str(e)
@@ -300,7 +308,7 @@ def install(source_id, entry_id, version_code=None, progress=None):
if not source['enabled']:
raise SourceError('This source is disabled')
entry = module.details(source, entry_id)
if entry.get('free') is not True or entry.get('downloadable') is not True:
if not obtainable(source, entry) or entry.get('downloadable') is not True:
raise SourceError('This app must be obtained from its developer page')
if progress:
progress('Downloading', None)
+2 -1
View File
@@ -31,7 +31,8 @@ import frame_host # noqa: E402
FRAME_USER = os.environ.get("FRAME_USER", "steamos")
USER_FROM_ENV = "FRAME_USER" in os.environ
FRAME_ALIAS = os.environ.get("FRAME_ALIAS", "frame")
SSH_DIR = Path.home() / ".ssh"
# $FRAME_CONTROL_SSH_DIR in tests, as frame_devices.ssh_dir(), so they never touch the real one.
SSH_DIR = Path(os.environ.get("FRAME_CONTROL_SSH_DIR") or Path.home() / ".ssh")
KEY = SSH_DIR / "id_ed25519_frame"
# The devkit service only accepts ssh-rsa keys (write_key in Valve's
# steamos-devkit-service), so pairing uses a second key next to the ed25519 one.
+118 -12
View File
@@ -28,8 +28,10 @@ remembered IP is caught.
Python stdlib only.
"""
import base64
import contextlib
import copy
import hashlib
import json
import os
import re
@@ -418,19 +420,100 @@ def seed_pin(device_id, hosts, port=22, sources=None):
if len(f) >= 3 and not line.startswith("#") and not f[0].startswith("@"):
keys.append(f"{name} {f[1]} {f[2]}")
if keys:
target = known_hosts(device_id)
target.parent.mkdir(**({} if frame_host.WINDOWS else {"mode": 0o700}), parents=True, exist_ok=True)
fd_, tmp = tempfile.mkstemp(prefix=".seed-", dir=str(target.parent))
with os.fdopen(fd_, "w", encoding="utf-8") as fh:
fh.write("\n".join(dict.fromkeys(keys)) + "\n")
os.replace(tmp, target) # whole file at once: ssh never sees half of it
_write_pin(device_id, keys)
return True
return False
def pin_dir():
"""Where the per-headset files go, created private: ssh won't create it itself."""
d = ssh_dir() / PIN_DIR
d.mkdir(**({} if frame_host.WINDOWS else {"mode": 0o700}), parents=True, exist_ok=True)
return d
def _write_pin(device_id, lines):
target = known_hosts(device_id)
fd_, tmp = tempfile.mkstemp(prefix=".seed-", dir=str(pin_dir()))
with os.fdopen(fd_, "w", encoding="utf-8") as fh:
fh.write("\n".join(dict.fromkeys(lines)) + "\n")
frame_host.make_private(tmp) # Windows: this user only, not the folder's inherited ACL
os.replace(tmp, target) # whole file at once: ssh never sees half of it
# The host key types OpenSSH writes to known_hosts.
KEY_TYPE_RE = re.compile(r"ssh-ed25519|ssh-rsa|ssh-dss|ecdsa-sha2-nistp(?:256|384|521)|"
r"sk-ssh-ed25519@openssh\.com|sk-ecdsa-sha2-nistp256@openssh\.com")
def fingerprint(blob):
"""OpenSSH's SHA256 fingerprint of a base64 key blob (what `ssh-keygen -l` and ssh -v print),
or None if it isn't base64."""
try:
raw = base64.b64decode(blob, validate=True)
except (ValueError, TypeError):
return None
return "SHA256:" + base64.b64encode(hashlib.sha256(raw).digest()).decode().rstrip("=")
def key_entry(line, name):
"""(type, blob, fingerprint) from a known_hosts line for `name`, or None."""
f = (line or "").split()
if len(f) < 3 or line.startswith("#") or name not in f[0].split(",") or not KEY_TYPE_RE.fullmatch(f[1]):
return None
fp = fingerprint(f[2])
return (f[1], f[2], fp) if fp else None
def pinned_keys(device_id):
"""[(type, fingerprint)] of the keys saved for a headset (plain entries only)."""
try:
lines = known_hosts(device_id).read_text(encoding="utf-8").splitlines()
except (OSError, UnicodeDecodeError):
return []
name = host_key_alias(device_id)
return [(e[0], e[2]) for e in (key_entry(line, name) for line in lines) if e]
def pin_key(device_id, key_type, blob):
"""Save one key as the headset's whole identity, replacing whatever was saved: what the
person confirmed after seeing its fingerprint (see frame_link.Link.trust_identity)."""
if not KEY_TYPE_RE.fullmatch(key_type or "") or not fingerprint(blob):
raise DeviceError("That isn't an SSH host key")
_write_pin(device_id, [f"{host_key_alias(device_id)} {key_type} {blob}"])
def scan_file(device_id):
"""(path, how ssh is told) for a throwaway known_hosts file that a key check writes the
headset's key to, without touching the saved one. `~` as in known_hosts_opt."""
name = f".scan-{device_id}-{secrets.token_hex(4)}"
path = pin_dir() / name
path.touch(**({} if frame_host.WINDOWS else {"mode": 0o600}))
frame_host.make_private(path)
return path, (str(path) if os.environ.get("FRAME_CONTROL_SSH_DIR") else f"~/.ssh/{PIN_DIR}/{name}")
def identity_established(device):
"""Whether this headset ever had an SSH identity here: the device's own flag (set once it
had a key or a connection that worked, and never cleared, whatever happens to its
addresses), a fingerprint recorded in devices.json, a pin file at all (even empty,
unreadable or not parseable), or an address that worked. Then a missing or unusable pin
fails closed, needing Check its key; only a headset with none of these may save the
first key it meets (accept-new)."""
if device.get("identity_established") or device.get("identity"):
return True
try:
if os.path.lexists(known_hosts(device["id"])):
return True
except OSError:
return True
return any(a.get("last_ok") for a in device.get("addresses") or ())
def forget_pin(device_id):
"""Drop a headset's saved key, e.g. after SteamOS was reinstalled. The next connection
trusts whatever key the headset shows, as a first connection does."""
"""Drop a headset's saved key (the headset was removed, or the API's forget-identity).
The next connection trusts whatever key the headset shows, as a first connection does;
the page's Trust a new key flow (frame_link) shows the key first instead."""
try:
known_hosts(device_id).unlink()
return True
@@ -510,6 +593,10 @@ class Registry:
for a in d["addresses"]:
a.setdefault("networks", [])
a.setdefault("label", "")
if d.get("identity") or any(a.get("last_ok") for a in d["addresses"]):
# Files from before the flag existed: it had an identity (or a connection that
# worked, which saved one). Kept from now on, whatever happens to its addresses.
d["identity_established"] = True
return NAME_RE.fullmatch(d.get("id", "")) is not None
except (KeyError, TypeError, DeviceError):
return False
@@ -712,8 +799,25 @@ class Registry:
a["networks"] = (a["networks"] + [network_id])[-16:]
a["last_ok"] = time.time()
a["last_rtt_ms"] = rtt_ms
self._find(device_id)["identity_established"] = True # survives editing the address away
self.save()
def record_identity(self, device_id, fingerprints):
"""Remember the fingerprints of the key saved for a headset, so that losing the pin
file later can't turn into trusting whatever answers (identity_established)."""
fingerprints = [f for f in fingerprints if isinstance(f, str) and f.startswith("SHA256:")][:8]
if not fingerprints:
return
with self._changing():
try:
d = self._find(device_id)
except DeviceError:
return
if d.get("identity") != fingerprints or not d.get("identity_established"):
d["identity"] = fingerprints
d["identity_established"] = True
self.save()
def undismiss(self, alias):
"""Set Up Connection is about to run for this alias: import its block again."""
with self._changing():
@@ -797,15 +901,17 @@ class Registry:
d["addresses"].append(dict(new_address(host), label="From Set Up Connection"))
d["config_host"] = host
changed = True
if seed and host:
seed_pin(d["id"], [host], b["port"])
if seed and host and seed_pin(d["id"], [host], b["port"]):
d["identity"] = [f for _, f in pinned_keys(d["id"])][:8] or d.get("identity")
d["identity_established"] = True
elif host and host != d.get("config_host"):
# Set Up Connection ran again and found the headset somewhere new.
d["config_host"] = host
if not any(a["host"] == host for a in d["addresses"]):
d["addresses"].insert(0, dict(new_address(host), label="From Set Up Connection"))
if seed:
seed_pin(d["id"], [host], b["port"])
if seed and seed_pin(d["id"], [host], b["port"]) and not d.get("identity"):
d["identity"] = [f for _, f in pinned_keys(d["id"])][:8] or None
d["identity_established"] = True
changed = True
if d.get("config_login") != [user, b["port"]]:
# Set Up Connection (or an edit) changed who to log in as, or the port.
+5 -1
View File
@@ -38,9 +38,13 @@ class HostError(RuntimeError):
# The start of a line in which ssh itself says the link to the headset failed (not the command it ran).
# That includes ssh refusing the headset's host key, and the headset refusing this computer's
# key ("user@host: Permission denied (publickey,...)."): either way no command ran.
SSH_LINK_FAILED = re.compile(r"^(?:ssh: connect to host |ssh: Could not resolve hostname |banner exchange: |"
r"kex_exchange_identification: |mux_client_\w+: |client_loop: |"
r"Connection (?:closed|reset) by \S+ port \d+|Connection timed out during banner exchange)",
r"Connection (?:closed|reset) by \S+ port \d+|Connection timed out during banner exchange|"
r"Host key verification failed\.|No \S+ host key is known for |"
r"[^\n@:]{1,64}@\S+: Permission denied \()", # a Windows user name may have spaces
re.M)
+411 -33
View File
@@ -28,11 +28,14 @@ Python stdlib only. Runs on this computer, never on the Frame.
import copy
import hashlib
import ipaddress
import os
import queue
import re
import shlex
import socket
import subprocess
import sys
import tempfile
import threading
import time
@@ -50,6 +53,7 @@ NETWORK_EVERY = 5 # how often the network fingerprint is read
TAILSCALE_EVERY = 30
RETRY = (5, 10, 20, 30) # seconds before automatic retries after a failure
REQUEST_GAP = 5 # a request may start a new attempt this long after the last one
SCAN_TTL = 600 # a key shown for "Trust a new key" may be trusted this long after it was read
STAGES = [("network", "Checking this computer's network"), ("find", "Finding the headset"),
("ssh", "Opening SSH"), ("identity", "Checking the headset's identity"),
@@ -90,6 +94,32 @@ def ssh_g(alias):
return got.get("hostname") or alias, port, got.get("user"), proxied
def proxy_route(alias):
"""How ~/.ssh/config sends ALIAS: (None, None) directly, ("jump", spec) through
ProxyJump, or ("command", text) through a ProxyCommand."""
try:
out = frame_host.run_ssh(["ssh", "-G", alias], capture_output=True, stdin=subprocess.DEVNULL, text=True,
timeout=10).stdout
except (OSError, subprocess.TimeoutExpired):
out = ""
got = {}
for line in out.splitlines():
k, _, v = line.partition(" ")
if k in ("proxyjump", "proxycommand") and k not in got:
got[k] = v.strip()
if got.get("proxyjump") not in (None, "", "none"):
return "jump", got["proxyjump"]
if got.get("proxycommand") not in (None, "", "none"):
return "command", got["proxycommand"]
return None, None
# One jump host, as ProxyJump names it: [user@]host[:port], nothing that could start an
# option, split a shell word or carry an ssh % token.
JUMP_RE = re.compile(r"(?:([A-Za-z0-9][A-Za-z0-9._-]{0,63})@)?"
r"(\[[0-9A-Fa-f:.]{2,45}\]|[A-Za-z0-9][A-Za-z0-9.-]{0,252})(?::(\d{1,5}))?")
def probe(host, port, timeout=PROBE_TIMEOUT, update=None):
"""Try a TCP connection to host:port. -> {"state", "detail", "ip", "rtt_ms"}.
@@ -257,6 +287,43 @@ def ssh_target(host, ip):
return host
def _unlink(path):
try:
os.unlink(path)
except OSError:
pass
def _scanned_key(path, name):
"""The first whole, valid known_hosts line for `name` that a key check wrote, or None.
Only lines ssh has finished (ending in a newline): a half-written key mustn't count."""
try:
with open(path, encoding="utf-8") as fh:
text = fh.read()
except (OSError, UnicodeDecodeError):
return None
for line in text.splitlines(True):
if line.endswith("\n"):
entry = frame_devices.key_entry(line.strip(), name)
if entry:
return entry
return None
def proxy_command(literal):
"""A ProxyCommand that runs `literal` (argv, ending with the jump host) with `-W` to the
headset, each word exactly as given. ssh first expands its %-tokens in the line, so every
% of ours is doubled; then it hands the line to a shell (`$SHELL -c` on macOS and Linux)
or, on Windows, starts it as a command line, so every word is quoted for that: a path
with spaces, $(...), backticks or quotes stays one plain word. The forward is
[%h]:%p, so an IPv6 address (with a zone) reads as one host."""
words = [w.replace("%", "%%") for w in literal]
target = "[%h]:%p"
if frame_host.WINDOWS:
return subprocess.list2cmdline([*words[:-1], "-W", target, words[-1]])
return " ".join(shlex.quote(w) for w in [*words[:-1], "-W", target, words[-1]])
def probe_raw(host, port, result):
"""ssh's own wording for a failed probe, so the server's UNREACHABLE table explains it."""
return {"unresolved": f"ssh: Could not resolve hostname {host}: not found",
@@ -275,7 +342,8 @@ class Link:
self.session_alias = env_alias # ...and stays selectable after switching away
self.mux_base = list(mux_base) # ["ssh", "-o", "BatchMode=yes", ControlPath...]
self.control = control # ControlPath, or None where ssh can't share connections
self.apply = apply # apply(alias, host_opts): point every ssh command at the headset
self._apply = apply # apply(alias, host_opts): point every ssh command at the headset
self.applied = (None, []) # what it was last given
self.explain = explain # ssh error text -> plain reason, or None
self.cond = threading.Condition()
self.version = 0
@@ -301,6 +369,7 @@ class Link:
self.routed_device = None
self.last_failure = None # {"stage", "category", "at"}: for report diagnostics, no free text
self.attempt_device = None # the headset the current attempt is for
self.scans = {} # device id -> the host key it showed for "Trust a new key"
# ---- publishing ----
def publish(self, **fields):
@@ -505,13 +574,76 @@ class Link:
return [*mux, *(device.get("frozen") or [])] # what ~/.ssh/config said when it was routed
if not host: # a headset with no addresses: reach nothing, not whatever ~/.ssh/config says
return ["-o", "HostName=no-address.invalid"]
# StrictHostKeyChecking=yes: whatever ~/.ssh/config says for Host *, every command
# checks the headset's pinned key (only the connector's first handshake may save one).
return [*mux, "-o", "StrictHostKeyChecking=yes", "-o", f"HostName={frame_devices.ssh_host(host)}",
# Whatever ~/.ssh/config says for Host *, every command checks the headset's pinned key:
# StrictHostKeyChecking=yes. Only a headset that never had an identity here (just
# added, never connected) gets accept-new: the first contact saves its key, as Set Up
# Connection and the connector's first handshake do, rather than every command failing
# with "No host key is known" until the connector gets there. Once it had one, a pin
# file that's gone, empty or unreadable fails closed (identity_established): accept-new
# can't tell that from a new headset and would trust whatever answers.
strict = self.strictness(device)
if strict == "accept-new":
try:
frame_devices.pin_dir() # ssh won't create the folder its known_hosts file goes in
except OSError:
pass # ssh then says it couldn't save the key, and goes on
return [*mux, "-o", f"StrictHostKeyChecking={strict}", "-o", f"HostName={frame_devices.ssh_host(host)}",
"-o", f"HostKeyAlias={frame_devices.host_key_alias(device['id'])}",
"-o", f"UserKnownHostsFile={frame_devices.known_hosts_opt(device['id'])}", "-o", "HashKnownHosts=no",
"-o", f"User={device['user']}", "-o", f"Port={device['port']}"]
def apply(self, alias, opts):
self.applied = (alias, list(opts))
self._apply(alias, opts)
def strictness(self, device):
"""yes, or accept-new for a headset that never had an identity. Judged on what the
registry says now, not only the copy handed in (a route frozen at connect, say, from
before its first key was recorded). A key saved since is recorded on the way."""
try:
current = self.reg.get(device["id"])
except frame_devices.DeviceError:
current = {}
if not (frame_devices.identity_established(device) or
(current and frame_devices.identity_established(current))):
return "accept-new"
self.settle_pin(dict(device, **current) if current else device, tighten=False)
return "yes"
def settle_pin(self, device=None, tighten=True):
"""A key ssh saved on a first contact, whatever happened next (logged in or refused,
by the handshake or any command): record it, make the file private, and from then on
route every command strictly. Idempotent and cheap when there's nothing to do."""
device = device or self.routed_device
if not device or device.get("transient") or device.get("none") or not device.get("id"):
return
did = device["id"]
if frame_devices.pinned(did):
fps = [f for _, f in frame_devices.pinned_keys(did)]
if fps and (device.get("identity") != fps or not device.get("identity_established")):
try:
current = self.reg.get(did)
except frame_devices.DeviceError:
return
if current.get("identity") != fps or not current.get("identity_established"):
frame_host.make_private(frame_devices.known_hosts(did)) # ssh wrote it
self.reg.record_identity(did, fps)
if not tighten:
return
with self.route_lock:
alias, opts = self.applied
if alias == device.get("alias") and "StrictHostKeyChecking=accept-new" in opts and \
frame_devices.identity_established(dict(device, **self._current(did))):
self.opts = [("StrictHostKeyChecking=yes" if o == "StrictHostKeyChecking=accept-new" else o)
for o in opts]
self.apply(alias, self.opts)
def _current(self, device_id):
try:
return self.reg.get(device_id)
except frame_devices.DeviceError:
return {}
def public_device(self, d):
return {k: d.get(k) for k in ("id", "name", "alias", "user", "port", "transient")}
@@ -541,6 +673,8 @@ class Link:
if t - last_ts >= TAILSCALE_EVERY and self.state["network"] and not reasons:
last_ts = t
self.refresh_network()
if "StrictHostKeyChecking=accept-new" in self.applied[1]:
self.settle_pin() # a command's first contact may have saved the key
phase = self.state["phase"]
if phase == "connected" and not self.alive():
reasons.append("dropped")
@@ -675,10 +809,11 @@ class Link:
return "Starting up"
return "Connecting"
def fail(self, sid, message, raw=""):
def fail(self, sid, message, raw="", **more):
"""A stage failed. `more`: what the page can offer to fix it (hostkey, auth)."""
self.stage(sid, "failed", message)
with self.cond:
self.state["error"] = {"stage": sid, "message": message, "raw": raw}
self.state["error"] = dict({"stage": sid, "message": message, "raw": raw}, **more)
probes = copy.deepcopy(self.state["probes"])
self.note_failure(sid, message, raw, probes)
@@ -867,6 +1002,7 @@ class Link:
if device.get("transient"):
return
self.reg.record_success(device["id"], host, net.get("id"), rtt)
self.reg.record_identity(device["id"], [f for _, f in frame_devices.pinned_keys(device["id"])])
# Set Up Connection may have changed the block while this attempt ran: take that
# in (and reconnect) rather than writing this attempt's older settings over it.
self.watch_config()
@@ -921,7 +1057,17 @@ class Link:
def handshake(self, device, a, found, user):
"""SSH to one address, following ssh -v through stages 3-5.
-> "ok", "next" (try another address) or "stop"."""
-> "ok", "next" (try another address) or "stop". A key ssh saved is recorded and
protected however it ends: a login refused after the key was saved, a timeout, a stop."""
try:
return self._handshake(device, a, found, user)
finally:
if not device.get("transient"):
with self.route_lock:
current = self.attempt_gen == self.gen
self.settle_pin(device, tighten=current)
def _handshake(self, device, a, found, user):
# The IP that answered (with a link-local IPv6 address's zone), so ssh doesn't
# look the name up again and stall on an address that didn't answer.
opts = self.host_opts(device, ssh_target(a["host"], found.get("ip")))
@@ -939,15 +1085,9 @@ class Link:
return "ok"
extra = []
if not device.get("transient"):
if frame_devices.pinned(device["id"]):
extra = ["-o", "StrictHostKeyChecking=yes"]
else:
# First connection since this headset was added: trust what it shows
# (as Set Up Connection does), and pin it from now on. ssh won't create
# the folder its known_hosts file goes in.
extra = ["-o", "StrictHostKeyChecking=accept-new"]
pins = frame_devices.known_hosts(device["id"]).parent
pins.mkdir(**({} if frame_host.WINDOWS else {"mode": 0o700}), parents=True, exist_ok=True)
# A headset's first connection trusts what it shows (as Set Up Connection does) and
# pins it; any later one checks the pin strictly (see host_opts).
extra = ["-o", f"StrictHostKeyChecking={self.strictness(device)}"]
if self.control:
# No ConnectTimeout: with it, OpenSSH's master takes ~5s to open its socket.
# `extra` first: ssh takes the first value of an option, and it may say accept-new.
@@ -955,27 +1095,70 @@ class Link:
"-o", "ServerAliveCountMax=2", "-N", alias]
else:
argv = [*self.mux_base, *extra, *opts, "-v", "-o", "ConnectTimeout=10", alias, "true"]
lines = queue.Queue()
collecting = [True]
reader_done = threading.Event()
errfile = None
try:
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE, **frame_host.DETACHED)
if self.control:
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE, **frame_host.DETACHED)
else:
# No master (Windows): ssh runs `true` and exits. Its stderr goes to a file read
# as it grows, not a pipe: Windows' OpenSSH can block writing to a redirected
# pipe (see frame_host.run_ssh), and the handshake would never finish.
fd_, errfile = tempfile.mkstemp(prefix="frame-ssh-")
try:
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=fd_, **frame_host.DETACHED)
finally:
os.close(fd_)
except OSError as e:
if errfile:
_unlink(errfile)
self.fail("ssh", f"Couldn't run ssh: {e}", str(e))
return "stop"
self.pending = proc # so stop() can end it mid-handshake
lines = queue.Queue()
collecting = [True]
def read():
for raw in iter(proc.stderr.readline, b""):
if collecting[0]:
lines.put(raw.decode("utf-8", "replace").rstrip())
reader_done.set()
lines.put(None)
proc.stderr.close()
threading.Thread(target=read, daemon=True).start()
def tail():
buf = b""
try:
with open(errfile, "rb") as fh:
while True:
chunk = fh.read()
if not chunk:
if proc.poll() is None:
time.sleep(0.05)
continue
chunk = fh.read() # whatever it wrote just before exiting
if not chunk:
break
buf += chunk
*done, buf = buf.split(b"\n")
for raw in done:
lines.put(raw.decode("utf-8", "replace").rstrip())
except OSError:
pass
if buf:
lines.put(buf.decode("utf-8", "replace").rstrip())
reader_done.set()
lines.put(None)
_unlink(errfile)
threading.Thread(target=read if errfile is None else tail, daemon=True).start()
step, said, authed = "ssh", [], False
deadline = time.monotonic() + HANDSHAKE_TIMEOUT
mismatch = False
shown = None # (type, fingerprint) of the key the headset showed
exited = None # when ssh was first seen to have exited
while True:
left = deadline - time.monotonic()
if self.stopped:
@@ -993,7 +1176,14 @@ class Link:
if authed and self.control and self.check(opts, alias):
break
if proc.poll() is not None and lines.empty():
line = None
# ssh exited: its last words (Permission denied…) may still be on their way.
# The reader's end marker says when they're all in; a pipe that something
# else holds open (a ProxyCommand's child) gets a short grace instead.
exited = exited or time.monotonic()
if reader_done.is_set() and lines.empty() or time.monotonic() - exited > 2:
line = None
else:
continue
else:
continue
if line is None: # ssh exited
@@ -1002,7 +1192,7 @@ class Link:
break
if authed and self.control and self.check(opts, alias):
break
return self.failed(step, said, mismatch, alias)
return self.failed(step, said, mismatch, alias, shown, a["host"], device)
if not line.startswith("debug"):
said.append(line)
m = CONNECTING.search(line)
@@ -1015,12 +1205,15 @@ class Link:
self.stage("identity", "active", "Waiting for the headset's host key")
elif HOSTKEY.search(line):
m = HOSTKEY.search(line)
shown = (m.group(1), m.group(2))
self.stage("identity", "active", f"It shows {m.group(1)} key {m.group(2)[:20]}…")
elif KNOWN.search(line):
self.stage("identity", "done", "Matches the identity saved for this headset")
step = "login"
self.stage("login", "active", f"Logging in as {user}")
elif ADDED.search(line):
if not device.get("transient"):
self.settle_pin(device, tighten=False) # now, before the login can fail
self.stage("identity", "done", "First connection: saved this headset's identity")
step = "login"
self.stage("login", "active", f"Logging in as {user}")
@@ -1059,18 +1252,34 @@ class Link:
# OpenSSH lower-cases host names (FRAME.LOCAL logs as frame.local).
return not m or m.group(1).lower() in (host.lower(), alias.lower()) or "Authentication succeeded" in line
def failed(self, step, said, mismatch, alias):
def failed(self, step, said, mismatch, alias, shown=None, host=None, device=None):
text = "\n".join(said).strip()
if mismatch:
self.fail("identity", "This address answered as a different headset (its SSH identity doesn't match). "
"If SteamOS was reinstalled, use Forget Identity on the Devices tab.", text)
# A key that differs from the saved one, or (strict, pinned) none of the saved key's
# type any more: either way not the identity saved. Never accepted here: the page
# shows the new key's fingerprint and the person decides (Trust a new key).
name = (device or {}).get("name") or alias
lost = bool(device) and not frame_devices.pinned(device["id"]) and \
frame_devices.identity_established(device)
if lost:
message = (f"The SSH identity saved for {name} is missing or can't be read, so Frame Control won't "
"trust whatever answers. Check its key and trust it on the Devices tab.")
else:
message = (f"This address answered with a different SSH identity from the one saved for {name}. "
"If SteamOS was reinstalled or reset, or this is a new headset, check its new key and trust "
"it on the Devices tab. If not, a different device may be answering here.")
self.fail("identity", message, text,
hostkey={"state": "missing" if lost else "changed", "type": shown[0] if shown else None,
"fingerprint": shown[1] if shown else None, "host": host})
return "next"
# A refused key is the same at every address: stop. (Judged by ssh's own words, not
# the step: a jump host's progress lines look like the headset's.) A forward that
# a jump host couldn't open is about this address only: try the next.
forward = re.search(r"open failed|forwarding failed|Connection refused|Connection closed|timed out", text)
if re.search(r"Permission denied", text) and not forward:
self.fail("login", self.explain(text) or "The headset didn't accept this computer's key.", text)
# Not something a retry fixes: the headset needs this computer's key (Set Up Connection).
self.fail("login", self.explain(text) or "The headset didn't accept this computer's key. Run Set Up "
"Connection: it pairs this computer with the headset again.", text, auth="denied")
return "stop"
if step == "connected":
self.fail("login", "Logged in, but the shared SSH connection didn't start.", text)
@@ -1110,9 +1319,11 @@ class Link:
lead = f"Answered in {res['rtt_ms']:g} ms" if res.get("rtt_ms") is not None else "Through the jump host"
rows[i]["ssh"] = "checking"
put()
# "yes" before host_opts (ssh takes an option's first value): a test only looks, so
# it reports a key not saved yet rather than saving it as host_opts' accept-new would.
argv = [*self.mux_base[:3], "-o", "ControlPath=none", "-o", "ConnectTimeout=8",
*self.host_opts(device, ssh_target(a["host"], res.get("ip"))),
"-o", "StrictHostKeyChecking=yes", device["alias"], "true"]
"-o", "StrictHostKeyChecking=yes",
*self.host_opts(device, ssh_target(a["host"], res.get("ip"))), device["alias"], "true"]
try:
r = frame_host.run_ssh(argv, capture_output=True, stdin=subprocess.DEVNULL, text=True,
errors="replace", timeout=20)
@@ -1140,6 +1351,156 @@ class Link:
put(done=True, finished=now())
self.devices_changed()
# ---- Trust a new key ----
# Every way a key check could do more than read the key: no login of any kind, no agent
# (nor forwarding it), no X11, port or tunnel forwarding, no local or remote command, no
# terminal, no shared connection, no learning keys. Given first, so ssh takes these over
# anything ~/.ssh/config (or host_opts) says. -N: no command, even if a device accepts "none".
SCAN_OPTS = ["BatchMode=yes", "ControlMaster=no", "ControlPath=none", "ConnectTimeout=8",
"StrictHostKeyChecking=accept-new", "UpdateHostKeys=no", "HashKnownHosts=no",
"PreferredAuthentications=none", "PubkeyAuthentication=no", "PasswordAuthentication=no",
"KbdInteractiveAuthentication=no", "HostbasedAuthentication=no", "IdentityAgent=none",
"ForwardAgent=no", "ForwardX11=no", "ForwardX11Trusted=no", "ClearAllForwardings=yes",
"Tunnel=no", "PermitLocalCommand=no", "RemoteCommand=none", "RequestTTY=no"]
SCAN_TIMEOUT = 20
def scan_identity(self, device_id):
"""Read the headset's host key without trusting it or logging in. ssh writes it to a
throwaway known_hosts file during the key exchange, before any authentication, and is
stopped as soon as it has (or when it gives up); SCAN_OPTS and -N rule out anything else.
`-F none` too, unless ~/.ssh/config sends the alias through a jump host (then it's
needed for that, and SCAN_OPTS still win). The address that showed the changed key goes
first, then the rest in order. -> {"type", "fingerprint", "host", "saved", "recorded",
"matches", "recorded_match"}; the key is kept (SCAN_TTL) for trust_identity, which pins
exactly it."""
device = self.reg.get(device_id)
if not device["addresses"]:
raise frame_devices.DeviceError("This headset has no addresses to ask")
hosts = [a["host"] for a in device["addresses"]]
with self.cond:
err = self.state.get("error") or {}
mine = (self.state.get("device") or {}).get("id") == device_id
first = (err.get("hostkey") or {}).get("host") if mine else None
if first in hosts:
hosts.remove(first)
hosts.insert(0, first)
route = self.scan_route(device)
reasons = []
for host in hosts:
key, err_text = self._scan_one(device, host, route)
if key:
saved = frame_devices.pinned_keys(device_id)
recorded = [f for f in device.get("identity") or [] if isinstance(f, str)]
scan = {"type": key[0], "blob": key[1], "fingerprint": key[2], "host": host,
"port": device["port"], "user": device["user"], "at": now()}
with self.cond:
self.scans[device_id] = scan
return {"type": key[0], "fingerprint": key[2], "host": host,
"saved": [{"type": t, "fingerprint": f} for t, f in saved], "recorded": recorded,
"matches": any(f == key[2] for _, f in saved), "recorded_match": key[2] in recorded}
text = err_text.strip()
reasons.append(f"{host}: " + (self.explain(text) or (text.splitlines() or ["no answer"])[-1]))
raise frame_devices.DeviceError("Couldn't read the headset's key. " + "; ".join(reasons))
# The jump host's own ssh (a ProxyCommand the scan builds itself): it may log in to the jump
# host as configured (keys, the agent), but it never forwards the agent, X11 or ports, runs
# no local or remote command, has no terminal, shares no connection and goes no further.
HOP_OPTS = ["BatchMode=yes", "ForwardAgent=no", "ForwardX11=no", "ForwardX11Trusted=no",
"ClearAllForwardings=yes", "Tunnel=no", "PermitLocalCommand=no", "RemoteCommand=none",
"RequestTTY=no", "ControlMaster=no", "ControlPath=none", "ProxyCommand=none", "ProxyJump=none"]
def scan_route(self, device):
"""The options that take a key check to the headset: `-F none` and nothing else when
it's reached directly. Through a jump host (ProxyJump), `-F none` and a ProxyCommand
built here, whose ssh gets HOP_OPTS ahead of ~/.ssh/config: config options can't be
overridden for ProxyJump's own ssh, which inherits ForwardAgent, LocalCommand and the
rest. Through a ProxyCommand from ~/.ssh/config, or several jump hosts, it refuses:
those can't be made safe from here."""
kind, spec = proxy_route(device["alias"])
if kind is None:
return ["-F", "none"]
m = JUMP_RE.fullmatch(spec or "") if kind == "jump" else None
if not m or (m.group(3) and not 1 <= int(m.group(3)) <= 65535):
how = "a ProxyCommand" if kind == "command" else "this jump host setting"
raise frame_devices.DeviceError(
f"{device['name']} is reached through {how} in ~/.ssh/config, which Frame Control can't make safe "
"for talking to a device it doesn't trust yet, so it won't read the key that way. Add an address "
"it can reach directly (or through a single ProxyJump host) and check its key again; or compare "
"the key by hand (on the headset, in Konsole: ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub)")
user, host, port = m.groups()
# The ssh this server runs, by its full path: the shell that runs a ProxyCommand may set
# up another PATH.
ssh = frame_host.which(self.mux_base[0])
if not ssh:
raise frame_devices.DeviceError(f"Couldn't find {self.mux_base[0]} to reach the jump host with")
hop = [ssh, *(a for o in self.HOP_OPTS for a in ("-o", o)), *(["-l", user] if user else []),
*(["-p", port] if port else []), host.strip("[]")] # ssh takes a bare IPv6 address here
return ["-F", "none", "-o", "ProxyCommand=" + proxy_command(hop)]
def _scan_one(self, device, host, route):
"""-> ((type, blob, fingerprint) or None, what ssh said)."""
name = frame_devices.host_key_alias(device["id"])
path, opt = frame_devices.scan_file(device["id"])
opts = [a for o in [*self.SCAN_OPTS, f"UserKnownHostsFile={opt}"] for a in ("-o", o)]
argv = [self.mux_base[0], *route, *opts, *self.host_opts(device, host), "-N", device["alias"]]
fd_, errfile = tempfile.mkstemp(prefix="frame-scan-") # a file, not a pipe (see frame_host.run_ssh)
key, proc = None, None
try:
try:
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=fd_,
**frame_host.DETACHED)
finally:
os.close(fd_)
deadline = time.monotonic() + self.SCAN_TIMEOUT
while key is None:
ended = proc.poll() is not None
key = _scanned_key(path, name)
if key or ended or self.stopped or time.monotonic() > deadline:
break
time.sleep(0.05)
except OSError as e:
return None, str(e)
finally:
if proc and proc.poll() is None:
proc.kill() # it has the key (or gave up): nothing more to do with that device
if proc:
try:
proc.wait(5)
except subprocess.TimeoutExpired:
pass
try:
with open(errfile, "rb") as fh:
said = fh.read().decode("utf-8", "replace")
except OSError:
said = ""
_unlink(errfile)
_unlink(path)
if key is None and not said.strip() and proc and proc.returncode is not None and proc.returncode < 0:
said = f"Timed out talking to {device['alias']}"
return key, said
def trust_identity(self, device_id, fingerprint):
"""Pin the key scan_identity read, and only if `fingerprint` is the one it showed (what
the person saw and confirmed), replacing whatever was saved; and only while the headset
is still reached as it was then (same port and user, the address still one of its own).
-> the scan."""
with self.cond:
scan = self.scans.get(device_id)
if not scan or now() - scan["at"] > SCAN_TTL:
self.scans.pop(device_id, None)
raise frame_devices.DeviceError("Check the headset's key again first: the one shown has expired")
if not isinstance(fingerprint, str) or fingerprint != scan["fingerprint"]:
raise frame_devices.DeviceError("That isn't the key the headset showed. Check its key again")
self.scans.pop(device_id)
d = self.reg.get(device_id) # still a headset here
if (d["port"], d["user"]) != (scan["port"], scan["user"]) or \
scan["host"] not in [a["host"] for a in d["addresses"]]:
raise frame_devices.DeviceError("The headset's address, port or user changed since its key was read. "
"Check its key again")
frame_devices.pin_key(device_id, scan["type"], scan["blob"])
self.reg.record_identity(device_id, [scan["fingerprint"]])
return scan
# ---- the page's API: /api/devices -------------------------------------------------
@@ -1158,6 +1519,9 @@ def devices_view(link):
view = {k: v for k, v in d.items() if k not in ("config_host", "addresses")}
view["active"] = d["id"] == active["id"]
view["pinned"] = frame_devices.pinned(d["id"])
view["keys"] = [{"type": t, "fingerprint": f} for t, f in frame_devices.pinned_keys(d["id"])]
# It had an identity, but the saved key is gone or unusable: nothing is trusted until checked.
view["identity_lost"] = not view["pinned"] and frame_devices.identity_established(d)
view["addresses"] = [dict(a, network_names=[names.get(n, "an unnamed network") for n in a["networks"]])
for a in d["addresses"]]
devices.append(view)
@@ -1190,7 +1554,7 @@ def devices_action(link, body, open_setup, busy=lambda: 0):
is_active = did == active["id"]
moves = action == "use" or (action == "retry" and link.alive()) or (is_active and (
# (a retry while connected would cut the install's connection)
action in ("remove", "address-remove", "forget-identity")
action in ("remove", "address-remove", "identity-trust")
or (action == "update" and login_change(reg, did, body))
or (action == "address-update" and body.get("newHost") not in (None, body.get("host")))))
if moves and busy():
@@ -1270,11 +1634,25 @@ def devices_action(link, body, open_setup, busy=lambda: 0):
threading.Thread(target=link.test, args=(did,), daemon=True).start()
msg = f"Testing {len(d['addresses'])} address{'es' * (len(d['addresses']) != 1)}"
elif action == "forget-identity":
# Dropping the pin used to let the next contact save whatever key answered, unseen.
# A new key is now always shown first: identity-scan, then identity-trust.
reg.get(did)
raise frame_devices.DeviceError("Forgetting a headset's identity is no longer supported: use Check its key "
"on the Devices tab (identity-scan, then identity-trust), which shows the "
"new key before saving it")
elif action == "identity-scan":
d = reg.get(did)
frame_devices.forget_pin(did)
scan = link.scan_identity(did)
msg = (f"{d['name']} shows the key already saved for it" if scan["matches"]
else f"{d['name']} shows a {scan['type']} key, {scan['fingerprint']}")
link.devices_changed()
return dict(devices_view(link), message=msg, scan=scan)
elif action == "identity-trust":
d = reg.get(did)
link.trust_identity(did, body.get("fingerprint"))
if is_active:
link.kick("switch")
msg = f"Forgot {d['name']}'s SSH identity; the next connection saves the one it shows"
link.kick("switch") # reconnect with the key now saved
msg = f"Saved {d['name']}'s new SSH identity; reconnecting"
elif action == "name-network":
reg.name_network(body.get("network"), body.get("name"))
if link.state["network"]:
+6 -3
View File
@@ -53,12 +53,14 @@ SENT_KEEP = 200
OUTBOX_MAX = 2000 # events kept while offline; the oldest go first
FLUSH_EVERY = 60
REPEAT_WINDOW = 600 # the same diagnostic error is sent at most once in this many seconds
# Not faults in Frame Control: the headset asleep, away or not set up yet. The status poll
# Not faults in Frame Control: the headset asleep, away or not set up yet, or ssh and the headset
# not trusting each other (a host key that changed, a key not authorised): the page says what to do
# about each (Devices tab, Set Up Connection). The status poll
# meets these every few seconds, so each is sent at most once per session, whatever the wording,
# but only for an error marked where ssh ran as ssh failing to reach the headset
# (frame_host.link_failure). The message alone isn't evidence: a download's "Connection reset by
# peer", or a file name with ssh's words in it, keeps the usual window.
EXPECTED_CATEGORIES = ('frame_unreachable', 'frame_not_set_up')
EXPECTED_CATEGORIES = ('frame_unreachable', 'frame_not_set_up', 'frame_auth')
DEFAULT_HOST = 'https://us.i.posthog.com'
LEVELS = ('usage', 'compat', 'diagnostics')
@@ -268,7 +270,8 @@ CATEGORIES = [
('cant_run_on_frame', re.compile(r"can't run on the Frame")),
('steam_shortcut', re.compile(r'(?i)steam did not return a shortcut|shortcut list|no Steam shortcut')),
('frame_not_set_up', re.compile(r'(?i)Could not resolve hostname|no "?frame"? (?:SSH )?alias')),
('frame_auth', re.compile(r'(?i)Permission denied|Host key verification failed')),
('frame_auth', re.compile(r'(?i)Permission denied|Host key verification failed|'
r'REMOTE HOST IDENTIFICATION HAS CHANGED|host key is known for')),
# Anything ssh says about its own connection: "ssh: connect to host … port 22: <reason>" (Windows
# says "Unknown error"), a dropped shared connection (mux_client_…, client_loop), and Windows'
# "banner exchange: Connection to UNKNOWN port -1" (its ssh can't name a peer whose connect
+98 -8
View File
@@ -700,6 +700,7 @@
<div class="grow"><div class="t" id="offMsg">Can't reach the Frame</div>
<div class="s" id="offDetail">Retrying every few seconds. Everything fills in when it answers.</div></div>
<button class="small desk-only" id="offDetails" hidden>Details</button>
<button class="small desk-only" id="offTrust" hidden>Check the new key…</button>
<button class="small" id="offSetup" hidden>Set Up Connection…</button>
<button class="action small" id="offRetry">Retry now</button>
</div>
@@ -1207,6 +1208,7 @@
<div class="sources-intro"><div class="store-eyebrow">Your app shelves</div><h1 id="sourcesTitle">A world of apps.</h1><p>Choose where you discover games and apps. We bring them together for you.</p></div>
<div id="sourcesList"></div>
<form id="addSource" class="source-repo-form"><h2>Add a source</h2><p>Have a favorite F-Droid repository? Add its address to browse its apps here.</p><label>Repository address<input name="url" type="url" required placeholder="https://example.org/fdroid/repo"></label><details><summary>Verify this source (optional)</summary><label>Repository fingerprint<input name="fingerprint" type="text" autocomplete="off" placeholder="Paste the fingerprint from the repository owner"></label></details><button class="action">Add source</button></form>
<form id="itchAccount" class="source-repo-form"><h2>My itch.io purchases</h2><p id="itchAccountNote">Install the Android builds of games you own on itch.io. Create an API key in itch.io settings › API keys and paste it here. It stays on this computer and is only sent to itch.io.</p><label id="itchKeyField">itch.io API key<input name="key" type="password" autocomplete="off" spellcheck="false" maxlength="300" placeholder="Paste your API key"></label><button class="action" id="itchConnect">Connect itch.io</button><button type="button" class="small" id="itchDisconnect" hidden>Disconnect itch.io</button></form>
<p id="sourceError" class="store-notice" role="status" aria-live="polite"></p>
</dialog>
@@ -1358,6 +1360,18 @@
<button type="submit" class="small danger" id="rmGo">Remove</button></div>
</form>
</dialog>
<dialog id="trustDlg" class="dlg" aria-labelledby="trustTitle">
<form method="dialog" id="trustForm">
<h2 id="trustTitle">Check the headset's new key</h2>
<div class="sub" id="trustText"></div>
<dl class="facts" id="trustKeys"></dl>
<div class="sub" id="trustHow" hidden>To check it, on the headset open the desktop from the SteamVR dashboard, start
Konsole and run <code id="trustCmd"></code>. The <code>SHA256:</code> value it prints must match the key above.</div>
<div class="row rep-actions"><span class="sub" id="trustMsg" role="status"></span><span class="spacer"></span>
<button type="button" class="small" id="trustCancel">Cancel</button>
<button type="submit" class="small action" id="trustGo" disabled>This is my headset: trust its new key</button></div>
</form>
</dialog>
<div class="toast" id="toast"></div>
<script>
@@ -3128,15 +3142,28 @@ function sourceMark(s) { const name=storeName(s); return `<span class="offer-mar
function sourceDescription(s) { return s.description || ({fdroid:"Independent, open-source apps",sidequest:"Discover something new in VR",itch:"Games from independent creators",github:"Direct from the people who make it"})[s.kind] || (s.name.includes("GitHub") ? "Direct from the people who make it" : s.name.includes("Droid") ? "Independent, open-source apps" : s.name.includes("SideQuest") ? "Discover something new in VR" : s.name.includes("itch") ? "Games from independent creators" : "Apps shared by this community"); }
async function loadSources() {
try {
const {sources} = await api("/api/sources"); sourceState.sources = sources;
const {sources, itch} = await api("/api/sources"); sourceState.sources = sources; showItchAccount(itch);
$("sourceFilter").innerHTML = `<option value="">All sources</option>` + sources.filter(s=>s.enabled).map(s=>`<option value="${esc(s.id)}">${esc(storeName(s.name))}</option>`).join("");
$("sourceFilter").value = sourceState.source;
$("sourcesList").innerHTML = sources.map(s=>`<div class="source-setting">${sourceMark(s.name)}<div class="grow"><strong>${esc(storeName(s.name))}</strong><p>${esc(sourceDescription(s))}</p><small>${esc(({official:"Official store",community:"Community",user:"Added by you"})[s.trust] || "Community")}</small>${s.builtin === false ? `<button class="small" data-remove="${esc(s.id)}" aria-label="Remove ${esc(s.name)}">Remove</button>` : ""}</div><input class="source-toggle" role="switch" type="checkbox" aria-label="Enable ${esc(storeName(s.name))}" data-enable="${esc(s.id)}" ${s.enabled ? "checked" : ""}></div>`).join("") || `<p class="store-notice">Your sources will appear here when they are available.</p>`;
} catch (e) { $("sourceError").textContent = e.message; }
}
// Owned on itch.io (the user's own purchases) or free: what the store shows instead of a price.
const storePrice = o => o.owned ? (o.owned_label || "Owned") : "Free";
function showItchAccount(itch) {
const on = !!itch?.connected;
$("itchAccountNote").textContent = on ? `Connected${itch.username ? " as " + itch.username : ""}. Android builds of games you own appear in the store, marked Owned on itch.io. Disconnecting deletes the API key from this computer.` : "Install the Android builds of games you own on itch.io. Create an API key in itch.io settings › API keys and paste it here. It stays on this computer and is only sent to itch.io.";
$("itchKeyField").hidden = on; $("itchConnect").hidden = on; $("itchDisconnect").hidden = !on;
$("itchAccount").elements.key.required = !on;
}
async function itchAccount(body) {
$("sourceError").textContent = body.action === 'itch-connect' ? "Checking the key with itch.io…" : "Disconnecting…";
try { const result=await api('/api/sources',body);$("sourceError").textContent=result.message;showItchAccount(result.itch);sourceState.source='';await loadSources();await searchSources(); }
catch(e) { $("sourceError").textContent=e.message;await loadSources(); }
}
function storeCard(app) {
const i=sourceState.apps.indexOf(app), o=app.offers[0];
return `<button class="store-card" data-app="${i}" aria-label="View ${esc(app.name)}">${storeArt(o)}<span class="store-card-content"><span class="store-card-title">${storeIcon(o)}<strong>${esc(app.name)}</strong></span><span class="store-summary">${esc(storeText(app.summary) || "A new addition to your Frame")}</span><span class="store-card-foot"><span>${o.vr ? "VR experience" : o.vr === false ? "Flat-screen app" : "Android app"}</span><b>Free</b></span></span></button>`;
return `<button class="store-card" data-app="${i}" aria-label="View ${esc(app.name)}">${storeArt(o)}<span class="store-card-content"><span class="store-card-title">${storeIcon(o)}<strong>${esc(app.name)}</strong></span><span class="store-summary">${esc(storeText(app.summary) || "A new addition to your Frame")}</span><span class="store-card-foot"><span>${o.vr ? "VR experience" : o.vr === false ? "Flat-screen app" : "Android app"}</span><b>${esc(storePrice(app.offers.find(x=>x.owned) || o))}</b></span></span></button>`;
}
function storeRow(title, subtitle, apps) { return apps.length ? `<section class="store-row"><div class="store-row-head"><h2>${esc(title)}</h2><p>${esc(subtitle)}</p></div><div class="store-grid">${apps.slice(0,5).map(storeCard).join("")}</div></section>` : ""; }
function storeEmpty() {
@@ -3181,7 +3208,7 @@ async function searchSources(quiet) {
finally { if(request===sourceState.request)$("searchGrid").setAttribute("aria-busy","false"); }
}
function currentOffer() { return sourceState.detail?.app.offers[sourceState.detail.choice]; }
function detailBadges(o) { return `<span class="store-badge">${o.vr ? "VR experience" : o.vr===false ? "Flat-screen app" : "Android app"}</span>${o.verdict?.tone==='works' ? '<span class="store-badge works">✓ Works on Frame</span>' : ''}<span class="store-badge">Free</span>${storeOpenSource(o) ? '<span class="store-badge">Open source</span>' : ''}`; }
function detailBadges(o) { return `<span class="store-badge">${o.vr ? "VR experience" : o.vr===false ? "Flat-screen app" : "Android app"}</span>${o.verdict?.tone==='works' ? '<span class="store-badge works">✓ Works on Frame</span>' : ''}<span class="store-badge">${esc(storePrice(o))}</span>${storeOpenSource(o) ? '<span class="store-badge">Open source</span>' : ''}`; }
// The store's app page. (Not renderDetail: the Devices page declares that further down,
// and a later function declaration silently replaces an earlier one; issue #68.)
// True when the page rendered; false when it shows the error instead.
@@ -3197,7 +3224,7 @@ function renderAppDetail() {
function renderAppDetailBody() {
const restoreOfferFocus=document.activeElement?.name==='storeOffer';
const d=sourceState.detail;if(!d)return;const o=currentOffer(), images=o.artwork?.screenshots||[], page=sourcePage(o.page);
$("detailBody").innerHTML=storeArt(o,"detail-banner",true)+`<div class="detail-content"><div class="detail-heading">${storeIcon(o)}<div><h1 id="detailTitle">${esc(o.name)}</h1><p>${esc(o.developer || o.author || 'From '+storeName(o.source_name))}</p></div></div><div class="store-badges">${detailBadges(o)}</div><div class="detail-layout"><div><div class="detail-gallery" id="detailGallery" aria-label="App screenshots" tabindex="0">${images.map((p,i)=>storeArt(o,"",false,p,`${o.name} screenshot ${i+1}`)).join("")}</div>${images.length ? `<div class="gallery-controls"><span>${images.length} screenshots · scroll to explore</span><div><button data-gallery="-1" aria-label="Previous screenshot">←</button> <button data-gallery="1" aria-label="Next screenshot">→</button></div></div>` : ''}<h2>About this app</h2><div class="detail-description">${esc(storeText(o.description || o.summary || "Discover this app on its developer's page."))}</div>${page ? `<p><a href="${esc(page)}" target="_blank" rel="noopener noreferrer">Meet the creators ↗</a></p>` : ''}<details class="detail-technical"><summary>Technical details</summary><dl><dt>Package</dt><dd>${esc(o.package||"Unknown")}</dd><dt>Version</dt><dd>${esc(o.version||"Unknown")} (${esc(o.version_code??"unknown")})</dd><dt>Android API</dt><dd>${esc(o.min_sdk??"Unknown")}</dd><dt>Architecture</dt><dd>${esc(o.abis?.join(', ')||"Unknown / no native code")}</dd><dt>Updated</dt><dd>${esc(o.updated||"Unknown")}</dd><dt>Compatibility</dt><dd>${(o.fit?.reasons||[]).map(esc).join(' · ')||'No known installation blockers'}</dd></dl></details></div><aside class="detail-buy"><div class="detail-price">Free</div><p class="detail-small">No purchase needed.</p><p class="detail-verdict ${esc(o.verdict?.tone||'unknown')}">${esc(o.verdict?.label||'Not yet checked on the Frame')}</p><button id="detailInstall" class="action store-primary">Install</button><progress id="detailProgress" max="100" hidden aria-label="Installation progress"></progress><p id="detailInstallNote" class="detail-small" role="status" aria-live="polite"></p><h2>Get it from</h2><div id="detailOffers">${d.app.offers.map((v,j)=>`<label class="offer-choice"><input type="radio" name="storeOffer" value="${j}" ${j===d.choice?'checked':''}><span>${sourceMark(v.source_name)}</span><span><strong>${esc(storeName(v.source_name))}${j===0?' · Recommended':''}</strong><small>${v.verified===true?'✓ Verified download':'Verification not provided'}</small><small>${esc(v.version ? 'v'+v.version : 'Latest release')}${v.size!=null?' · '+(v.size/1048576).toFixed(0)+' MB':''}</small></span></label>`).join('')}</div></aside></div></div>`;
$("detailBody").innerHTML=storeArt(o,"detail-banner",true)+`<div class="detail-content"><div class="detail-heading">${storeIcon(o)}<div><h1 id="detailTitle">${esc(o.name)}</h1><p>${esc(o.developer || o.author || 'From '+storeName(o.source_name))}</p></div></div><div class="store-badges">${detailBadges(o)}</div><div class="detail-layout"><div><div class="detail-gallery" id="detailGallery" aria-label="App screenshots" tabindex="0">${images.map((p,i)=>storeArt(o,"",false,p,`${o.name} screenshot ${i+1}`)).join("")}</div>${images.length ? `<div class="gallery-controls"><span>${images.length} screenshots · scroll to explore</span><div><button data-gallery="-1" aria-label="Previous screenshot">←</button> <button data-gallery="1" aria-label="Next screenshot">→</button></div></div>` : ''}<h2>About this app</h2><div class="detail-description">${esc(storeText(o.description || o.summary || "Discover this app on its developer's page."))}</div>${page ? `<p><a href="${esc(page)}" target="_blank" rel="noopener noreferrer">Meet the creators ↗</a></p>` : ''}<details class="detail-technical"><summary>Technical details</summary><dl><dt>Package</dt><dd>${esc(o.package||"Unknown")}</dd><dt>Version</dt><dd>${esc(o.version||"Unknown")} (${esc(o.version_code??"unknown")})</dd><dt>Android API</dt><dd>${esc(o.min_sdk??"Unknown")}</dd><dt>Architecture</dt><dd>${esc(o.abis?.join(', ')||"Unknown / no native code")}</dd><dt>Updated</dt><dd>${esc(o.updated||"Unknown")}</dd><dt>Compatibility</dt><dd>${(o.fit?.reasons||[]).map(esc).join(' · ')||'No known installation blockers'}</dd></dl></details></div><aside class="detail-buy"><div class="detail-price">${esc(storePrice(o))}</div><p class="detail-small">${o.owned ? "Downloads with your itch.io account." : "No purchase needed."}</p><p class="detail-verdict ${esc(o.verdict?.tone||'unknown')}">${esc(o.verdict?.label||'Not yet checked on the Frame')}</p><button id="detailInstall" class="action store-primary">Install</button><progress id="detailProgress" max="100" hidden aria-label="Installation progress"></progress><p id="detailInstallNote" class="detail-small" role="status" aria-live="polite"></p><h2>Get it from</h2><div id="detailOffers">${d.app.offers.map((v,j)=>`<label class="offer-choice"><input type="radio" name="storeOffer" value="${j}" ${j===d.choice?'checked':''}><span>${sourceMark(v.source_name)}</span><span><strong>${esc(storeName(v.source_name))}${j===0?' · Recommended':''}</strong><small>${v.verified===true?'✓ Verified download':'Verification not provided'}</small><small>${esc(v.version ? 'v'+v.version : 'Latest release')}${v.size!=null?' · '+(v.size/1048576).toFixed(0)+' MB':''}</small></span></label>`).join('')}</div></aside></div></div>`;
updateInstall();
if(restoreOfferFocus)document.querySelector('#detailOffers input:checked')?.focus();
}
@@ -3267,6 +3294,9 @@ async function addSource(body) {
if(result){$("addSource").reset();sourceState.source='';await loadSources();await searchSources();}
}
$("addSource").onsubmit=e=>{e.preventDefault();const f=new FormData(e.target);addSource({action:'add',url:f.get('url'),fingerprint:f.get('fingerprint')});};
// The key goes to the server once and leaves the page straight away: the field is cleared before the reply.
$("itchAccount").onsubmit=e=>{e.preventDefault();const input=e.target.elements.key, key=input.value.trim();input.value='';if(key)itchAccount({action:'itch-connect',key});};
$("itchDisconnect").onclick=()=>itchAccount({action:'itch-disconnect'});
document.querySelectorAll('[data-close]').forEach(b=>b.onclick=()=>$(b.dataset.close).close());
$('appDetail').addEventListener('close',()=>{sourceState.detailRequest++;});
document.querySelectorAll('[data-store-tab]').forEach(b=>b.onclick=()=>{const browse=b.dataset.storeTab==='browse';$('appStore').hidden=!browse;$('androidLibrary').hidden=browse;$('repNew').hidden=browse;$('sourcesOpen').hidden=!browse;document.querySelectorAll('[data-store-tab]').forEach(t=>{t.classList.toggle('on',t===b);t.setAttribute('aria-pressed',t===b);});if(!browse)loadAndroid();});
@@ -4542,6 +4572,10 @@ function renderBanner() {
const s = link.s;
if (!link.live || !s) return;
$("offDetails").hidden = false;
// A changed host key: never accepted by itself. The person checks the new key and decides.
const hk = s.phase === "failed" && s.error && s.error.hostkey && s.device ? s.device.id : null;
$("offTrust").hidden = !hk;
if (hk) $("offTrust").onclick = () => openTrust(hk);
if (s.phase === "failed" && s.error) {
$("offline").hidden = false;
$("offMsg").textContent = s.error.message;
@@ -4642,6 +4676,9 @@ function onConnection(s) {
loadDevices();
}
link.phase = s.phase; link.device = devId;
// What the Devices tab offers for the failure (check a new key, pair again) follows it.
const problem = s.phase === "failed" && s.error ? (s.error.hostkey ? "hostkey" : s.error.auth || "") : "";
if (problem !== link.problem) { link.problem = problem; renderDevices(); }
if (page === "devices") renderTests();
}
@@ -4801,8 +4838,15 @@ function renderDetail() {
<button class="small" data-mv="-1" title="Try earlier"${i ? "" : " disabled"}>↑</button>
<button class="small" data-mv="1" title="Try later"${i < d.addresses.length - 1 ? "" : " disabled"}>↓</button>
<button class="small" data-ed>Edit</button><button class="small danger" data-rm>Remove</button></div>`).join("");
const err = d.active && link.s && link.s.phase === "failed" ? link.s.error : null;
const problem = err && err.hostkey ? `<div class="item" role="alert"><div class="grow"><div class="t">${err.hostkey.state === "missing"
? "Its saved SSH identity is missing" : "Its SSH identity has changed"}</div>
<div class="s">${esc(err.message)}</div></div><button class="small action" id="dvTrustNow">Check the new key…</button></div>`
: err && err.auth === "denied" ? `<div class="item" role="alert"><div class="grow"><div class="t">The headset didn't accept this computer's key</div>
<div class="s">Set Up Connection pairs this computer with it again. On the headset, Developer Mode must be on.</div></div>
<button class="small action" id="dvPair">Set Up Connection…</button></div>` : "";
el.innerHTML = `<div class="shelf-head"><h2>${esc(d.name)}</h2>${d.active ? '<span class="count">In use</span>' : ""}<span class="spacer"></span>
${d.active ? "" : `<button class="small action" id="dvUse">Use this headset</button>`}</div>
${d.active ? "" : `<button class="small action" id="dvUse">Use this headset</button>`}</div>${problem}
<form class="dev-form" id="dvForm">
<label>Name<input type="text" id="dvName" value="${esc(d.name)}" maxlength="60"></label>
<label>SSH alias<input type="text" value="${esc(d.alias)}" readonly title="Terminal: ssh ${esc(d.alias)}"></label>
@@ -4823,10 +4867,12 @@ function renderDetail() {
<button class="small" id="dvTs">Find on Tailscale</button>
<button class="small" id="dvMdns">Find on this network</button>
<span class="spacer"></span>
<button class="small" id="dvForget" title="After reinstalling SteamOS the headset shows a new identity">Forget identity</button>
<button class="small" id="dvTrust" title="After SteamOS is reinstalled or reset the headset shows a new identity">Check its key…</button>
<button class="small danger" id="dvRemove">Remove…</button></div>
<div class="found" id="dvFound"></div>
<div class="hint">Identity: ${d.pinned ? "saved. A different device answering at one of these addresses is refused."
<div class="hint">Identity: ${d.pinned ? `saved${(d.keys || []).length ? ` (${esc(d.keys.map(k => k.fingerprint).join(", "))})` : ""}.
A different device answering at one of these addresses is refused.`
: d.identity_lost ? "the saved key is missing or can't be read, so nothing is trusted until you check its key."
: "not saved yet; the next connection saves it."} Kind "auto" is picked from the address:
<code>.local</code> names are mDNS, <code>100.x</code> and <code>.ts.net</code> are Tailscale.</div>`;
$("adKind").insertAdjacentHTML("afterbegin", `<option value="" selected>Kind: auto</option>`);
@@ -4859,7 +4905,9 @@ function renderDetail() {
$("dvTest").onclick = e => devAction(`Test ${d.name}'s addresses`, { action: "test", id: d.id }, e.currentTarget);
$("dvTs").onclick = e => findOn("tailscale", d, e.currentTarget);
$("dvMdns").onclick = e => findOn("mdns", d, e.currentTarget);
$("dvForget").onclick = e => devAction(`Forget ${d.name}'s identity`, { action: "forget-identity", id: d.id }, e.currentTarget);
$("dvTrust").onclick = () => openTrust(d.id);
if ($("dvTrustNow")) $("dvTrustNow").onclick = () => openTrust(d.id);
if ($("dvPair")) $("dvPair").onclick = () => setUpHeadset(d.alias);
$("dvRemove").onclick = () => askRemove(d);
renderFound(d);
renderTests();
@@ -4912,6 +4960,48 @@ function renderFound(d) {
});
}
// Trust a new key: read the key the headset shows now (nothing is saved), show it next to the
// saved one, and save it only when the person says it's their headset. The server pins exactly
// the key it showed here (frame_link.Link.trust_identity), never whatever answers next.
const HOST_KEY_FILE = { "ssh-ed25519": "ed25519", "ssh-rsa": "rsa", "ssh-dss": "dsa" };
async function openTrust(id) {
const d = dv.list.find(x => x.id === id);
const name = d ? d.name : "the headset";
$("trustTitle").textContent = `Check ${name}'s key`;
$("trustText").textContent = "Asking the headset for its key…";
$("trustKeys").innerHTML = ""; $("trustMsg").textContent = "";
$("trustHow").hidden = true; $("trustGo").disabled = true;
$("trustCancel").onclick = () => $("trustDlg").close();
$("trustForm").onsubmit = e => e.preventDefault();
if (!$("trustDlg").open) $("trustDlg").showModal();
let res;
try { res = await api("/api/devices", { action: "identity-scan", id }); }
catch (e) { $("trustText").textContent = e.message; return; }
const scan = res.scan;
const saved = (scan.saved || []).map(k => `${k.type} ${k.fingerprint}`).join("\n")
|| ((scan.recorded || []).length ? `${scan.recorded.join("\n")}\n(recorded; the saved file is missing or unreadable)` : "None");
$("trustKeys").innerHTML = [["Saved key", saved], ["It shows now", `${scan.type} ${scan.fingerprint}`], ["At", scan.host]]
.map(([k, v]) => `<dt>${esc(k)}</dt><dd style="white-space:pre-wrap; word-break:break-all"><code>${esc(v)}</code></dd>`).join("");
if (scan.matches) {
$("trustText").textContent = `${name} shows the key already saved for it, so there's nothing to change. If it still won't connect, try Test now.`;
return;
}
$("trustText").textContent = scan.recorded_match
? `${name} shows the key recorded for it before, but the saved copy is missing or can't be read. Trust it to save it again.`
: (scan.saved || []).length || (scan.recorded || []).length
? `${name} now shows a different SSH key from the one saved. That's expected after SteamOS was reinstalled or reset, or if this is a new headset at the same address. If none of that happened, don't trust it: another device may be answering in its place.`
: `No key is saved for ${name} yet. Trust this one if it's your headset.`;
$("trustCmd").textContent = `ssh-keygen -lf /etc/ssh/ssh_host_${HOST_KEY_FILE[scan.type] || "ecdsa"}_key.pub`;
$("trustHow").hidden = false;
$("trustGo").disabled = false;
$("trustForm").onsubmit = async e => {
e.preventDefault();
const ok = await devAction(`Trust ${name}'s new key`, { action: "identity-trust", id, fingerprint: scan.fingerprint }, $("trustGo"));
if (ok) $("trustDlg").close();
else $("trustMsg").textContent = "Not saved. Close this and check the key again.";
};
}
function askRemove(d) {
$("rmDevTitle").textContent = `Remove ${d.name}?`;
$("rmDevText").textContent = `Frame Control forgets this headset, its ${d.addresses.length} address${d.addresses.length === 1 ? "" : "es"} and its saved identity. Its SSH keys stay.`;
+17 -4
View File
@@ -46,6 +46,7 @@ import frame_assistant # noqa: E402
import frame_android # noqa: E402
from apk_sources import search as apk_search, SourceError # noqa: E402
from apk_sources import fdroid as apk_fdroid # noqa: E402
from apk_sources import _itch_account as itch_account # noqa: E402
import frame_apk_versions # noqa: E402
import frame_catalog # noqa: E402
import frame_devices # noqa: E402
@@ -211,9 +212,14 @@ UNREACHABLE = [
(re.compile(r"port \d+: Connection refused"),
"The Frame refused the connection. Check Developer Mode is still on."),
(re.compile(r"Permission denied \(publickey"),
"The Frame didn't accept this computer's SSH key. Run Set Up Connection again."),
(re.compile(r"Host key verification failed"),
"The Frame's SSH identity changed (after a reinstall, or a different device). Run Set Up Connection again."),
"The Frame didn't accept this computer's SSH key. Set Up Connection pairs this computer with it "
"again (Developer Mode must be on)."),
# Before "Host key verification failed", which ssh prints after each of these.
(re.compile(r"No \S+ host key is known for "),
"The Frame's SSH identity isn't saved yet. Reconnecting saves it; or check and trust it on the Devices tab."),
(re.compile(r"REMOTE HOST IDENTIFICATION HAS CHANGED|Host key verification failed"),
"The Frame's SSH identity has changed: after SteamOS was reinstalled or reset, or a different device "
"answering at its address. Check the new key on the Devices tab before trusting it."),
(re.compile(r"kex_exchange_identification|Connection closed by .* port \d+|Connection reset by .* port \d+"),
"The connection to the Frame dropped. Try again."),
]
@@ -2205,6 +2211,13 @@ def source_manage(body):
if action == 'remove':
apk_search.manage_repo('remove_repo', source_id=source_text(body, 'source'))
return {'message': 'Repository removed'}
if action == 'itch-connect': # the key is checked with itch.io, saved 0600, never echoed
key = body.get('key')
if not isinstance(key, str) or not key.strip() or len(key) > 300:
raise Failure('Paste your itch.io API key', 400)
return itch_account.connect(key)
if action == 'itch-disconnect':
return itch_account.disconnect()
raise Failure('Unknown source action', 400)
except SourceError as e:
raise Failure(str(e), 400)
@@ -2421,7 +2434,7 @@ class Handler(BaseHTTPRequestHandler):
except SourceError as e:
raise Failure(str(e), 400)
elif path == "/api/sources":
self.send_json({"sources": apk_search.sources()})
self.send_json({"sources": apk_search.sources(), "itch": itch_account.status()})
elif path == "/api/search":
self.send_json(source_search(url.query))
elif path == "/api/apk-versions":