Source logins: refuse addresses with a backslash and a login; decode escapes in one pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/apk-repos.md

@@ -58,8 +58,61 @@ form shows the pinned one ("Trusted on first use: …") so you can compare it.
 Re-adding an existing URL preserves its pin; changing it requires deliberately
 removing and re-adding it.
 
+### Repositories behind a login
+
+Some self-hosted repositories sit behind HTTP basic auth. As in F-Droid's own
+client, the login can be part of the address
+(`https://user:pass@example.org/fdroid/repo`), or given separately: in the
+store's **Add a source** form under "This repository needs a login", in the
+server API (`username` and `password` beside `url`), or on the command line,
+where the password is read from standard input so it stays out of shell history
+and `ps`:
+
+```sh
+printf '%s\n' "$PASSWORD" | python3 ui/apk_sources/fdroid.py add 'https://example.org/fdroid/repo' --username me --password-stdin
+```
+
+How the login is kept and used:
+
+- **Kept out of the address.** The `user:pass@` part is removed before anything
+  is saved, so the saved repository URL, its id, the source list, job results
+  and the page never contain it. The store only says "Uses a login".
+- **Stored separately.** Logins live in
+  `frame_host.data_dir('source-credentials.json')`
+  (`~/Library/Application Support/Frame Control/source-credentials.json` on
+  macOS), written with mode 0600 and keyed by source id. They are stored in
+  plain text there, protected by file permissions only (on Windows,
+  the folder's normal ACLs). Removing the repository deletes its login, and
+  the file goes when the last login is removed. Each change holds a lock
+  file (`source-credentials.lock`), so the app and the command line can't undo
+  each other's saves or removals.
+- **HTTPS only.** An `http://` address with a login is refused before any request.
+- **Inside the repository only.** The `Authorization` header is decided
+  separately for every request, redirects included: index, `entry.jar`,
+  APKs and the store's artwork proxy. It is sent only when the request's
+  scheme, host and port match the repository's and its path is inside the
+  repository's directory by whole path segments. A redirect to another host
+  or port, or to another path on the same server, gets no login, and that
+  request's own `Authorization` header is removed. Paths with `.` or `..`
+  segments (plain or percent-encoded), encoded `/` or `\`, double encoding
+  or `;` get no login, since a server could normalise them to a path
+  outside the repository. Index file names like that are refused outright.
+- **Never echoed.** Logins are not written to logs, error messages, API
+  responses or the cache. The CLI prints only the saved source.
+- **Checked before saving.** As with any repository, the whole index must
+  download and verify before anything is saved. HTTP 401 (or 403 when a login
+  was sent) stops the add with "check the username and password" and saves
+  nothing: no repository, no login, no rollback state. A 403 with a login is
+  not treated as rate limiting.
+- **Same verification.** The login changes only who may download. Signature,
+  fingerprint pin or TOFU, index hash and rollback checks apply exactly as for
+  any other repository.
+
+Re-adding the same address with a new login replaces the saved one. Re-adding
+it without a login keeps the saved one.
+
 The API for the search/server integration is in `ui/apk_sources/fdroid.py`:
-`add_repo(url, fingerprint=None, name=None)`, `remove_repo(source_id)`,
+`add_repo(url, fingerprint=None, name=None, username=None, password=None)`, `remove_repo(source_id)`,
 `set_enabled(source_id, enabled)`, and `user_repos()`. The module also exposes
 `sources`, `search`, `details`, and `download` from the shared source contract.
 This change supplies the CLI and API; the integrated source-management UI is

docs/apk-sources.md

@@ -45,10 +45,23 @@ search. Results outside the curated list stay page-only, even if a repository
 claims an open-source license. This prevents an arbitrary tagged mirror from
 becoming a trusted downloader. Extend the curated JSON after provenance review.
 
-Set optional `FRAME_GITHUB_TOKEN` in the process environment for a higher API
-quota. Tokens are sent only to `api.github.com`, never written to the cache,
-never sent to asset hosts, and removed on redirects. The adapter does not
-read `gh` credentials automatically. Metadata is cached for one hour under
+For a higher API quota, add a GitHub access token in the store's source
+settings (the GitHub row's token field) or through the server API
+(`POST /api/sources` with `action: "set-token"` or `"clear-token"` and
+`source: "github"`). `FRAME_GITHUB_TOKEN` in the process environment still
+works and wins over a saved token. A saved token is kept with repository
+logins in `source-credentials.json` (mode 0600, see
+[APK repositories](apk-repos.md#repositories-behind-a-login)). The source list
+reports only whether a token is set, where it came from and whether GitHub
+rejected it, never its value. Tokens are sent as `Authorization: Bearer` only to
+`api.github.com`, never written to the cache, never sent to asset hosts (public
+release downloads don't need them), and removed on redirects. If GitHub answers
+401 to the token, the request is retried once without it, as Obtainium does.
+The rejection is logged without the token and shown in source settings. The
+token is not tried again until it is saved again or the app restarts. Rate-limit
+backoff is kept per login (anonymous, or each token), so saving a token after
+anonymous requests were limited lets the next request go out with it. The
+adapter does not read `gh` credentials automatically. Metadata is cached for one hour under
 `frame_host.cache_dir('apk-sources', 'publisher')`. A cold details request
 fetches at most ten releases. Rate-limit errors are surfaced without retry
 loops. Asset IDs and release tags are not Android version codes: metadata

tests/apk_fixtures.py

@@ -0,0 +1,87 @@
+"""Small APKs built in memory (binary manifest plus resource table), for test_frame_apk
+and the end-to-end tests. Deliberately doesn't import sandbox: tests/e2e runs in a
+throwaway container and keeps its real home, which sandbox would move."""
+import io
+import os
+import struct
+import sys
+import zipfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ui'))
+import frame_apk  # noqa: E402
+
+
+def pool(strings, utf8=False):
+    """A ResStringPool chunk."""
+    data, offsets = b'', []
+    for s in strings:
+        offsets.append(len(data))
+        if utf8:
+            b = s.encode()
+            data += bytes([len(s), len(b)]) + b + b'\0'
+        else:
+            data += struct.pack('<H', len(s)) + s.encode('utf-16-le') + b'\0\0'
+    data += b'\0' * (-len(data) % 4)
+    start = 28 + 4 * len(strings)
+    body = struct.pack(f'<{len(strings)}I', *offsets) + data
+    return struct.pack('<HHIIIIII', 1, 28, 28 + len(body), len(strings), 0, 0x100 if utf8 else 0, start, 0) + body
+
+
+def manifest(package, label_ref, version_ref, min_sdk, package_raw=True, foreign_label=False):
+    """<manifest package versionName><uses-sdk minSdkVersion/><application label icon/></manifest>.
+
+    package_raw=False drops the package's raw string (as some repackers do);
+    foreign_label adds a non-android `label` attribute after android:label.
+    """
+    strings = ['label', 'icon', 'versionName', 'minSdkVersion', 'package', 'manifest', 'uses-sdk',
+               'application', package, 'junk', 'label', 'versionCode']  # the second 'label' has no android id
+    resmap = struct.pack('<4I', 0x01010001, 0x01010002, 0x0101021c, 0x0101020c)
+    resmap = struct.pack('<HHI', 0x0180, 8, 8 + len(resmap)) + resmap
+
+    def element(name, attrs):
+        body = struct.pack('<IIHHHHHH', 0xffffffff, name, 20, 20, len(attrs), 0, 0, 0)
+        for aname, raw, dtype, value in attrs:
+            body += struct.pack('<IIIHBBI', 0xffffffff, aname, raw, 8, 0, dtype, value)
+        return struct.pack('<HHIII', 0x0102, 16, 16 + len(body), 1, 0xffffffff) + body
+
+    none = 0xffffffff
+    chunks = (pool(strings) + resmap
+              + element(5, [(4, 8 if package_raw else none, frame_apk.T_STRING, 8),
+                            (2, none, frame_apk.T_REF, version_ref),
+                            (11, none, frame_apk.T_INT_DEC, 210)])
+              + element(6, [(3, none, frame_apk.T_INT_DEC, min_sdk)])
+              + element(7, [(0, none, frame_apk.T_REF, label_ref), (1, none, frame_apk.T_REF, 0x7f020000)]
+                        + ([(10, 9, frame_apk.T_STRING, 9)] if foreign_label else [])))
+    return struct.pack('<HHI', 3, 8, 8 + len(chunks)) + chunks
+
+
+def resources(values):
+    """resources.arsc with package 0x7f; values: {(type id, entry, language, density): global string index}."""
+    strings = ['French label', 'App label', '2.1', 'res/icon_lo.png', 'res/icon_hi.png', 'res/icon.xml']
+    pkg_body = b''
+    for (tid, lang, density), entries in values.items():
+        cfg = struct.pack('<I4x2s4xH', 64, lang.encode().ljust(2, b'\0'), density).ljust(64, b'\0')
+        count = max(entries) + 1
+        offsets, data = [], b''
+        for i in range(count):
+            if i in entries:
+                offsets.append(len(data))
+                data += struct.pack('<HHI', 8, 0, 0) + struct.pack('<HBBI', 8, 0, frame_apk.T_STRING, entries[i])
+            else:
+                offsets.append(0xffffffff)
+        header = 20 + 64
+        estart = header + 4 * count
+        body = struct.pack(f'<{count}I', *offsets) + data
+        pkg_body += struct.pack('<HHIBBHII', 0x0201, header, header + len(body), tid, 0, 0, count, estart) + cfg + body
+    pkg_header = struct.pack('<HHII', 0x0200, 288, 288 + len(pkg_body), 0x7f).ljust(288, b'\0')
+    pkg = pkg_header + pkg_body
+    table = pool(strings, utf8=True) + pkg
+    return struct.pack('<HHII', 2, 12, 12 + len(table), 1) + table
+
+
+def apk(files):
+    buf = io.BytesIO()
+    with zipfile.ZipFile(buf, 'w') as z:
+        for name, data in files.items():
+            z.writestr(name, data)
+    return buf.getvalue()

tests/e2e/harness.py

@@ -10,6 +10,7 @@ import atexit
 import http.client
 import json
 import os
+import shutil
 import socket
 import subprocess
 import sys
@@ -22,7 +23,22 @@ from pathlib import Path
 ROOT = Path(__file__).resolve().parents[2]
 sys.path[:0] = [str(ROOT / 'ui'), str(ROOT / 'tests'), str(ROOT / 'tests' / 'smoke')]
 
-ENABLED = os.environ.get('FRAME_E2E') == '1'
+# Only inside the compose `host` container (FRAME_E2E_CONTAINER from compose.yaml, plus
+# the container runtime's marker file): these tests drive ui/server.py with the real
+# home, ~/.ssh and cache, which is only safe in a throwaway container.
+REQUESTED = os.environ.get('FRAME_E2E') == '1'
+IN_CONTAINER = os.environ.get('FRAME_E2E_CONTAINER') == '1' and (
+    os.path.exists('/.dockerenv') or os.path.exists('/run/.containerenv'))
+ENABLED = REQUESTED and IN_CONTAINER
+if ENABLED:
+    # What tests/sandbox.py does, minus moving HOME: the container's real ~/.ssh is what
+    # points `ssh frame` at the fake Frame, and OpenSSH reads ~/.ssh from the password
+    # database, not $HOME. (These tests used to get this through test_frame_apk.)
+    _data = tempfile.mkdtemp(prefix='frame-control-e2e-')
+    atexit.register(shutil.rmtree, _data, ignore_errors=True)
+    os.environ['FRAME_CONTROL_DATA_DIR'] = _data
+    os.environ['FRAME_CONTROL_TELEMETRY'] = '0'
+    os.environ['FRAME_COMPAT_DB_URL'] = 'http://127.0.0.1:9'
 CTL = os.environ.get('FAKEFRAME_CTL', 'http://fakeframe:9999').rstrip('/')
 FAKE_HOST = os.environ.get('FAKEFRAME_HOST', 'fakeframe')
 HOME = '/home/steamos'
@@ -30,9 +46,14 @@ SERVER_LOG = os.path.join(tempfile.gettempdir(), 'fakeframe-e2e-server.log')
 
 
 def require():
-    """Call at module level: skips the module unless the fake Frame is up."""
-    if not ENABLED:
-        raise unittest.SkipTest('needs the fake Frame: run scripts/e2e.sh (sets FRAME_E2E=1)')
+    """Skips (the module, at module level; the class, in setUpClass) unless this is
+    scripts/e2e.sh's container with the fake Frame up."""
+    if ENABLED:
+        return
+    if REQUESTED:
+        raise unittest.SkipTest('FRAME_E2E=1 outside the e2e container: these tests use the real home, ~/.ssh '
+                                'and app cache, so they only run in scripts/e2e.sh\'s throwaway container')
+    raise unittest.SkipTest('needs the fake Frame: run scripts/e2e.sh (sets FRAME_E2E=1)')
 
 
 # ---- the fake Frame's control port --------------------------------------------
@@ -97,6 +118,7 @@ class Server:
 
     @classmethod
     def start(cls):
+        require()  # never a server on this computer's real data
         if cls.proc and cls.proc.poll() is None:
             return
         with socket.socket() as s:
@@ -193,6 +215,7 @@ class FrameTestCase(unittest.TestCase):
 
     @classmethod
     def setUpClass(cls):
+        require()
         Server.start()
 
     def setUp(self):

tests/e2e/test_agents.py

@@ -6,6 +6,8 @@ import sys
 import harness
 from harness import api, ok, finished, ssh
 
+harness.require()
+
 sys.path.insert(0, str(harness.ROOT / 'ui'))
 import frame_mcp
 

tests/e2e/test_android.py

@@ -5,7 +5,7 @@ import unittest
 
 import harness
 from harness import HOME, exists, ok, state, upload, wait_for
-from test_frame_apk import apk, manifest, resources
+from apk_fixtures import apk, manifest, resources
 
 harness.require()
 

tests/fakeframe/compose.yaml

@@ -43,6 +43,7 @@ services:
       FAKEFRAME_CTL: http://fakeframe:9999
       FAKEFRAME_HOST: fakeframe
       FRAME_E2E: "1"
+      FRAME_E2E_CONTAINER: "1"  # tests/e2e/harness.py refuses to run anywhere else
     healthcheck:
       test: ["CMD", "test", "-f", "/home/tester/.ready"]
       interval: 1s

tests/page/source_login.mjs

@@ -0,0 +1,79 @@
+// The store's "Add a source" login handling and GitHub token field, run in node against
+// the real functions from ui/index.html. A login typed into the address must leave the
+// address field (into the login fields) before anything is sent, and stay out of it.
+import { readFileSync } from "fs";
+const src = readFileSync(new URL("../../ui/index.html", import.meta.url), "utf8");
+const lines = src.split("\n");
+const fail = msg => { console.log("FAIL " + msg); process.exit(1); };
+
+function grab(name) {
+  const start = lines.findIndex(l => new RegExp(`^(?:(?:async )?function ${name}\\(|const ${name} = )`).test(l));
+  if (start < 0) fail(`no top-level ${name} in index.html`);
+  let depth = 0;
+  for (let i = start; i < lines.length; i++) {
+    for (const c of lines[i]) depth += c === "{" ? 1 : c === "}" ? -1 : 0;
+    if (depth <= 0) return lines.slice(start, i + 1).join("\n") + "\n";
+  }
+  fail(`unbalanced ${name}`);
+}
+const { moveLogin, sourceToken } = new Function(
+  "const esc = s => String(s).replace(/[&<>\"']/g, c => `&#${c.charCodeAt(0)};`); const storeName = s => s;\n" +
+  grab("decodeLenient") + grab("moveLogin") + grab("sourceToken") + "return { moveLogin, sourceToken };")();
+
+function form(url, username = "", password = "") {
+  const details = { open: false };
+  const field = value => ({ value, closest: () => details });
+  return { details, elements: { url: field(url), username: field(username), password: field(password) } };
+}
+
+let f = form("https://alice:s3cr%40t@repo.example/fdroid/repo");
+moveLogin(f);
+if (f.elements.url.value !== "https://repo.example/fdroid/repo") fail("address still holds the login: " + f.elements.url.value);
+if (f.elements.username.value !== "alice" || f.elements.password.value !== "s3cr@t") fail("login not moved into its fields");
+if (!f.details.open) fail("login fields not shown");
+
+f = form("https://alice:fromurl@repo.example/repo", "typed", "typedpw");
+moveLogin(f);
+if (f.elements.url.value.includes("fromurl") || f.elements.username.value !== "typed" || f.elements.password.value !== "typedpw")
+  fail("typed login replaced, or address kept the login");
+
+// Undecodable logins and unusual spellings the URL parser accepts still lose their login; undecodable bytes stay as typed.
+for (const [url, address, user, pass] of [
+  ["https://alice:100%done@repo.example/private/repo", "https://repo.example/private/repo", "alice", "100%done"],
+  ["https:\t//alice:s3cret@repo.example/repo", "https://repo.example/repo", "alice", "s3cret"],
+  ["ht\ntps://alice:s3cret@repo.example/repo", "https://repo.example/repo", "alice", "s3cret"],
+  ["https:alice:s3cret@repo.example/repo", "https://repo.example/repo", "alice", "s3cret"],
+  ["\u0000https://alice:s3cret@repo.example/repo", "https://repo.example/repo", "alice", "s3cret"],
+  ["https://alice:%FF@repo.example/private/repo", "https://repo.example/private/repo", "alice", "%FF"],
+  ["https://al%20ice:p%40ss:word@repo.example/r", "https://repo.example/r", "al ice", "p@ss:word"],
+  ["https://alice:p%FF\u00e4@repo.example/repo", "https://repo.example/repo", "alice", "p%FF\u00e4"],
+  ["fdroidrepos://alice:pw@repo.example/repo?fingerprint=AB", "fdroidrepos://repo.example/repo?fingerprint=AB", "alice", "pw"],
+  ["https://alice@repo.example/repo", "https://repo.example/repo", "alice", ""],
+]) {
+  f = form(url);
+  moveLogin(f);
+  if (f.elements.url.value !== address) fail(`address kept the login: ${url} -> ${f.elements.url.value}`);
+  if (f.elements.username.value !== user || f.elements.password.value !== pass)
+    fail(`login for ${url}: ${f.elements.username.value} / ${f.elements.password.value}`);
+}
+
+// An address the parser rejects that may hold a login is refused and left as typed, never sent.
+for (const url of ["https://alice\\name:pw@repo.example/repo", "https:\\\\alice:s3cret@repo.example/repo", "fdroidrepos://alice:p\\w@repo.example/repo", "https://alice:s3cret@repo example/repo", "https:alice:s3cret@repo example/repo", "https://repo.example:bad\\a@b"]) {
+  f = form(url);
+  if (moveLogin(f) !== false) fail("accepted an invalid address with a login: " + url);
+  if (f.elements.username.value || f.elements.password.value) fail("guessed a login from an invalid address: " + url);
+}
+
+for (const url of ["https://repo.example/repo", "not a url", "https://repo.example/a@b", "fdroidrepos:///a@b", "fdroidrepos:/a@b", "https://repo.example/r?x=a@b#c@d"]) {
+  f = form(url);
+  moveLogin(f);
+  if (f.elements.url.value !== url || f.elements.username.value) fail("changed an address without a login: " + url);
+}
+
+const html = sourceToken({ id: "github", name: "GitHub releases", token: { set: true, from: "saved", rejected: true } });
+if (!html.includes("rejected") || !html.includes('data-clear-token="github"')) fail("token row: " + html);
+if (sourceToken({ id: "fdroid", name: "F-Droid" }) !== "") fail("token field on a source without one");
+const slow = "https://a:" + "%FF".repeat(4000) + "@repo.example/r", t0 = Date.now();
+f = form(slow); moveLogin(f);
+if (Date.now() - t0 > 200 || f.elements.password.value !== "%FF".repeat(4000)) fail("decoding malformed escapes is slow or wrong");
+console.log("ok");

tests/sandbox.py

@@ -1,12 +1,20 @@
 """Imported first by every test module: nothing a test does reaches this person's
-app data, their telemetry, or the shared compatibility database.
+app data, their ~/.ssh, their telemetry, or the shared compatibility database.
 
 Must run before any ui module is imported, since those read these at import time.
+The variables are set in os.environ, so servers and scripts the tests start
+inherit them too (they pass {**os.environ, ...}).
 """
 import atexit
 import os
 import shutil
 import tempfile
+from pathlib import Path
+
+# The environment and real ~/.ssh from before the changes below: test_sandbox
+# reruns tests from that environment and checks ~/.ssh is left alone.
+ORIGINAL_ENV = dict(os.environ)
+REAL_SSH_DIR = Path.home() / ".ssh"
 
 _dir = tempfile.mkdtemp(prefix="frame-control-tests-")
 atexit.register(shutil.rmtree, _dir, ignore_errors=True)
@@ -14,3 +22,25 @@ os.environ["FRAME_CONTROL_DATA_DIR"] = _dir
 os.environ["FRAME_CONTROL_TELEMETRY"] = "0"
 # A maintainer's machine holds the database key; send anything that slips through nowhere.
 os.environ["FRAME_COMPAT_DB_URL"] = "http://127.0.0.1:9"
+
+# A throwaway home, so Path.home() / "~" land here: ~/.ssh (config, host-key pins,
+# its lock file) and the other per-user folders some modules use. A server started
+# without FRAME_CONTROL_SSH_DIR used to import the real ~/.ssh/config and seed a pin
+# into the real ~/.ssh/frame-control-hosts on every run.
+#
+# tests/e2e must not import this (directly or through a test_* module): they run in
+# a throwaway container whose real ~/.ssh points `ssh frame` at the fake Frame, and
+# OpenSSH finds ~/.ssh from the password database, not $HOME. Shared helpers they
+# need live in sandbox-free modules such as apk_fixtures.py.
+HOME = os.path.join(_dir, "home")
+SSH_DIR = os.path.join(HOME, ".ssh")
+os.makedirs(SSH_DIR, mode=0o700)
+os.environ["HOME"] = HOME
+os.environ["USERPROFILE"] = HOME  # Path.home() on Windows
+os.environ["FRAME_CONTROL_SSH_DIR"] = SSH_DIR  # frame_devices.ssh_dir(), frame_connect.SSH_DIR
+# The per-user folders that don't follow HOME when these are set: frame_host.cache_dir()
+# (servers warm indexes there and sweep its staging folders) and data_dir()'s fallbacks.
+for _name, _sub in (("LOCALAPPDATA", "AppData/Local"), ("APPDATA", "AppData/Roaming"),
+                    ("XDG_CACHE_HOME", ".cache"), ("XDG_DATA_HOME", ".local/share"),
+                    ("XDG_STATE_HOME", ".local/state"), ("XDG_CONFIG_HOME", ".config")):
+    os.environ[_name] = os.path.join(HOME, *_sub.split("/"))

tests/test_agent.py

@@ -1,4 +1,5 @@
 """MCP protocol, exact-action approvals and explicit assistant data sharing."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import io
 import json
 import os

tests/test_apk_artwork.py

@@ -1,3 +1,4 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import http.client
 import json
 from pathlib import Path

tests/test_apk_more_sources.py

@@ -1,3 +1,4 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import hashlib, io, json, os, sys, tempfile, time, unittest, urllib.error, urllib.request, zipfile
 from pathlib import Path
 from unittest.mock import patch

tests/test_apk_search.py

@@ -1,3 +1,4 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import http.client
 import json
 from pathlib import Path
@@ -324,7 +325,7 @@ class EndpointTests(SettingsTest):
             mod.add_repo.side_effect = SourceError('repository fingerprint mismatch')
             job = self.wait(self.request('POST', '/api/sources', {'action': 'add', 'url': link})[1]['job'])
             self.assertEqual(job['error'], 'repository fingerprint mismatch')  # no "SourceError:" prefix
-            for url in ('http://repo.example/repo', 'fdroidrepo://repo.example/repo', 'https://u@repo.example/'):
+            for url in ('http://repo.example/repo', 'fdroidrepo://repo.example/repo', 'https://repo.example:x/'):
                 self.assertEqual(self.request('POST', '/api/sources', {'action': 'add', 'url': url})[0], 400)
             self.assertEqual(self.request('POST', '/api/sources', {'action': 'remove', 'source': 'fdroid'})[0], 200)
             mod.remove_repo.assert_called_once_with(source_id='fdroid')

tests/test_comfort.py

@@ -1,4 +1,5 @@
 """Fake-Frame session clock/actions plus real helper serialization and sensor probes."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import os
 import shutil
 import json

tests/test_comfort_ui.py

@@ -1,4 +1,5 @@
 """Run the actual shared page's comfort renderer against a minimal DOM/bridge."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import pathlib
 import shutil
 import subprocess

tests/test_fdroid_sources.py

@@ -1,4 +1,5 @@
 """Offline authenticated repository fixtures; no tests contact a server."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import io
 import json
 import os
@@ -80,7 +81,7 @@ class Repositories(unittest.TestCase):
             state.clear()
             self.addCleanup(state.clear)
 
-    def fetch(self, url, path, maximum):
+    def fetch(self, url, path, maximum, auth=None):
         name = url.rsplit('/', 1)[-1]
         if self.v1 and name == 'entry.jar':
             raise urllib.error.HTTPError(url, 404, 'missing', None, None)
@@ -310,7 +311,7 @@ class Repositories(unittest.TestCase):
         source = self.add()
         other = dict(source, id='other-repo')
         started, release = threading.Event(), threading.Event()
-        def fetch(url, path, maximum):
+        def fetch(url, path, maximum, auth=None):
             if threading.current_thread().name == 'slow':
                 started.set()
                 release.wait(5)
@@ -354,7 +355,7 @@ class Repositories(unittest.TestCase):
         self.files['entry.jar'], _ = entry_jar(50)
         source = fdroid.add_repo(URL)
         jars = {'older': entry_jar(100)[0], 'newer': entry_jar(200)[0]}
-        def fetch(url, path, maximum):
+        def fetch(url, path, maximum, auth=None):
             name = threading.current_thread().name
             if name in jars and url.endswith('entry.jar'):
                 Path(path).write_bytes(jars[name])
@@ -400,7 +401,7 @@ class Repositories(unittest.TestCase):
         source = dict(id='overlap', name='Overlap', url=URL, fingerprint=None)
         self.v1 = True
         inner = []
-        def fetch(url, path, maximum):
+        def fetch(url, path, maximum, auth=None):
             if url.endswith('index-v1.jar') and not inner:
                 inner.append(1)  # the v1 load passed its fallback check; a v2 load finishes now
                 self.v1 = False
@@ -470,7 +471,7 @@ class Repositories(unittest.TestCase):
         source = self.add()
         self.expire(source)
         release = threading.Event()
-        def slow(url, path, maximum):
+        def slow(url, path, maximum, auth=None):
             release.wait(5)
             self.fetch(url, path, maximum)
         self.fetch_mock.side_effect = slow
@@ -527,7 +528,7 @@ class Repositories(unittest.TestCase):
         self.expire(source)
         before = self.fetch_mock.call_count
         release = __import__('threading').Event()
-        def slow(url, path, maximum):
+        def slow(url, path, maximum, auth=None):
             release.wait(5)
             self.fetch(url, path, maximum)
         self.fetch_mock.side_effect = slow

tests/test_frame_android_data.py

@@ -1,3 +1,4 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import io
 import json
 import os

tests/test_frame_android_library.py

@@ -1,4 +1,5 @@
 """Offline artwork, Steam API and launcher supervision regressions."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import importlib.util
 import json
 import os

tests/test_frame_apk.py

@@ -11,82 +11,7 @@ import zipfile
 
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ui'))
 import frame_apk  # noqa: E402
-
-
-def pool(strings, utf8=False):
-    """A ResStringPool chunk."""
-    data, offsets = b'', []
-    for s in strings:
-        offsets.append(len(data))
-        if utf8:
-            b = s.encode()
-            data += bytes([len(s), len(b)]) + b + b'\0'
-        else:
-            data += struct.pack('<H', len(s)) + s.encode('utf-16-le') + b'\0\0'
-    data += b'\0' * (-len(data) % 4)
-    start = 28 + 4 * len(strings)
-    body = struct.pack(f'<{len(strings)}I', *offsets) + data
-    return struct.pack('<HHIIIIII', 1, 28, 28 + len(body), len(strings), 0, 0x100 if utf8 else 0, start, 0) + body
-
-
-def manifest(package, label_ref, version_ref, min_sdk, package_raw=True, foreign_label=False):
-    """<manifest package versionName><uses-sdk minSdkVersion/><application label icon/></manifest>.
-
-    package_raw=False drops the package's raw string (as some repackers do);
-    foreign_label adds a non-android `label` attribute after android:label.
-    """
-    strings = ['label', 'icon', 'versionName', 'minSdkVersion', 'package', 'manifest', 'uses-sdk',
-               'application', package, 'junk', 'label', 'versionCode']  # the second 'label' has no android id
-    resmap = struct.pack('<4I', 0x01010001, 0x01010002, 0x0101021c, 0x0101020c)
-    resmap = struct.pack('<HHI', 0x0180, 8, 8 + len(resmap)) + resmap
-
-    def element(name, attrs):
-        body = struct.pack('<IIHHHHHH', 0xffffffff, name, 20, 20, len(attrs), 0, 0, 0)
-        for aname, raw, dtype, value in attrs:
-            body += struct.pack('<IIIHBBI', 0xffffffff, aname, raw, 8, 0, dtype, value)
-        return struct.pack('<HHIII', 0x0102, 16, 16 + len(body), 1, 0xffffffff) + body
-
-    none = 0xffffffff
-    chunks = (pool(strings) + resmap
-              + element(5, [(4, 8 if package_raw else none, frame_apk.T_STRING, 8),
-                            (2, none, frame_apk.T_REF, version_ref),
-                            (11, none, frame_apk.T_INT_DEC, 210)])
-              + element(6, [(3, none, frame_apk.T_INT_DEC, min_sdk)])
-              + element(7, [(0, none, frame_apk.T_REF, label_ref), (1, none, frame_apk.T_REF, 0x7f020000)]
-                        + ([(10, 9, frame_apk.T_STRING, 9)] if foreign_label else [])))
-    return struct.pack('<HHI', 3, 8, 8 + len(chunks)) + chunks
-
-
-def resources(values):
-    """resources.arsc with package 0x7f; values: {(type id, entry, language, density): global string index}."""
-    strings = ['French label', 'App label', '2.1', 'res/icon_lo.png', 'res/icon_hi.png', 'res/icon.xml']
-    pkg_body = b''
-    for (tid, lang, density), entries in values.items():
-        cfg = struct.pack('<I4x2s4xH', 64, lang.encode().ljust(2, b'\0'), density).ljust(64, b'\0')
-        count = max(entries) + 1
-        offsets, data = [], b''
-        for i in range(count):
-            if i in entries:
-                offsets.append(len(data))
-                data += struct.pack('<HHI', 8, 0, 0) + struct.pack('<HBBI', 8, 0, frame_apk.T_STRING, entries[i])
-            else:
-                offsets.append(0xffffffff)
-        header = 20 + 64
-        estart = header + 4 * count
-        body = struct.pack(f'<{count}I', *offsets) + data
-        pkg_body += struct.pack('<HHIBBHII', 0x0201, header, header + len(body), tid, 0, 0, count, estart) + cfg + body
-    pkg_header = struct.pack('<HHII', 0x0200, 288, 288 + len(pkg_body), 0x7f).ljust(288, b'\0')
-    pkg = pkg_header + pkg_body
-    table = pool(strings, utf8=True) + pkg
-    return struct.pack('<HHII', 2, 12, 12 + len(table), 1) + table
-
-
-def apk(files):
-    buf = io.BytesIO()
-    with zipfile.ZipFile(buf, 'w') as z:
-        for name, data in files.items():
-            z.writestr(name, data)
-    return buf.getvalue()
+from apk_fixtures import apk, manifest, pool, resources  # noqa: E402,F401
 
 
 class ApkInfo(unittest.TestCase):

tests/test_frame_apk_vr.py

@@ -1,4 +1,5 @@
 """Local-only VR manifest, raw ZIP and independent signature checks."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import io
 import os
 from pathlib import Path
@@ -15,7 +16,7 @@ import frame_apk
 import frame_apk_vr as vr
 import frame_apk_sign as signing
 import frame_android
-from test_frame_apk import pool
+from apk_fixtures import pool
 
 
 DEFAULT = 'android.intent.category.DEFAULT'

tests/test_input.py

@@ -6,6 +6,7 @@ identity line, then starts TLS as the *client*.
 
 Run: python3 -m unittest discover -s tests
 """
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import json
 import os
 import shutil

tests/test_library_entrypoints.py

@@ -1,4 +1,5 @@
 """Every public sideload entry point reaches the mandatory five-slot writer."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import contextlib
 import io
 import json

tests/test_macview.py

@@ -7,6 +7,7 @@ input reaching the source, and closing.
 
 Run: python3 -m unittest discover -s tests
 """
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import base64
 import http.client
 import json

tests/test_media.py

@@ -1,4 +1,5 @@
 """Owned media planning, eye isolation, decoder choice and fake-Frame ownership."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import argparse
 import io
 import json

tests/test_panels.py

@@ -1,4 +1,5 @@
 """Fake-Frame panel responses and the headset page's real HTTP guards."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import http.client
 import json
 from pathlib import Path

tests/test_sandbox.py

@@ -0,0 +1,122 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT / "ui"))
+import frame_connect  # noqa: E402
+import frame_devices  # noqa: E402
+import frame_host  # noqa: E402
+
+
+def inside(path, folder):
+    path, folder = os.path.realpath(path), os.path.realpath(folder)
+    return path == folder or path.startswith(folder + os.sep)
+
+
+def snapshot(folder):
+    """{relative path: (size, mtime)} for everything under FOLDER."""
+    out = {}
+    for base, dirs, files in os.walk(folder):
+        for name in dirs + files:
+            p = os.path.join(base, name)
+            try:
+                st = os.lstat(p)
+            except OSError:
+                continue
+            out[os.path.relpath(p, folder)] = (st.st_size, st.st_mtime_ns)
+    return out
+
+
+class Sandbox(unittest.TestCase):
+    """Tests once wrote a host-key pin into the real ~/.ssh/frame-control-hosts on every
+    run (a server started without FRAME_CONTROL_SSH_DIR imported the real ~/.ssh/config)."""
+
+    def test_home_and_ssh_dir_are_throwaway(self):
+        tmp = tempfile.gettempdir()
+        self.assertTrue(inside(Path.home(), tmp), Path.home())
+        self.assertTrue(inside(os.path.expanduser("~"), tmp))
+        for d in (frame_devices.ssh_dir(), frame_devices.pin_dir(), frame_connect.SSH_DIR,
+                  os.environ["FRAME_CONTROL_SSH_DIR"], frame_host.cache_dir(), frame_host.data_dir()):
+            self.assertTrue(inside(d, tmp), d)
+            self.assertFalse(inside(d, sandbox.REAL_SSH_DIR), d)
+
+    def test_cache_folders_on_every_platform_are_throwaway(self):
+        """cache_dir() reads LOCALAPPDATA on Windows and XDG_CACHE_HOME on Linux, which a
+        moved HOME alone doesn't cover: servers sweep staging folders there."""
+        tmp = tempfile.gettempdir()
+        for platform in ("MAC", "WINDOWS", None):
+            with mock.patch.object(frame_host, "MAC", platform == "MAC"), \
+                    mock.patch.object(frame_host, "WINDOWS", platform == "WINDOWS"):
+                self.assertTrue(inside(frame_host.cache_dir("apk-sources"), tmp), platform)
+        for name in ("LOCALAPPDATA", "APPDATA", "XDG_CACHE_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME"):
+            self.assertTrue(inside(os.environ[name], tmp), name)
+
+    def test_end_to_end_tests_stay_out_of_it(self):
+        """tests/e2e keeps the container's real home (OpenSSH reads ~/.ssh from the password
+        database, not $HOME), so nothing there may import sandbox, even through a test_* module."""
+        here = Path(__file__).resolve().parent
+        pattern = re.compile(r"^\s*(import sandbox|from sandbox |import test_|from test_)", re.M)
+        for f in [*(here / "e2e").glob("*.py"), *(here / "smoke").glob("*.py"), here / "apk_fixtures.py"]:
+            self.assertIsNone(pattern.search(f.read_text(encoding="utf-8")), f)
+
+    def test_end_to_end_tests_start_no_server_outside_their_container(self):
+        """Plain discovery over tests/e2e, with or without FRAME_E2E=1, must skip everything
+        and never start ui/server.py: outside scripts/e2e.sh's container it would run on
+        this computer's real app data, ~/.ssh and cache."""
+        e2e = str(Path(__file__).resolve().parent / "e2e")
+        code = ("import io, sys, unittest; sys.path.insert(0, sys.argv[1]); import harness; calls = []\n"
+                "harness.Server.start = classmethod(lambda cls: calls.append(cls))\n"
+                "suite = unittest.defaultTestLoader.discover(sys.argv[1], top_level_dir=sys.argv[1])\n"
+                "r = unittest.TextTestRunner(stream=io.StringIO()).run(suite)\n"
+                "print(len(calls), r.testsRun, len(r.skipped), len(r.errors) + len(r.failures))")
+        for extra in ({}, {"FRAME_E2E": "1"}):
+            env = {k: v for k, v in os.environ.items() if not k.startswith("FRAME_E2E")}
+            env.update(extra, PYTHONDONTWRITEBYTECODE="1")
+            out = subprocess.run([sys.executable, "-c", code, e2e], env=env, capture_output=True, text=True,
+                                 timeout=120)
+            self.assertEqual(out.returncode, 0, out.stderr[-2000:])
+            started, ran, skipped, broken = map(int, out.stdout.split())
+            self.assertEqual((started, broken), (0, 0), (extra, out.stdout))
+            self.assertEqual(ran, skipped, (extra, out.stdout))
+            self.assertGreater(skipped, 0, extra)
+
+    def test_child_processes_inherit_it(self):
+        out = subprocess.run(
+            [sys.executable, "-c", "import sys; sys.path.insert(0, sys.argv[1]); import frame_devices, frame_connect;"
+             " from pathlib import Path; print(Path.home()); print(frame_devices.ssh_dir()); print(frame_connect.SSH_DIR)",
+             str(ROOT / "ui")], env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"},
+            capture_output=True, text=True, check=True).stdout.split("\n")
+        for d in filter(None, out):
+            self.assertFalse(inside(d, sandbox.REAL_SSH_DIR), d)
+            self.assertTrue(inside(d, tempfile.gettempdir()), d)
+
+    def test_real_ssh_folder_untouched_by_tests_that_start_servers(self):
+        """Reruns, from the environment the suite was started in, the tests that used to
+        leak (servers started without FRAME_CONTROL_SSH_DIR) plus the device and connect
+        tests, and checks nothing under the real ~/.ssh changed."""
+        real = sandbox.REAL_SSH_DIR
+        if inside(real, tempfile.gettempdir()):
+            self.skipTest("this run's HOME is already a temporary folder")
+        before = snapshot(real)
+        env = {k: v for k, v in sandbox.ORIGINAL_ENV.items() if k != "FRAME_CONTROL_SSH_DIR"}
+        env["PYTHONDONTWRITEBYTECODE"] = "1"
+        r = subprocess.run([sys.executable, "-m", "unittest", "-q", "test_server.OneServer", "test_server.LocalMode",
+                            "test_devices", "test_connect"],
+                           cwd=str(Path(__file__).resolve().parent), env=env, capture_output=True, text=True,
+                           timeout=300)
+        self.assertEqual(r.returncode, 0, r.stderr[-3000:])
+        after = snapshot(real)
+        changed = sorted(k for k in set(before) | set(after) if before.get(k) != after.get(k)
+                         and not k.startswith("agent"))  # ssh-agent sockets come and go on their own
+        self.assertEqual(changed, [], f"tests changed the real {real}")
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_sidequest.py

@@ -1,3 +1,4 @@
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import json
 from pathlib import Path
 import sys

tests/test_source_logins.py

@@ -0,0 +1,566 @@
+"""Repositories and sources behind a login: F-Droid basic auth and GitHub tokens.
+
+A fake HTTPS layer stands in for the network (urllib's real redirect, error and
+request processing still run), so these tests see every request and its headers.
+"""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
+import base64
+import contextlib
+import http.client
+import io
+import json
+import os
+from pathlib import Path
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import unittest
+from unittest.mock import Mock, patch
+import urllib.request
+import urllib.response
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
+from apk_sources import SourceAuthError, SourceError, SourceLimited, _credentials, _images, _web, fdroid, github, search
+import server
+
+FIXTURES = Path(__file__).parent / 'fixtures' / 'fdroid'
+PIN = (FIXTURES / 'fingerprint.txt').read_text().strip()
+REPO = 'https://repo.example/fdroid/repo/'
+USER, PASSWORD = 'alice', 's3cr@t-pässword'
+TOKEN = 'ghp_frameControlTestToken0123456789'
+SECRETS = (PASSWORD, TOKEN, base64.b64encode((USER + ':' + PASSWORD).encode()).decode())
+REAL_BUILD_OPENER = urllib.request.build_opener
+
+
+def basic(user, password):
+    return 'Basic ' + base64.b64encode((user + ':' + password).encode()).decode()
+
+
+class FakeHTTPS(urllib.request.HTTPSHandler):
+    """Answers https requests from route(url, authorization) -> (status, headers, body)."""
+
+    def __init__(self, route):
+        super().__init__()
+        self.route, self.seen = route, []
+
+    def https_open(self, req):
+        auth = req.get_header('Authorization')
+        self.seen.append((req.full_url, auth))
+        status, headers, body = self.route(req.full_url, auth)
+        message = http.client.HTTPMessage()
+        for key, value in headers.items():
+            message[key] = value
+        response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, status)
+        response.msg = 'fake'
+        return response
+
+
+class Base(unittest.TestCase):
+    def setUp(self):
+        tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(tmp.cleanup)
+        self.root = Path(tmp.name)
+        for name, value in [('data_dir', lambda *p: self.root.joinpath('data', *p)),
+                            ('cache_dir', lambda *p: self.root.joinpath('cache', *p))]:
+            mock = patch.object(fdroid.frame_host, name, value)
+            mock.start()
+            self.addCleanup(mock.stop)
+        self.web = FakeHTTPS(self.route)
+        mock = patch('urllib.request.build_opener', lambda *handlers: REAL_BUILD_OPENER(*handlers, self.web))
+        mock.start()
+        self.addCleanup(mock.stop)
+        mock = patch.dict(os.environ, {github.ENV: ''})
+        mock.start()
+        self.addCleanup(mock.stop)
+        for state in (_web._limited, fdroid._stale, fdroid._retry_at, fdroid._refreshing, github._rejected):
+            state.clear()
+            self.addCleanup(state.clear)
+        self.stderr = io.StringIO()
+        mock = patch('sys.stderr', self.stderr)
+        mock.start()
+        self.addCleanup(mock.stop)
+
+    def route(self, url, auth):
+        raise AssertionError('unexpected request ' + url)
+
+    def assertNoSecrets(self, *texts):
+        for text in texts:
+            for secret in SECRETS + (USER + ':',):
+                self.assertNotIn(secret, text)
+
+    def disk(self):
+        """Everything saved under the data and cache folders except the credentials file."""
+        out = []
+        for path in self.root.rglob('*'):
+            if path.is_file() and path.name != 'source-credentials.json':
+                out.append(path.read_bytes().decode('utf-8', 'replace'))
+        return '\n'.join(out)
+
+    def credentials_file(self):
+        return self.root / 'data' / 'source-credentials.json'
+
+
+class Repository(Base):
+    """A repository at REPO that wants USER and PASSWORD."""
+    def setUp(self):
+        super().setUp()
+        self.login = basic(USER, PASSWORD)
+        self.redirect = None  # file name -> absolute URL it redirects to
+        self.forbidden = 401
+
+    def serve(self, name):
+        return (FIXTURES / ('example.apk' if name.endswith('.apk') else name)).read_bytes()
+
+    def route(self, url, auth):
+        for mirror in ('https://mirror.example/fdroid/repo/', 'https://repo.example/elsewhere/'):
+            if url.startswith(mirror):  # serves anyone; a login must never get here unless same-origin
+                return 200, {}, self.serve(url[len(mirror):])
+        self.assertTrue(url.startswith(REPO), url)
+        if auth != self.login:
+            return self.forbidden, {'WWW-Authenticate': 'Basic realm="repo"'}, b'no'
+        name = url[len(REPO):]
+        if self.redirect and name in self.redirect:
+            return 302, {'Location': self.redirect[name]}, b''
+        return 200, {}, self.serve(name)
+
+    def add(self, **kw):
+        return fdroid.add_repo('https://alice:s3cr%40t-p%C3%A4ssword@repo.example/fdroid/repo', PIN, **kw)
+
+
+class RepositoryLogin(Repository):
+    def test_add_search_download_with_login_in_the_url(self):
+        source = self.add()
+        self.assertEqual(source['url'], REPO)
+        self.assertEqual(source['fingerprint'], PIN)
+        saved = fdroid.user_repos()[0]
+        self.assertEqual(saved['url'], REPO)
+        self.assertNotIn('login', saved)
+        self.assertEqual(len(fdroid.search(source, 'example offline')), 1)
+        downloaded = fdroid.download(source, 'org.example.app', 1)
+        self.assertTrue(downloaded['verified'])
+        self.assertTrue(self.web.seen)
+        self.assertTrue(all(auth == self.login for url, auth in self.web.seen))
+        self.assertTrue(all('@' not in url for url, auth in self.web.seen))  # never in the URL
+        self.assertTrue(next(s for s in fdroid.sources() if s['id'] == source['id'])['login'])
+        self.assertNoSecrets(json.dumps(source), json.dumps(fdroid.sources()), self.disk(), self.stderr.getvalue())
+        # The settings file holds no part of the login; the credentials file holds it, privately.
+        settings = (self.root / 'data' / 'apk-repos.json').read_text()
+        self.assertNotIn(USER, settings)
+        self.assertTrue(self.credentials_file().is_file())
+        if os.name != 'nt':
+            self.assertEqual(stat.S_IMODE(self.credentials_file().stat().st_mode), 0o600)
+
+    def test_login_given_apart_and_saved_login_reused_on_refresh(self):
+        source = fdroid.add_repo(REPO, PIN, username=USER, password=PASSWORD)
+        self.assertEqual(source['url'], REPO)
+        self.web.seen.clear()
+        fdroid._load(fdroid.user_repos()[0], force=True)  # as a background refresh would
+        self.assertTrue(self.web.seen)
+        self.assertTrue(all(auth == self.login for url, auth in self.web.seen))
+        fdroid.add_repo(REPO)  # re-adding without a login keeps the saved one
+        self.assertTrue(_credentials.has(source['id']))
+
+    def test_wrong_or_missing_login_saves_nothing(self):
+        for kw, message in [({'username': USER, 'password': 'wrong'}, 'check the username and password'),
+                            ({}, 'needs a username and password')]:
+            with self.subTest(kw=kw):
+                with self.assertRaisesRegex(SourceAuthError, message) as caught:
+                    fdroid.add_repo(REPO, PIN, **kw)
+                self.assertNoSecrets(str(caught.exception), repr(caught.exception))
+                self.assertNotIn('wrong', str(caught.exception))
+                self.assertEqual(fdroid.user_repos(), [])
+                self.assertFalse(self.credentials_file().exists())
+                self.assertFalse((self.root / 'data' / 'apk-repo-state.json').exists())
+                self.assertEqual(_web._limited, {})  # an auth failure isn't a rate limit
+
+    def test_403_with_a_login_is_a_login_error_not_a_rate_limit(self):
+        self.forbidden = 403
+        with self.assertRaisesRegex(SourceAuthError, 'check the username and password'):
+            fdroid.add_repo(REPO, PIN, username=USER, password='wrong')
+        self.assertEqual(_web._limited, {})
+        self.assertEqual(fdroid.user_repos(), [])
+
+    def test_signature_checks_still_apply(self):
+        with self.assertRaisesRegex(SourceError, 'fingerprint mismatch'):
+            fdroid.add_repo(REPO, '0' * 64, username=USER, password=PASSWORD)
+        self.assertEqual(fdroid.user_repos(), [])
+        self.assertFalse(self.credentials_file().exists())
+
+    def test_cross_origin_redirect_never_gets_the_login(self):
+        self.redirect = {'entry.jar': 'https://mirror.example/fdroid/repo/entry.jar',
+                         'index-v2.json': 'https://repo.example/elsewhere/index-v2.json'}
+        source = self.add()
+        self.assertEqual(source['fingerprint'], PIN)  # the mirror's copy still had to verify
+        mirror = [auth for url, auth in self.web.seen if url.startswith('https://mirror.example/')]
+        same_origin = [auth for url, auth in self.web.seen if url.startswith('https://repo.example/elsewhere/')]
+        self.assertEqual(mirror, [None])
+        self.assertEqual(same_origin, [None])  # same origin but outside the repository: no login either
+
+    def test_traversal_names_are_refused_and_never_get_the_login(self):
+        for name in ('%252e%252e/%252e%252e/evil/collect.apk', '..;/..;/evil/collect.apk', 'a%2f..%2f..%2fevil.apk',
+                     'a%5c..%5cevil.apk', '%2e%2e/evil.apk', 'x;y.apk', '100%25.apk'):
+            with self.subTest(name=name), self.assertRaisesRegex(SourceError, 'unsafe|outside'):
+                fdroid._child(REPO, name)
+        self.assertEqual(fdroid._child(REPO, '/app/en-US/phoneScreenshots/#0 a.png'),
+                         REPO + 'app/en-US/phoneScreenshots/%230%20a.png')
+        # Even if such a URL reached the downloader directly, the login stays home.
+        login = _credentials.basic(REPO, USER, PASSWORD)
+        for url in (REPO + '%252e%252e/%252e%252e/evil/collect.apk', REPO + '..;/..;/evil/collect.apk'):
+            with self.subTest(url=url), tempfile.TemporaryDirectory() as tmp:
+                self.web.seen.clear()
+                with contextlib.suppress(SourceError):  # the repository answers 401 without the login
+                    fdroid._fetch(url, Path(tmp) / 'out', 1 << 20, auth=login)
+                self.assertEqual(self.web.seen, [(url, None)])
+
+    def test_same_origin_redirect_outside_the_repository_gets_no_login(self):
+        self.redirect = {'example1.apk': 'https://repo.example/elsewhere/example1.apk'}
+        source = self.add()
+        self.web.seen.clear()
+        self.assertTrue(fdroid.download(source, 'org.example.app', 1)['verified'])
+        self.assertEqual(self.web.seen, [(REPO + 'example1.apk', self.login),
+                                         ('https://repo.example/elsewhere/example1.apk', None)])
+
+    def test_other_ports_and_http_never_get_the_login(self):
+        login = _credentials.basic(REPO, USER, PASSWORD)
+        for url in ('https://repo.example:8443/fdroid/repo/x', 'http://repo.example/fdroid/repo/x',
+                    'https://other.example/fdroid/repo/x', 'https://repo.example.evil/x'):
+            self.assertFalse(login.matches(url), url)
+        self.assertTrue(login.matches('https://REPO.example:443/anything'))
+        self.assertNoSecrets(repr(login))
+        handler = _credentials.Handler(login)
+        req = urllib.request.Request('https://other.example/x', headers={'Authorization': 'Basic stale'})
+        self.assertIsNone(handler.https_request(req).get_header('Authorization'))
+
+    def test_http_and_bad_logins_are_refused_before_any_request(self):
+        for url, kw in [('http://alice:pw@repo.example/fdroid/repo', {}),
+                        ('http://repo.example/fdroid/repo', {'username': USER, 'password': PASSWORD}),
+                        (REPO, {'username': 'a:b', 'password': PASSWORD}),
+                        (REPO, {'username': USER, 'password': 'x\r\nX-Evil: 1'}),
+                        (REPO, {'password': PASSWORD}),
+                        ('https://alice:pw@repo.example/fdroid/repo', {'username': 'bob'})]:
+            with self.subTest(url=url, kw=kw), self.assertRaises(SourceError) as caught:
+                fdroid.add_repo(url, PIN, **kw)
+            self.assertNotIn('pw', str(caught.exception).replace('password', ''))
+            self.assertNoSecrets(str(caught.exception))
+        self.assertEqual(self.web.seen, [])
+
+    def test_remove_deletes_the_login(self):
+        source = self.add()
+        other = _credentials.bearer('https://api.github.com/', TOKEN)
+        _credentials.put('github', other)
+        fdroid.remove_repo(source['id'])
+        self.assertFalse(_credentials.has(source['id']))
+        self.assertNotIn(PASSWORD, self.credentials_file().read_text())
+        _credentials.remove('github')
+        self.assertFalse(self.credentials_file().exists())  # nothing left: the file goes
+
+    def test_cli_reads_the_password_from_stdin(self):
+        out = io.StringIO()
+        argv = ['fdroid.py', 'add', REPO, '--fingerprint', PIN, '--username', USER, '--password-stdin']
+        with patch.object(sys, 'argv', argv), patch.object(sys, 'stdin', io.StringIO(PASSWORD + '\n')), \
+                contextlib.redirect_stdout(out):
+            fdroid.main()
+        self.assertEqual(json.loads(out.getvalue())['url'], REPO)
+        self.assertNoSecrets(out.getvalue())
+        self.assertTrue(_credentials.has(fdroid.user_repos()[0]['id']))
+
+    def test_artwork_from_the_repository_gets_the_login_and_only_there(self):
+        source = self.add()
+        self.assertEqual(_credentials.for_url(REPO + 'icons/a.png').header(), self.login)
+        self.assertTrue(_credentials.has(source['id']))
+        self.assertIsNone(_credentials.for_url('https://repo.example/other/a.png'))  # outside the repository
+        self.assertIsNone(_credentials.for_url('https://cdn.example/fdroid/repo/a.png'))
+
+
+class GitHubToken(Base):
+    def setUp(self):
+        super().setUp()
+        self.accept = {TOKEN}
+        self.releases = json.dumps([]).encode()
+        self.anonymous_limited = False
+
+    def route(self, url, auth):
+        if url.startswith('https://api.github.com/'):
+            if auth is not None and auth not in {'Bearer ' + t for t in self.accept}:
+                return 401, {}, b'{"message": "Bad credentials"}'
+            if auth is None and self.anonymous_limited:
+                return 403, {'X-RateLimit-Remaining': '0', 'X-RateLimit-Reset': str(int(time.time()) + 1800)}, b'{}'
+            if url.endswith('/redirected'):
+                return 302, {'Location': 'https://github.com/landed'}, b''
+            return 200, {}, self.releases
+        if url == 'https://github.com/landed':
+            return 200, {}, b'{"ok": true}'
+        raise AssertionError('unexpected request ' + url)
+
+    def test_saved_token_goes_only_to_the_api(self):
+        github.set_token(TOKEN)
+        self.assertEqual(github._api('/repos/a/b/releases'), [])
+        self.assertEqual(self.web.seen, [('https://api.github.com/repos/a/b/releases', 'Bearer ' + TOKEN)])
+        auth = github._credential(github._token()[0])
+        self.assertEqual(_web.read('https://api.github.com/redirected', ('api.github.com', 'github.com'), auth=auth),
+                         b'{"ok": true}')
+        self.assertEqual(self.web.seen[-2:], [('https://api.github.com/redirected', 'Bearer ' + TOKEN),
+                                              ('https://github.com/landed', None)])  # redirect: no token
+        status = github.sources()[0]['token']
+        self.assertEqual(status, {'set': True, 'from': 'saved', 'rejected': False})
+        self.assertNoSecrets(json.dumps(github.sources()), self.disk(), self.stderr.getvalue())
+        if os.name != 'nt':
+            self.assertEqual(stat.S_IMODE(self.credentials_file().stat().st_mode), 0o600)
+
+    def test_rejected_token_retries_once_without_it(self):
+        github.set_token(TOKEN)
+        self.accept = set()
+        self.assertEqual(github._api('/repos/a/b/releases'), [])
+        self.assertEqual([auth for _, auth in self.web.seen], ['Bearer ' + TOKEN, None])
+        self.assertTrue(github.token_status()['rejected'])
+        self.assertIn('rejected', self.stderr.getvalue())
+        self.assertNoSecrets(self.stderr.getvalue())
+        self.web.seen.clear()
+        github._api('/repos/a/b/other')
+        self.assertEqual([auth for _, auth in self.web.seen], [None])  # not tried again this run
+        github.set_token(TOKEN)  # saving it again gives it a fresh chance
+        self.assertFalse(github.token_status()['rejected'])
+
+    def test_environment_variable_wins(self):
+        github.set_token(TOKEN)
+        env = 'ghp_fromTheEnvironment'
+        self.accept = {env}
+        with patch.dict(os.environ, {github.ENV: env}):
+            github._api('/repos/a/b/releases')
+            self.assertEqual(self.web.seen[-1][1], 'Bearer ' + env)
+            self.assertEqual(github.token_status(), {'set': True, 'from': 'env', 'rejected': False})
+        self.assertEqual(github.token_status()['from'], 'saved')
+
+    def test_clear_removes_it_from_disk(self):
+        github.set_token(TOKEN)
+        github.clear_token()
+        self.assertFalse(self.credentials_file().exists())
+        self.assertEqual(github.token_status(), {'set': False, 'from': None, 'rejected': False})
+        github._api('/repos/a/b/releases')
+        self.assertEqual(self.web.seen[-1][1], None)
+
+    def test_saving_a_token_escapes_the_anonymous_rate_limit(self):
+        self.anonymous_limited = True
+        with self.assertRaisesRegex(SourceLimited, 'GitHub is limiting requests'):
+            github._api('/repos/a/b/releases')
+        self.assertEqual(self.web.seen[-1][1], None)
+        github.set_token(TOKEN)
+        self.assertEqual(github._api('/repos/a/b/other'), [])  # tried, with the token's own quota
+        self.assertEqual(self.web.seen[-1], ('https://api.github.com/repos/a/b/other', 'Bearer ' + TOKEN))
+        github.clear_token()
+        count = len(self.web.seen)
+        with self.assertRaisesRegex(SourceLimited, 'GitHub is limiting requests'):
+            github._api('/repos/a/b/third')
+        self.assertEqual(len(self.web.seen), count)  # anonymous is still backing off: no request
+
+    def test_bad_tokens_are_refused(self):
+        for token in ('', '   ', 'two words', 'x\nAuthorization: y'):
+            with self.subTest(token=token), self.assertRaises(SourceError):
+                github.set_token(token)
+        self.assertFalse(self.credentials_file().exists())
+
+
+PNG = b'\x89PNG\r\n\x1a\nfixture'
+PRIVATE = 'https://repo.example/private/repo/'
+TRAVERSALS = [PRIVATE + '%2e%2e/%2e%2e/evil/collect.png', PRIVATE + '%2E%2E/x.png', PRIVATE + '../../evil/x.png',
+              PRIVATE + './icons/a.png', PRIVATE + '..%2f..%2fevil/x.png', PRIVATE + 'a%5c..%5c..%5cevil.png',
+              PRIVATE + 'a;b/../../evil.png', PRIVATE + '%252e%252e/%252e%252e/evil.png',
+              'https://repo.example/private/repository/x.png', 'https://repo.example/private/repo%2fx.png']
+
+
+class ArtworkScope(Base):
+    """The artwork proxy: actual outgoing headers, through _images.get's own HTTP code."""
+
+    def setUp(self):
+        super().setUp()
+        self.sent, self.redirects = [], {}
+        _credentials.put('fdroid-user-private', _credentials.basic(PRIVATE, USER, PASSWORD))
+        test = self
+
+        class Response:
+            def __init__(self, status, location=''):
+                self.status, self.location, self.body = status, location, PNG if status == 200 else b''
+
+            def getheader(self, name, default=''):
+                return self.location if name == 'Location' else default
+
+            def read1(self, n):
+                chunk, self.body = self.body[:n], self.body[n:]
+                return chunk
+
+        class Connection:
+            def __init__(self, host, port, timeout=None):
+                self.host, self.port = host, port
+
+            def request(self, method, path, headers):
+                url = 'https://%s%s' % (self.host, path)
+                test.sent.append((url, headers.get('Authorization')))
+                self.url = url
+
+            def getresponse(self):
+                if self.url in test.redirects:
+                    return Response(302, test.redirects[self.url])
+                return Response(200)
+
+            def close(self):
+                pass
+
+        # A public address, no real socket or TLS, and Connection in place of http.client's.
+        for target, name, value in [(_images, '_resolve', lambda host, port, timeout: [(2, 1, 6, '', ('93.184.216.34', port))]),
+                                    (_images.socket, 'create_connection', lambda *a, **k: Mock()),
+                                    (_images.ssl, 'create_default_context', lambda: Mock()),
+                                    (_images.http.client, 'HTTPConnection', Connection)]:
+            mock = patch.object(target, name, value)
+            mock.start()
+            self.addCleanup(mock.stop)
+
+    def test_artwork_in_the_repository_gets_the_login(self):
+        self.assertEqual(_images.get(PRIVATE + 'icons/a%20b.png'), PNG)
+        self.assertEqual(self.sent, [(PRIVATE + 'icons/a%20b.png', basic(USER, PASSWORD))])
+
+    def test_paths_a_server_could_normalise_outside_the_repository_get_nothing(self):
+        for url in TRAVERSALS:
+            with self.subTest(url=url):
+                self.sent.clear()
+                self.assertEqual(_images.get(url), PNG)
+                self.assertEqual(self.sent, [(url, None)])
+                self.assertIsNone(_credentials.for_url(url))
+
+    def test_redirect_to_a_traversal_gets_nothing(self):
+        start = PRIVATE + 'icons/a.png'
+        self.redirects[start] = PRIVATE + '%2e%2e/%2e%2e/evil/collect.png'
+        self.assertEqual(_images.get(start), PNG)
+        self.assertEqual(self.sent, [(start, basic(USER, PASSWORD)), (self.redirects[start], None)])
+
+
+class ScopedHandler(Base):
+    """Handler(scoped=True), for callers (artwork-like) that must stay inside a scope, redirects included."""
+
+    def route(self, url, auth):
+        if url == PRIVATE + 'icons/a.png':
+            return 302, {'Location': PRIVATE + '%2e%2e/%2e%2e/evil/collect.png'}, b''
+        return 200, {}, PNG
+
+    def test_redirect_into_a_traversal_is_stripped(self):
+        login = _credentials.basic(PRIVATE, USER, PASSWORD)
+        opener = urllib.request.build_opener(_credentials.Handler(login, scoped=True))
+        with opener.open(PRIVATE + 'icons/a.png') as r:
+            self.assertEqual(r.read(), PNG)
+        self.assertEqual(self.web.seen, [(PRIVATE + 'icons/a.png', basic(USER, PASSWORD)),
+                                         (PRIVATE + '%2e%2e/%2e%2e/evil/collect.png', None)])
+        for url in TRAVERSALS:
+            self.assertFalse(_credentials.in_scope(login, url), url)
+        self.assertTrue(_credentials.in_scope(login, PRIVATE + 'a/b%20c.png?x=1'))
+        self.assertTrue(_credentials.in_scope(login, PRIVATE.rstrip('/')))
+
+
+CHURN = '''
+import sys
+sys.path.insert(0, sys.argv[1])
+from apk_sources import _credentials
+for i in range(int(sys.argv[3])):
+    key = sys.argv[2] + str(i)
+    _credentials.put(key, _credentials.basic('https://h.example/', 'u', 'p'))
+    if i % 2:
+        _credentials.remove(key)
+'''
+
+
+class TwoProcesses(Base):
+    def test_interleaved_saves_and_removals_are_not_lost(self):
+        ui = str(Path(__file__).resolve().parents[1] / 'ui')
+        env = dict(os.environ, FRAME_CONTROL_DATA_DIR=str(self.root / 'data'))
+        count = 60
+        procs = [subprocess.Popen([sys.executable, '-c', CHURN, ui, prefix, str(count)], env=env,
+                                  stdout=subprocess.PIPE, stderr=subprocess.PIPE) for prefix in ('a', 'b')]
+        for p in procs:
+            out, err = p.communicate(timeout=120)
+            self.assertEqual(p.returncode, 0, err.decode())
+        saved = set(json.loads(self.credentials_file().read_text()))
+        self.assertEqual(saved, {p + str(i) for p in 'ab' for i in range(0, count, 2)})
+        if os.name != 'nt':
+            self.assertEqual(stat.S_IMODE(self.credentials_file().stat().st_mode), 0o600)
+
+
+@unittest.skipUnless(shutil.which('node'), 'needs node')
+class PageLogin(unittest.TestCase):
+    def test_login_leaves_the_address_field_before_sending(self):
+        """tests/page/source_login.mjs runs the real page functions from index.html."""
+        r = subprocess.run(['node', str(Path(__file__).parent / 'page' / 'source_login.mjs')],
+                           capture_output=True, text=True)
+        self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
+        self.assertIn('ok', r.stdout)
+
+
+class Endpoints(Repository):
+    """The server API: logins in, never out, in responses or the server log."""
+
+    def setUp(self):
+        super().setUp()
+        mock = patch.object(search, 'modules', return_value=([fdroid, github], []))
+        mock.start()
+        self.addCleanup(mock.stop)
+        self.httpd = server.ThreadingHTTPServer(('127.0.0.1', 0), server.Handler)
+        threading.Thread(target=self.httpd.serve_forever, daemon=True).start()
+        self.addCleanup(self.httpd.server_close)
+        self.addCleanup(self.httpd.shutdown)
+
+    def request(self, method, path, body=None):
+        c = http.client.HTTPConnection('127.0.0.1', self.httpd.server_port)
+        c.request(method, path, json.dumps(body) if body is not None else None,
+                  {'X-Frame-UI': '1', 'Content-Type': 'application/json'})
+        r = c.getresponse()
+        raw = r.read().decode()
+        c.close()
+        self.replies.append(raw)
+        return r.status, json.loads(raw)
+
+    def job(self, reply):
+        for _ in range(500):
+            job = self.request('GET', '/api/job?id=' + reply['job'])[1]
+            if job['done']:
+                return job
+            time.sleep(.01)
+        self.fail('job did not finish')
+
+    def test_add_with_login_and_token_settings(self):
+        self.replies = []
+        bad = self.job(self.request('POST', '/api/sources', {'action': 'add', 'url': REPO, 'fingerprint': PIN,
+                                                             'username': USER, 'password': 'wrong'})[1])
+        self.assertIn('check the username and password', bad['error'])
+        self.assertEqual(fdroid.user_repos(), [])
+        code, reply = self.request('POST', '/api/sources', {'action': 'add', 'url': REPO, 'fingerprint': PIN,
+                                                            'username': USER, 'password': PASSWORD})
+        self.assertEqual(code, 200)
+        job = self.job(reply)
+        self.assertIsNone(job['error'])
+        listed = self.request('GET', '/api/sources')[1]['sources']
+        repo = next(s for s in listed if s['id'] == job['result']['source']['id'])
+        self.assertEqual((repo['url'], repo['login']), (REPO, True))
+        self.assertEqual(self.request('POST', '/api/sources', {'action': 'set-token', 'source': 'github',
+                                                               'token': TOKEN})[0], 200)
+        github_row = next(s for s in self.request('GET', '/api/sources')[1]['sources'] if s['id'] == 'github')
+        self.assertEqual(github_row['token'], {'set': True, 'from': 'saved', 'rejected': False})
+        self.assertEqual(self.request('POST', '/api/sources', {'action': 'set-token', 'source': 'fdroid',
+                                                               'token': TOKEN})[0], 400)
+        self.assertEqual(self.request('POST', '/api/sources', {'action': 'clear-token', 'source': 'github'})[0], 200)
+        self.assertFalse(_credentials.has('github'))
+        self.assertEqual(self.request('POST', '/api/sources', {'action': 'remove', 'source': repo['id']})[0], 200)
+        self.assertFalse(self.credentials_file().exists())
+        self.assertNoSecrets(*self.replies, self.stderr.getvalue(), self.disk())
+
+    def test_login_in_the_url_is_accepted(self):
+        self.replies = []
+        url = 'https://alice:s3cr%40t-p%C3%A4ssword@repo.example/fdroid/repo'
+        job = self.job(self.request('POST', '/api/sources', {'action': 'add', 'url': url, 'fingerprint': PIN})[1])
+        self.assertIsNone(job['error'])
+        self.assertEqual(fdroid.user_repos()[0]['url'], REPO)
+        self.assertNoSecrets(*self.replies, self.stderr.getvalue())
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_touch.py

@@ -2,6 +2,7 @@
 
 Run: python3 -m unittest discover -s tests
 """
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import io
 import json
 import os

tests/test_vr.py

@@ -1,4 +1,5 @@
 """Own VR telemetry, optional catalogue and entitlement guards; no headset needed."""
+import sandbox  # noqa: F401  (first: keeps tests off real data and services)
 import ctypes
 import json
 from pathlib import Path

ui/apk_sources/__init__.py

@@ -45,6 +45,10 @@ class SourceError(Exception):
     """User-readable failure from a source (network, format, verification)."""
 
 
+class SourceAuthError(SourceError):
+    """The server refused our login (HTTP 401, or 403 when a login was sent)."""
+
+
 class SourceLimited(SourceError):
     """The source's host asked us to slow down; retry_after is in seconds."""
 

ui/apk_sources/_credentials.py

@@ -0,0 +1,270 @@
+"""Logins for sources behind one: HTTP basic auth and access tokens.
+
+Saved apart from the source settings, in frame_host.data_dir('source-credentials.json')
+(mode 0600), keyed by a caller-chosen name: an F-Droid source id, 'github'. Each
+credential belongs to one exact origin (https, host, port) and is only ever sent
+to a request for that origin: Handler adds it to each request, redirects
+included, and removes any Authorization header from every other request.
+
+Scoped use (for_url, and Handler(..., scoped=True), which F-Droid repositories and
+artwork use) is stricter: the URL must also be under the credential's scope by whole
+path segments, with no dot segments, encoded slashes or ';' a server could normalise
+into a path outside it. Origin-only matching suits a host-wide token (GitHub's API).
+
+Secrets leave this module only as an Authorization header value. Credential's
+repr hides them, and nothing here puts them in an exception message.
+
+Each change is a read-modify-write under an inter-process lock (the CLI and the
+app may both save), so one process can't undo another's save or removal.
+"""
+import base64
+import contextlib
+import errno
+import json
+import os
+import tempfile
+import threading
+import urllib.parse
+import urllib.request
+
+import frame_host
+
+_lock = threading.RLock()
+MAX = 4096  # longest username, password or token accepted
+
+
+def _path():
+    return frame_host.data_dir('source-credentials.json')
+
+
+@contextlib.contextmanager
+def _changing():
+    """This thread, then this process, holds the credentials file for a read-modify-write."""
+    path = _path().with_name('source-credentials.lock')
+    with _lock:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        with open(str(path), 'a+b') as f:
+            if os.name == 'nt':
+                import msvcrt
+                while True:
+                    try:
+                        f.seek(0)
+                        msvcrt.locking(f.fileno(), msvcrt.LK_LOCK, 1)
+                        break
+                    except OSError as e:  # LK_LOCK gives up after ~10 s of contention; keep waiting
+                        if e.errno not in (errno.EACCES, errno.EDEADLK):
+                            raise
+                try:
+                    yield
+                finally:
+                    f.seek(0)
+                    msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, 1)
+            else:
+                import fcntl
+                fcntl.flock(f.fileno(), fcntl.LOCK_EX)
+                try:
+                    yield
+                finally:
+                    fcntl.flock(f.fileno(), fcntl.LOCK_UN)
+
+
+def path_segments(path):
+    """A URL path's decoded segments, or None if a server might read it as another path:
+    '.' or '..' (plain or percent-encoded), encoded '/' or '\\', '%' left after decoding
+    (double encoding), or ';' path parameters."""
+    if ';' in path or '\\' in path:
+        return None
+    parts = [urllib.parse.unquote(part) for part in path.split('/')]
+    if any(part in ('.', '..') or '/' in part or '\\' in part or '%' in part or ';' in part for part in parts):
+        return None
+    return parts
+
+
+def in_scope(credential, url):
+    """True if url is on credential's origin and under its scope by whole, plain path segments."""
+    if credential is None or not credential.matches(url):
+        return False
+    try:
+        path, scope = urllib.parse.urlsplit(url).path, urllib.parse.urlsplit(credential.scope).path
+    except ValueError:
+        return False
+    parts, base = path_segments(path), path_segments(scope)
+    if parts is None or base is None:
+        return False
+    if base and base[-1] == '':
+        base = base[:-1]  # 'https://h/repo/' covers '/repo/x', not '/repository'
+    return parts[:len(base)] == base
+
+
+def origin(url):
+    """'https://host:port' for an https URL; ValueError for anything else."""
+    p = urllib.parse.urlsplit(url)
+    port = p.port  # raises ValueError on a bad port
+    if p.scheme.lower() != 'https' or not p.hostname:
+        raise ValueError('credentials are only sent over HTTPS')
+    host = p.hostname.lower()
+    return 'https://%s:%d' % ('[%s]' % host if ':' in host else host, port or 443)
+
+
+def _check(value, what, allow_empty=False):
+    if not isinstance(value, str) or (not value and not allow_empty) or len(value) > MAX:
+        raise ValueError('invalid ' + what)
+    if any(ord(c) < 32 or ord(c) == 127 for c in value):
+        raise ValueError(what + ' must not contain control characters')
+    return value
+
+
+class Credential:
+    """A username and password (basic) or a token (bearer) for one origin.
+
+    scope is the URL prefix it belongs under: in_scope checks it (by path segments),
+    matches() checks only the origin."""
+    __slots__ = ('origin', 'scope', 'username', '_secret')
+
+    def __init__(self, scope, secret, username=None):
+        self.origin = origin(scope)
+        self.scope = scope
+        if username is not None:
+            _check(username, 'username')
+            if ':' in username:
+                raise ValueError('username must not contain ":"')
+            _check(secret, 'password', allow_empty=True)
+        else:
+            _check(secret, 'token')
+            if any(c.isspace() for c in secret):
+                raise ValueError('token must not contain spaces')
+        self.username, self._secret = username, secret
+
+    @property
+    def kind(self):
+        return 'bearer' if self.username is None else 'basic'
+
+    def header(self):
+        if self.username is None:
+            return 'Bearer ' + self._secret
+        return 'Basic ' + base64.b64encode((self.username + ':' + self._secret).encode('utf-8')).decode('ascii')
+
+    def matches(self, url):
+        """True only for a URL on exactly this credential's origin."""
+        try:
+            return origin(url) == self.origin
+        except ValueError:
+            return False
+
+    def same(self, other):
+        return isinstance(other, Credential) and (self.origin, self.username, self._secret) == (
+            other.origin, other.username, other._secret)
+
+    def __repr__(self):
+        return '<Credential %s for %s>' % (self.kind, self.origin)
+
+    def _json(self):
+        return {'scope': self.scope, 'username': self.username, 'secret': self._secret}
+
+
+def basic(scope, username, password):
+    """Basic auth for scope's origin, e.g. basic('https://example.org/fdroid/repo/', 'me', 'pw')."""
+    return Credential(scope, password, username)
+
+
+def bearer(scope, token):
+    """An access token for scope's origin, sent as 'Authorization: Bearer <token>'."""
+    return Credential(scope, token)
+
+
+def _read():
+    try:
+        with open(str(_path()), encoding='utf-8') as f:
+            data = json.load(f)
+        if not isinstance(data, dict):
+            raise ValueError
+        return data
+    except FileNotFoundError:
+        return {}
+    except (OSError, ValueError):
+        return {}  # unreadable: as if none were saved; the next save rewrites it
+
+
+def _save(data):
+    path = _path()
+    if not data:
+        try:
+            os.remove(str(path))
+        except FileNotFoundError:
+            pass
+        return
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix='.credentials-', suffix='.part')  # created 0600
+    try:
+        with os.fdopen(fd, 'w', encoding='utf-8') as f:
+            json.dump(data, f, separators=(',', ':'))
+        os.chmod(tmp, 0o600)
+        os.replace(tmp, str(path))
+    finally:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+
+
+def get(key):
+    """The saved Credential for key, or None."""
+    with _lock:
+        item = _read().get(key)
+    try:
+        return Credential(item['scope'], item['secret'], item.get('username'))
+    except (TypeError, KeyError, ValueError):
+        return None
+
+
+def has(key):
+    return get(key) is not None
+
+
+def put(key, credential):
+    if not isinstance(credential, Credential):
+        raise TypeError('put() needs a Credential')
+    with _changing():
+        data = _read()
+        data[key] = credential._json()
+        _save(data)
+
+
+def remove(key):
+    """Delete key's credential from disk; True if there was one."""
+    with _changing():
+        data = _read()
+        if key not in data:
+            return False
+        del data[key]
+        _save(data)
+        return True
+
+
+def for_url(url):
+    """A saved credential url is in the scope of (see in_scope), or None. Call it again for
+    each redirect hop: the answer depends only on the URL being requested."""
+    with _lock:
+        keys = list(_read())
+    for key in keys:
+        credential = get(key)
+        if in_scope(credential, url):
+            return credential
+    return None
+
+
+class Handler(urllib.request.BaseHandler):
+    """Adds the credential to requests for its origin (redirects included) and strips
+    Authorization from every other request, so a redirect elsewhere never gets it.
+    scoped=True also requires each request, redirects included, to be in_scope."""
+    handler_order = 900  # after the default processors
+
+    def __init__(self, credential, scoped=False):
+        self.credential, self.scoped = credential, scoped
+
+    def _request(self, req):
+        req.remove_header('Authorization')
+        allowed = in_scope if self.scoped else lambda c, url: c is not None and c.matches(url)
+        if allowed(self.credential, req.full_url):
+            req.add_unredirected_header('Authorization', self.credential.header())
+        return req
+
+    http_request = https_request = _request

ui/apk_sources/_images.py

@@ -9,7 +9,7 @@ import threading
 import time
 from urllib.parse import urljoin, urlsplit
 
-from apk_sources import SourceError
+from apk_sources import SourceError, _credentials
 
 _lock = threading.Lock()
 _urls = OrderedDict()
@@ -141,7 +141,11 @@ def get(url, headers=None, redirects=3, deadline=None, limit=MAX_IMAGE):
         path = p.path or '/'
         if p.query:
             path += '?' + p.query
-        conn.request('GET', path, headers={'User-Agent': 'FrameControl/0.3.1', **(headers or {})})
+        sent = {'User-Agent': 'FrameControl/0.3.1', **(headers or {})}
+        login = _credentials.for_url(url)  # a repository behind a login: its own artwork only, per hop
+        if login:
+            sent['Authorization'] = login.header()
+        conn.request('GET', path, headers=sent)
         live[0].settimeout(min(10, left()))
         response = conn.getresponse()
         if response.status in (301, 302, 303, 307, 308) and redirects:

ui/apk_sources/_web.py

@@ -3,7 +3,7 @@ import hashlib, os, shutil, tempfile, threading, time, urllib.error, urllib.pars
 from email.utils import parsedate_to_datetime
 
 import frame_host
-from . import SourceError, SourceLimited
+from . import SourceAuthError, SourceError, SourceLimited, _credentials
 
 UA = 'FrameControl/0.1'
 APK_CAP = 2 * 1024 ** 3  # cached APKs across all sources, least recently used go first
@@ -38,7 +38,7 @@ def release(path):
         else:
             _in_use.pop(path, None)
 BACKOFF = 600  # seconds to leave a host alone after 403/429 without Retry-After
-_limited = {}  # host -> time.time() before which we don't contact it
+_limited = {}  # host (+ login identity) -> time.time() before which we don't ask it with that login
 _limited_lock = threading.Lock()
 
 
@@ -46,8 +46,16 @@ def _host(url):
     return (urllib.parse.urlsplit(url).hostname or '').lower()
 
 
-def throttle(url, headers=None):
-    """Remember that url's host asked us to back off; return the delay in seconds."""
+def _who(url, auth=None):
+    """Backoff key: the host, per login. A token has its own quota, so anonymous
+    rate limiting mustn't hold back requests made with one (or the reverse)."""
+    if auth is None:
+        return _host(url)
+    return _host(url) + ' ' + hashlib.sha256(auth.header().encode()).hexdigest()[:16]
+
+
+def throttle(url, headers=None, auth=None):
+    """Remember that url's host asked us (with this login) to back off; return the delay in seconds."""
     headers = headers or {}
     now, delay = time.time(), None
     value = (headers.get('Retry-After') or '').strip()
@@ -62,14 +70,15 @@ def throttle(url, headers=None):
     if delay is None and headers.get('X-RateLimit-Remaining') == '0' and reset.isdigit():
         delay = int(reset) - now  # GitHub
     delay = min(max(delay if delay is not None else BACKOFF, 1), 6 * 3600)
+    key = _who(url, auth)
     with _limited_lock:
-        _limited[_host(url)] = max(_limited.get(_host(url), 0), now + delay)
+        _limited[key] = max(_limited.get(key, 0), now + delay)
     return delay
 
 
-def wait_time(url):
+def wait_time(url, auth=None):
     with _limited_lock:
-        return max(0, _limited.get(_host(url), 0) - time.time())
+        return max(0, _limited.get(_who(url, auth), 0) - time.time())
 
 
 def limited_error(name, seconds, hint=''):
@@ -136,6 +145,8 @@ def checked_url(url, hosts):
 
 
 class Redirect(urllib.request.HTTPRedirectHandler):
+    """Only to the allowed hosts, and never carrying an Authorization header along:
+    a credential passed to open_url() is added again, by its Handler, only on its origin."""
     def __init__(self, hosts):
         self.hosts = hosts
 
@@ -147,13 +158,15 @@ class Redirect(urllib.request.HTTPRedirectHandler):
         return result
 
 
-def open_url(url, hosts, headers=None):
+def open_url(url, hosts, headers=None, auth=None):
+    """auth: an _credentials.Credential, sent only to requests on its own origin."""
     checked_url(url, hosts)
-    return urllib.request.build_opener(Redirect(hosts)).open(
+    handlers = [Redirect(hosts)] + ([_credentials.Handler(auth)] if auth else [])
+    return urllib.request.build_opener(*handlers).open(
         urllib.request.Request(url, headers={'User-Agent': UA, **(headers or {})}), timeout=60)
 
 
-def read(url, hosts, headers=None, ttl=3600, name=None, hint=''):
+def read(url, hosts, headers=None, ttl=3600, name=None, hint='', auth=None):
     path = os.path.join(cache(), hashlib.sha256(url.encode()).hexdigest() + '.data')
     name = name or _host(url) or 'The source'
 
@@ -165,13 +178,13 @@ def read(url, hosts, headers=None, ttl=3600, name=None, hint=''):
         if os.path.isfile(path) and time.time() - os.path.getmtime(path) < ttl:
             with open(path, 'rb') as f:
                 return f.read()
-        wait = wait_time(url)
+        wait = wait_time(url, auth)
         if wait:
             data = cached()
             if data is None:
                 raise limited_error(name, wait, hint)
             return data
-        with open_url(url, hosts, headers) as r:
+        with open_url(url, hosts, headers, auth) as r:
             data = r.read(8 * 1024 * 1024 + 1)
         if len(data) > 8 * 1024 * 1024:
             raise SourceError('Source index is too large')
@@ -185,8 +198,10 @@ def read(url, hosts, headers=None, ttl=3600, name=None, hint=''):
                 os.remove(tmp)
         return data
     except urllib.error.HTTPError as e:
+        if e.code == 401:  # not 403: GitHub also uses that for rate limits, token or not
+            raise SourceAuthError('%s refused the login (HTTP 401)' % name) from e
         if e.code in (403, 429):
-            delay = throttle(url, e.headers)
+            delay = throttle(url, e.headers, auth)
             data = cached()
             if data is None:
                 raise limited_error(name, delay, hint) from e

ui/apk_sources/fdroid.py

@@ -20,7 +20,7 @@ import zipfile
 
 if __package__ in (None, ''):
     sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-from apk_sources import SourceError, SourceLimited, _web
+from apk_sources import SourceAuthError, SourceError, SourceLimited, _credentials, _web
 import frame_host
 from frame_apk_sign import _der_parts, _cert_key, der
 from frame_catalog import _IndexReader, _reduce_index, _sha256
@@ -71,6 +71,39 @@ def _url(url, fingerprint=None):
     return urllib.parse.urlunsplit(('https', p.netloc.lower(), p.path.rstrip('/') + '/', '', '')), pin
 
 
+def _split_login(url, username=None, password=None):
+    """(url without user:pass, credential or None). The address may carry the login, as
+    F-Droid's client accepts (https://user:pass@host/fdroid/repo), or it may be given apart."""
+    try:
+        p = urllib.parse.urlsplit(url)
+        p.port
+    except ValueError as e:
+        raise SourceError('invalid repository URL') from e
+    if '@' in p.netloc:
+        if p.username is None:
+            raise SourceError('invalid repository URL')
+        linked = urllib.parse.unquote(p.username), None if p.password is None else urllib.parse.unquote(p.password)
+        if username not in (None, '', linked[0]) or (linked[1] is not None and password not in (None, '', linked[1])):
+            raise SourceError('the address and the username/password fields disagree')
+        username = linked[0]
+        password = linked[1] if linked[1] is not None else password
+        url = urllib.parse.urlunsplit((p.scheme, p.netloc.rpartition('@')[2], p.path, p.query, p.fragment))
+    if username in (None, ''):
+        if password:
+            raise SourceError('a password needs a username')
+        return url, None
+    clean, _ = _url(url)
+    try:
+        return url, _credentials.basic(clean, username, password or '')
+    except ValueError as e:
+        raise SourceError('cannot use that login: ' + str(e)) from None
+
+
+def _login(source):
+    """The saved login for a user repository, or None."""
+    return None if source.get('builtin') else _credentials.get(source['id'])
+
+
 def _child(base, name):
     name = str(name).lstrip('/')
     decoded = urllib.parse.unquote(name)
@@ -81,6 +114,9 @@ def _child(base, name):
     url = urllib.parse.urljoin(base, urllib.parse.quote(decoded, safe='/'))  # names may contain '#' or spaces
     if not url.startswith(base) or urllib.parse.urlsplit(url).query or urllib.parse.urlsplit(url).fragment:
         raise SourceError('repository file is outside its repository')
+    # Nothing a server could normalise elsewhere: double-encoded dots, ';' parameters, encoded slashes.
+    if _credentials.path_segments(urllib.parse.urlsplit(url).path) is None:
+        raise SourceError('unsafe repository file name')
     return url
 
 
@@ -91,14 +127,17 @@ class _HTTPSRedirect(urllib.request.HTTPRedirectHandler):
         return super().redirect_request(req, fp, code, msg, headers, newurl)
 
 
-def _fetch(url, path, maximum):
+def _fetch(url, path, maximum, auth=None):
+    """auth: the repository's login, sent only inside the repository (its origin and directory,
+    by whole plain path segments), on every request including redirects."""
     host = urllib.parse.urlsplit(url).hostname
-    wait = _web.wait_time(url)
+    wait = _web.wait_time(url, auth)
     if wait:
         raise _web.limited_error(host, wait)
     request = urllib.request.Request(url, headers={'User-Agent': 'FrameControl/1.0'})
+    handlers = [_HTTPSRedirect()] + ([_credentials.Handler(auth, scoped=True)] if auth else [])
     try:
-        with urllib.request.build_opener(_HTTPSRedirect()).open(request, timeout=60) as r, open(path, 'wb') as f:
+        with urllib.request.build_opener(*handlers).open(request, timeout=60) as r, open(path, 'wb') as f:
             total = 0
             while True:
                 chunk = r.read(1 << 20)
@@ -109,8 +148,11 @@ def _fetch(url, path, maximum):
                     raise SourceError('repository file exceeds size limit')
                 f.write(chunk)
     except urllib.error.HTTPError as e:
+        if e.code == 401 or (e.code == 403 and auth):
+            raise SourceAuthError('The repository refused the login (HTTP %d); check the username and password'
+                                  % e.code if auth else 'This repository needs a username and password') from None
         if e.code in (403, 429):
-            raise _web.limited_error(host, _web.throttle(url, e.headers)) from e
+            raise _web.limited_error(host, _web.throttle(url, e.headers, auth)) from e
         raise
 
 
@@ -342,7 +384,7 @@ def sources():
     return [dict(id=i, kind=KIND, name=n, url=u, fingerprint=p, builtin=True,
                  # The archive only holds superseded versions; it clutters search unless asked for.
                  enabled=settings['enabled'].get(i, i != 'fdroid-archive'), trust='community')
-            for i, n, u, p in builtins] + settings['repos']
+            for i, n, u, p in builtins] + [dict(s, login=_credentials.has(s['id'])) for s in settings['repos']]
 
 
 def _text(value):
@@ -514,10 +556,12 @@ def _source_lock(source_id):
         return _load_locks.setdefault(source_id, threading.Lock())
 
 
-def _load(source, force=False):
+def _load(source, force=False, auth=None):
+    """auth: a login to use instead of the saved one (while adding, before anything is saved)."""
     if not re.fullmatch(r'[a-z0-9-]+', source['id']):
         raise SourceError('invalid source id')
     _url(source['url'], source.get('fingerprint'))
+    auth = auth or _login(source)
     cache = frame_host.cache_dir('apk-sources', source['id'] + '.json')
     found = None if force else _cached(source, cache)
     if found:
@@ -535,7 +579,7 @@ def _load(source, force=False):
                 jar, raw = Path(tmp) / 'index.jar', Path(tmp) / 'index.json'
                 v2 = True
                 try:
-                    _fetch(source['url'] + 'entry.jar', jar, 8 * 1024 * 1024)
+                    _fetch(source['url'] + 'entry.jar', jar, 8 * 1024 * 1024, auth=auth)
                 except urllib.error.HTTPError as e:
                     if e.code not in (404, 410):
                         raise
@@ -543,7 +587,7 @@ def _load(source, force=False):
                         raise SourceError('repository no longer serves its signed v2 index; '
                                           'refusing to fall back to the older v1 index') from e
                     v2 = False
-                    _fetch(source['url'] + 'index-v1.jar', jar, 256 * 1024 * 1024)
+                    _fetch(source['url'] + 'index-v1.jar', jar, 256 * 1024 * 1024, auth=auth)
                     content, pin = _jar(jar, 'index-v1.json', source.get('fingerprint'))
                     timestamp = _v1(content, raw)
                     _check_timestamp(source, timestamp)
@@ -552,7 +596,7 @@ def _load(source, force=False):
                     signed = json.loads(content)
                     timestamp, entry = signed.get('timestamp'), signed['index']
                     _check_timestamp(source, timestamp)
-                    _fetch(_child(source['url'], entry['name']), raw, 512 * 1024 * 1024)
+                    _fetch(_child(source['url'], entry['name']), raw, 512 * 1024 * 1024, auth=auth)
                     if _sha256(raw) != entry['sha256'] or (entry.get('size') is not None and raw.stat().st_size != entry['size']):
                         raise SourceError('index SHA-256 or size mismatch')
                 apps = _reduce(raw, source)
@@ -572,7 +616,10 @@ def _load(source, force=False):
             raise SourceError('cannot load repository: ' + str(e)) from e
 
 
-def add_repo(url, fingerprint=None, name=None):
+def add_repo(url, fingerprint=None, name=None, username=None, password=None):
+    """Check and save a repository. A login (in the URL or apart) is kept out of the
+    saved URL, in _credentials, and saved only once the repository's index verified."""
+    url, login = _split_login(url, username, password)
     url, pin = _url(url, fingerprint)
     with _LOCK:
         existing = next((s for s in _read()['repos'] if s['url'] == url), None)
@@ -583,15 +630,26 @@ def add_repo(url, fingerprint=None, name=None):
     source = dict(id='fdroid-user-' + hashlib.sha256(url.encode()).hexdigest()[:20], kind=KIND,
                   name=name or (existing or {}).get('name') or urllib.parse.urlsplit(url).hostname,
                   url=url, builtin=False, enabled=True, trust='user', fingerprint=pin)
-    _, source['fingerprint'] = _load(source, force=True)  # network work outside the settings lock
+    # Network work outside the settings lock; without a new login, a saved one (re-adding) is used.
+    _, source['fingerprint'] = _load(source, force=True, auth=login)
     source['trust_on_first_use'] = existing.get('trust_on_first_use', False) if existing else pin is None
     with _LOCK:
         settings = _read()
         current = next((s for s in settings['repos'] if s['id'] == source['id']), None)
         if current and current['fingerprint'] != source['fingerprint']:
             raise SourceError('repository already has a different pinned fingerprint; remove it first')
-        settings['repos'] = [s for s in settings['repos'] if s['id'] != source['id']] + [source]
-        _write(_storage(), settings)
+        previous = _credentials.get(source['id'])
+        if login:
+            _credentials.put(source['id'], login)
+        try:
+            settings['repos'] = [s for s in settings['repos'] if s['id'] != source['id']] + [source]
+            _write(_storage(), settings)
+        except BaseException:
+            if login and previous:  # not saved after all: leave the login as it was
+                _credentials.put(source['id'], previous)
+            elif login:
+                _credentials.remove(source['id'])
+            raise
     return source
 
 
@@ -602,6 +660,7 @@ def remove_repo(source_id):
             raise SourceError('unknown user repository')
         settings['repos'] = [s for s in settings['repos'] if s['id'] != source_id]
         _write(_storage(), settings)
+        _credentials.remove(source_id)
     with _state_file_lock(), _LOCK:  # lock order: state file, then _LOCK
         states = _states()
         if states.pop(source_id, None) is not None:  # re-adding is a deliberate new trust decision
@@ -663,7 +722,7 @@ def download(source, entry_id, version_code=None):
             fd, tmp = tempfile.mkstemp(dir=str(path.parent), suffix='.part')
             os.close(fd)
             try:
-                _fetch(_child(source['url'], version['name']), tmp, 4 * 1024 ** 3)
+                _fetch(_child(source['url'], version['name']), tmp, 4 * 1024 ** 3, auth=_login(source))
                 if _sha256(tmp) != sha:
                     raise SourceError('APK SHA-256 mismatch; download discarded')
                 os.replace(tmp, path)
@@ -684,6 +743,8 @@ def main():
     add.add_argument('url')
     add.add_argument('--fingerprint')
     add.add_argument('--name')
+    add.add_argument('--username', help='for a repository behind a login (HTTP basic auth)')
+    add.add_argument('--password-stdin', action='store_true', help='read the password from the first line of stdin')
     sub.add_parser('list')
     remove = sub.add_parser('remove')
     remove.add_argument('source')
@@ -702,7 +763,12 @@ def main():
 def _cli(parser, args):
     try:
         if args.command == 'add':
-            result = add_repo(args.url, args.fingerprint, args.name)
+            password = None
+            if args.password_stdin:
+                if not args.username:
+                    raise SourceError('--password-stdin needs --username')
+                password = sys.stdin.readline().rstrip('\r\n')
+            result = add_repo(args.url, args.fingerprint, args.name, args.username, password)
         elif args.command == 'list':
             result = sources()
         elif args.command == 'remove':

ui/apk_sources/github.py

@@ -1,17 +1,77 @@
 """Curated publisher releases; optional topic discovery is page-only."""
-import fnmatch, json, os, re, urllib.parse
+import fnmatch, hashlib, json, os, re, sys, threading, urllib.parse
 
-from . import SourceError, _web
+from . import SourceAuthError, SourceError, _credentials, _web
 
 KIND = 'github'
 API = 'https://api.github.com'
+ENV = 'FRAME_GITHUB_TOKEN'  # wins over a token saved in source settings
+CREDENTIAL = 'github'  # _credentials key; only ever sent to api.github.com
+_rejected = set()  # SHA-256 of tokens GitHub answered 401 to (this run only)
+_rejected_lock = threading.Lock()
 TOPICS = ('oculus-quest', 'openxr', 'quest')
 HOSTS = ('github.com', 'release-assets.githubusercontent.com', 'objects.githubusercontent.com')
 
 
 def sources():
     return [{'id': KIND, 'kind': KIND, 'name': 'GitHub releases', 'url': 'https://github.com',
-             'builtin': True, 'enabled': True, 'trust': 'community'}]
+             'builtin': True, 'enabled': True, 'trust': 'community', 'token': token_status()}]
+
+
+def _token():
+    """(token, where it came from) or (None, None). The environment variable wins."""
+    env = os.environ.get(ENV)
+    if env:
+        return env, 'env'
+    saved = _credentials.get(CREDENTIAL)
+    if saved and saved.kind == 'bearer':
+        return saved, 'saved'
+    return None, None
+
+
+def _credential(token):
+    if isinstance(token, _credentials.Credential):
+        return token
+    try:
+        return _credentials.bearer(API + '/', token)
+    except ValueError as e:
+        raise SourceError('%s is not a usable GitHub token: %s' % (ENV, e)) from None
+
+
+def _digest(credential):
+    return hashlib.sha256(credential.header().encode()).hexdigest()
+
+
+def token_status():
+    """Whether a token is set, where from, and whether GitHub rejected it; never the token."""
+    token, where = _token()
+    rejected = False
+    if token is not None:
+        try:
+            with _rejected_lock:
+                rejected = _digest(_credential(token)) in _rejected
+        except SourceError:
+            rejected = True
+    return {'set': token is not None, 'from': where, 'rejected': rejected}
+
+
+def set_token(token):
+    """Save a personal access token for api.github.com (FRAME_GITHUB_TOKEN still wins)."""
+    if not isinstance(token, str) or not token.strip():
+        raise SourceError('Paste a GitHub token')
+    try:
+        credential = _credentials.bearer(API + '/', token.strip())
+    except ValueError as e:
+        raise SourceError('That is not a usable GitHub token: ' + str(e)) from None
+    _credentials.put(CREDENTIAL, credential)
+    with _rejected_lock:
+        _rejected.discard(_digest(credential))  # a fresh chance
+    return {'message': 'GitHub token saved' + (' (%s is set and is used instead)' % ENV if os.environ.get(ENV) else '')}
+
+
+def clear_token():
+    _credentials.remove(CREDENTIAL)
+    return {'message': 'GitHub token removed'}
 
 
 def _curated():
@@ -20,13 +80,33 @@ def _curated():
 
 
 def _api(path):
+    """GET from the GitHub API, with the user's token if there is one. If GitHub rejects
+    the token (401), retry once without it, like Obtainium, and remember the rejection."""
     headers = {'Accept': 'application/vnd.github+json', 'X-GitHub-Api-Version': '2022-11-28'}
-    token = os.environ.get('FRAME_GITHUB_TOKEN')
-    if token:
-        headers['Authorization'] = 'Bearer ' + token
+    token, _ = _token()
+    auth = _credential(token) if token is not None else None
+    with _rejected_lock:
+        if auth and _digest(auth) in _rejected:
+            auth = None  # already refused this run: don't spend a request on it again
+    hint = ' (add a GitHub token in the store\'s source settings, or set %s, to raise the limit)' % ENV
+    if token is not None and not auth:
+        hint = ' (GitHub rejected your token; replace it in the store\'s source settings)'
+
+    def get(auth):
+        return _web.read(API + path, ('api.github.com',), headers, name='GitHub',
+                         hint='' if auth else hint, auth=auth)
     try:
-        return json.loads(_web.read(API + path, ('api.github.com',), headers, name='GitHub',
-                                    hint='' if token else ' (set FRAME_GITHUB_TOKEN to raise the limit)'))
+        try:
+            data = get(auth)
+        except SourceAuthError:
+            if not auth:
+                raise
+            with _rejected_lock:
+                _rejected.add(_digest(auth))
+            print('GitHub rejected the access token (HTTP 401); continuing without it', file=sys.stderr)
+            hint = ' (GitHub rejected your token; replace it in the store\'s source settings)'
+            data = get(None)
+        return json.loads(data)
     except (ValueError, TypeError) as e:
         raise SourceError('Invalid GitHub response') from e
 

ui/apk_sources/search.py

@@ -113,6 +113,14 @@ def manage_repo(action, **kwargs):
     return getattr(module, action)(**kwargs)
 
 
+def set_token(source_id, token):
+    """Save (token) or remove (None) a source's access token; status is in sources()."""
+    module, _ = resolve(source_id)
+    if not hasattr(module, 'set_token'):
+        raise SourceError('This source does not use an access token')
+    return module.set_token(token) if token is not None else module.clear_token()
+
+
 def normalise(name):
     return ' '.join(re.findall(r'\w+', unicodedata.normalize('NFKC', name or '').casefold()))
 

ui/frame_connect.py

@@ -31,7 +31,8 @@ import frame_host  # noqa: E402
 FRAME_USER = os.environ.get("FRAME_USER", "steamos")
 USER_FROM_ENV = "FRAME_USER" in os.environ
 FRAME_ALIAS = os.environ.get("FRAME_ALIAS", "frame")
-SSH_DIR = Path.home() / ".ssh"
+# $FRAME_CONTROL_SSH_DIR in tests, as frame_devices.ssh_dir(), so they never touch the real one.
+SSH_DIR = Path(os.environ.get("FRAME_CONTROL_SSH_DIR") or Path.home() / ".ssh")
 KEY = SSH_DIR / "id_ed25519_frame"
 # The devkit service only accepts ssh-rsa keys (write_key in Valve's
 # steamos-devkit-service), so pairing uses a second key next to the ed25519 one.

ui/index.html

@@ -611,6 +611,8 @@
   .source-repo-form input { margin-top:7px; display:block; border-radius:6px; }
   .source-repo-form details { font-size:11px; color:var(--muted); }
   .source-repo-form button { justify-self:start; border-radius:6px; }
+  .source-token { display:flex; flex-wrap:wrap; align-items:center; gap:7px; margin:8px 0 4px; }
+  .source-token input { flex:1; min-width:150px; border-radius:6px; font-size:12px; }
   #appStore :focus-visible, .store-dialog :focus-visible, .store-tabs :focus-visible { outline:2px solid var(--link); outline-offset:4px; }
   .store-search input:focus-visible { outline:0!important; }
   @media(min-width:1400px) { .store-grid { grid-template-columns:repeat(5,minmax(0,1fr)); } }
@@ -1207,7 +1209,7 @@
   <button class="store-close" data-close="sourcesDialog" aria-label="Close source settings">×</button>
   <div class="sources-intro"><div class="store-eyebrow">Your app shelves</div><h1 id="sourcesTitle">A world of apps.</h1><p>Choose where you discover games and apps. We bring them together for you.</p></div>
   <div id="sourcesList"></div>
-  <form id="addSource" class="source-repo-form"><h2>Add a source</h2><p>Have a favorite F-Droid repository? Add its address to browse its apps here.</p><label>Repository address<input name="url" type="url" required placeholder="https://example.org/fdroid/repo"></label><details><summary>Verify this source (optional)</summary><label>Repository fingerprint<input name="fingerprint" type="text" autocomplete="off" placeholder="Paste the fingerprint from the repository owner"></label></details><button class="action">Add source</button></form>
+  <form id="addSource" class="source-repo-form"><h2>Add a source</h2><p>Have a favorite F-Droid repository? Add its address to browse its apps here.</p><label>Repository address<input name="url" type="url" required placeholder="https://example.org/fdroid/repo"></label><details><summary>Verify this source (optional)</summary><label>Repository fingerprint<input name="fingerprint" type="text" autocomplete="off" placeholder="Paste the fingerprint from the repository owner"></label></details><details><summary>This repository needs a login (optional)</summary><label>Username<input name="username" type="text" autocomplete="off" autocapitalize="none" spellcheck="false"></label><label>Password<input name="password" type="password" autocomplete="new-password"></label><p>Kept on this computer, apart from the address, and only sent to this repository over HTTPS.</p></details><button class="action">Add source</button></form>
   <p id="sourceError" class="store-notice" role="status" aria-live="polite"></p>
 </dialog>
 
@@ -3139,12 +3141,18 @@ function storeIcon(o) { return `<span class="store-icon" style="--hue:${storeHue
 document.addEventListener("error", e => { if (e.target.matches?.(".store-art img, .store-icon img")) e.target.remove(); }, true);
 function sourceMark(s) { const name=storeName(s); return `<span class="offer-mark" style="--hue:${storeHue(name)}" aria-hidden="true">${esc(name === "SideQuest" ? "SQ" : name === "F-Droid" ? "Fd" : name.slice(0,2))}</span>`; }
 function sourceDescription(s) { return s.description || ({fdroid:"Independent, open-source apps",sidequest:"Discover something new in VR",itch:"Games from independent creators",github:"Direct from the people who make it"})[s.kind] || (s.name.includes("GitHub") ? "Direct from the people who make it" : s.name.includes("Droid") ? "Independent, open-source apps" : s.name.includes("SideQuest") ? "Discover something new in VR" : s.name.includes("itch") ? "Games from independent creators" : "Apps shared by this community"); }
+// GitHub's optional access token: only whether one is set is ever shown, never its value.
+function sourceToken(s) {
+  const t=s.token;if(!t)return "";
+  const note=t.rejected ? "GitHub rejected this token, so searches continue without it. Replace or clear it." : t.from==="env" ? "Using the token from FRAME_GITHUB_TOKEN." : t.set ? "A token is saved." : "Optional: a token raises GitHub's limit of 60 requests an hour.";
+  return `<form class="source-token" data-token="${esc(s.id)}"><input name="token" type="password" autocomplete="off" spellcheck="false" aria-label="${esc(storeName(s.name))} access token" placeholder="${t.set ? "Replace access token" : "Access token"}"><button class="small">Save</button>${t.from==="saved" ? `<button type="button" class="small" data-clear-token="${esc(s.id)}">Clear</button>` : ""}</form><small>${esc(note)}</small>`;
+}
 async function loadSources() {
   try {
     const {sources} = await api("/api/sources"); sourceState.sources = sources;
     $("sourceFilter").innerHTML = `<option value="">All sources</option>` + sources.filter(s=>s.enabled).map(s=>`<option value="${esc(s.id)}">${esc(storeName(s.name))}</option>`).join("");
     $("sourceFilter").value = sourceState.source;
-    $("sourcesList").innerHTML = sources.map(s=>`<div class="source-setting">${sourceMark(s.name)}<div class="grow"><strong>${esc(storeName(s.name))}</strong><p>${esc(sourceDescription(s))}</p><small>${esc(({official:"Official store",community:"Community",user:"Added by you"})[s.trust] || "Community")}</small>${s.builtin === false ? `<button class="small" data-remove="${esc(s.id)}" aria-label="Remove ${esc(s.name)}">Remove</button>` : ""}</div><input class="source-toggle" role="switch" type="checkbox" aria-label="Enable ${esc(storeName(s.name))}" data-enable="${esc(s.id)}" ${s.enabled ? "checked" : ""}></div>`).join("") || `<p class="store-notice">Your sources will appear here when they are available.</p>`;
+    $("sourcesList").innerHTML = sources.map(s=>`<div class="source-setting">${sourceMark(s.name)}<div class="grow"><strong>${esc(storeName(s.name))}</strong><p>${esc(sourceDescription(s))}</p><small>${esc(({official:"Official store",community:"Community",user:"Added by you"})[s.trust] || "Community")}${s.login ? " · Uses a login" : ""}</small>${sourceToken(s)}${s.builtin === false ? `<button class="small" data-remove="${esc(s.id)}" aria-label="Remove ${esc(s.name)}">Remove</button>` : ""}</div><input class="source-toggle" role="switch" type="checkbox" aria-label="Enable ${esc(storeName(s.name))}" data-enable="${esc(s.id)}" ${s.enabled ? "checked" : ""}></div>`).join("") || `<p class="store-notice">Your sources will appear here when they are available.</p>`;
   } catch (e) { $("sourceError").textContent = e.message; }
 }
 function storeCard(app) {
@@ -3271,7 +3279,8 @@ $("detailBody").onclick=e=>{if(e.target.closest('[data-close]'))return $("appDet
 $("detailBody").onchange=e=>{if(e.target.name==='storeOffer'){sourceState.detail.choice=+e.target.value;renderAppDetail();loadOfferDetails();}};
 $("sourcesOpen").onclick=()=>{loadSources();$('sourcesDialog').showModal();};
 $("sourcesList").onchange=e=>{if(e.target.matches('[data-enable]'))changeSource({action:'enable',source:e.target.dataset.enable,enabled:e.target.checked});};
-$("sourcesList").onclick=e=>{const b=e.target.closest('[data-remove]');if(b)changeSource({action:'remove',source:b.dataset.remove});};
+$("sourcesList").onclick=e=>{const b=e.target.closest('[data-remove]');if(b)changeSource({action:'remove',source:b.dataset.remove});const c=e.target.closest('[data-clear-token]');if(c)changeSource({action:'clear-token',source:c.dataset.clearToken});};
+$("sourcesList").onsubmit=e=>{const f=e.target.closest('[data-token]');if(!f)return;e.preventDefault();const token=f.elements.token.value.trim();f.reset();if(token)changeSource({action:'set-token',source:f.dataset.token,token});};
 async function addSource(body) {
   $("sourceError").textContent="Checking the repository. Large ones can take a minute…";
   let error="";
@@ -3279,7 +3288,49 @@ async function addSource(body) {
   $("sourceError").textContent=result ? result.message : error || "We couldn't add that repository.";
   if(result){$("addSource").reset();sourceState.source='';await loadSources();await searchSources();}
 }
-$("addSource").onsubmit=e=>{e.preventDefault();const f=new FormData(e.target);addSource({action:'add',url:f.get('url'),fingerprint:f.get('fingerprint')});};
+// A login typed into the address moves into the login fields, on the page too, before anything
+// is sent: the address is shown and logged, the login never. It stays there if adding fails.
+// Plain text, not the URL parser: an address it rejects (or a login it can't decode) must
+// still lose its login first. Decoding falls back to the text as typed.
+// Percent-decode what decodes; keep any byte sequence that isn't UTF-8 as typed.
+// One forward pass: each lead byte gives its sequence length (at most 4 bytes).
+function decodeLenient(s) {
+  return s.replace(/(?:%[0-9a-f]{2})+/gi, run => {
+    const b=run.match(/%[0-9a-f]{2}/gi); let out='', i=0;
+    while(i<b.length){
+      const v=parseInt(b[i].slice(1),16);
+      const n=v<0x80 ? 1 : v>=0xC2&&v<0xE0 ? 2 : v>=0xE0&&v<0xF0 ? 3 : v>=0xF0&&v<0xF5 ? 4 : 0;
+      if(n){ try { out+=decodeURIComponent(b.slice(i,i+n).join('')); i+=n; continue; } catch {} }
+      out+=b[i]; i++;
+    }
+    return out;
+  });
+}
+// Moves a login typed into the address into the login fields. Returns false for an
+// address the URL parser rejects that may hold a login: it is refused, not sent.
+function moveLogin(form) {
+  // Normalise the way URL parsers do: trim C0 controls and spaces, drop tabs and newlines.
+  const el=form.elements;
+  el.url.value=el.url.value.replace(/^[\u0000- ]+|[\u0000- ]+$/g,'').replace(/[\t\n\r]/g,'');
+  let u;
+  // A backslash and an '@' together are read differently by browsers and by the
+  // server, so the address can't be split safely: refuse it rather than guess.
+  if(el.url.value.includes('@') && el.url.value.includes('\\'))return false;
+  try { u=new URL(el.url.value); } catch { return !el.url.value.includes('@'); }
+  if(!u.username && !u.password)return true;
+  const user=decodeLenient(u.username), pass=decodeLenient(u.password);
+  u.username=''; u.password=''; el.url.value=u.href;
+  if(!el.username.value)el.username.value=user;
+  if(!el.password.value)el.password.value=pass;
+  el.username.closest('details').open=true;
+  return true;
+}
+$("addSource").onsubmit=e=>{
+  e.preventDefault();
+  if(!moveLogin(e.target)){ $("sourceError").textContent="That address isn't valid. Put the username and password in the login fields below, not in the address."; return; }
+  const f=new FormData(e.target), url=f.get('url'), username=f.get('username'), password=f.get('password');
+  addSource({action:'add',url,fingerprint:f.get('fingerprint'),...(username ? {username,password} : {})});
+};
 document.querySelectorAll('[data-close]').forEach(b=>b.onclick=()=>$(b.dataset.close).close());
 $('appDetail').addEventListener('close',()=>{sourceState.detailRequest++;});
 document.querySelectorAll('[data-store-tab]').forEach(b=>b.onclick=()=>{const browse=b.dataset.storeTab==='browse';$('appStore').hidden=!browse;$('androidLibrary').hidden=browse;$('repNew').hidden=browse;$('sourcesOpen').hidden=!browse;document.querySelectorAll('[data-store-tab]').forEach(t=>{t.classList.toggle('on',t===b);t.setAttribute('aria-pressed',t===b);});if(!browse)loadAndroid();});

ui/server.py

@@ -2153,6 +2153,16 @@ def source_text(body, key, optional=False):
     return value.strip()
 
 
+def login_text(body, key):
+    """A username, password or token: kept exactly (no stripping) and never echoed back."""
+    value = body.get(key)
+    if value in (None, ''):
+        return None
+    if not isinstance(value, str) or len(value) > 4096:
+        raise Failure('Provide a valid ' + key, 400)
+    return value
+
+
 def source_search(query):
     args = parse_qs(query)
     q = args.get('q', [''])[0]
@@ -2191,19 +2201,31 @@ def source_manage(body):
         if action == 'add':
             url = source_text(body, 'url')
             fingerprint, name = source_text(body, 'fingerprint', True), source_text(body, 'name', True)
-            parts = urlparse(url)
-            if parts.scheme not in ('https', 'fdroidrepos') or not parts.hostname or parts.username:
-                raise Failure('Use an HTTPS or fdroidrepos:// repository URL without credentials', 400)
+            username, password = login_text(body, 'username'), login_text(body, 'password')
+            try:
+                parts = urlparse(url)
+                parts.port
+            except ValueError:
+                raise Failure('Use an HTTPS or fdroidrepos:// repository URL', 400) from None
+            if parts.scheme not in ('https', 'fdroidrepos') or not parts.hostname:
+                raise Failure('Use an HTTPS or fdroidrepos:// repository URL', 400)
             apk_search.repo_module()  # fail now if this build can't manage repositories
 
             def add():  # downloads and verifies the whole index: a job, not a request
-                source = apk_search.manage_repo('add_repo', url=url, fingerprint=fingerprint, name=name)
+                # A login (here or in the URL) is stored apart and never comes back in a response.
+                login = {'username': username, 'password': password} if username or password else {}
+                source = apk_search.manage_repo('add_repo', url=url, fingerprint=fingerprint, name=name, **login)
                 message = 'Added ' + source['name']
                 if source.get('trust_on_first_use'):
                     message += '. Trusted on first use: ' + source['fingerprint'].upper()
                 return {'message': message, 'source': {k: source.get(k) for k in
                                                        ('id', 'name', 'fingerprint', 'trust_on_first_use')}}
             return start_job('Add repository', add)
+        if action in ('set-token', 'clear-token'):
+            token = login_text(body, 'token') if action == 'set-token' else None
+            if action == 'set-token' and not token:
+                raise Failure('Paste a token', 400)
+            return apk_search.set_token(source_text(body, 'source'), token)
         if action == 'game-data':
             package = source_text(body, 'package')
             return start_job('Add game data', lambda: apk_search.add_game_data(package))