diff --git a/.claude/NOTES-sidequest.md b/.claude/NOTES-sidequest.md new file mode 100644 index 0000000..8e085e0 --- /dev/null +++ b/.claude/NOTES-sidequest.md @@ -0,0 +1,55 @@ +# SideQuest implementation notes + +2026-09-28. Worktree steam-frame-sidequest, branch sidequest. No delegation, +Frame mutations, installs, launches, pushes or issue edits. + +- Read shared source interface, APK/VR docs, catalogue README and Python backend. +- SideQuest robots: crawl delay 3; disallow /search/, /user/*, /sideload/*. +- Current /terms Angular text (main-4MMXZRXL.js): Prohibited Activities (i) + prohibits scraping; (xi) limits access to provided/authorised technologies; + (xii) forbids bypass. Public API address is not permission for a third-party + integration. Page-only source; no automated store downloads or metadata crawl. +- api.sidequestvr.com/robots.txt returned HTTP 403. First shared JS chunk also + returned 403. No attempt to bypass either response. +- Public SideQuest desktop source cloned inside .claude/research for inspection. + /install-from-key takes a website-issued token and returns apps[].urls[] with + provider APK/OBB/Github Release/Mod and link_url. Do not reproduce token flow. +- Two SSH read-only attempts to frame timed out (exit 255). Exact host-side + /sdcard mapping cannot be claimed. internal/ is private app data, + not evidence of an OBB mapping. Use the running container's /sdcard path for + OBB writes, without launching it; backup only documented internal/. +- Scope: OBB helper/CLI, private-data backup/restore helper/CLI, compliant + SideQuest page-only source. No search UI, artwork installer or install edits. +- Cross-provider review not launched: task explicitly forbids delegation; + parent brief reserves integration and review for the parent. + +## Implementation and verification + +- Added ui/frame_android_data.py and frame/android/app-data.py; additive wrappers + and CLI branches only in frame_android.py (install/_install unchanged). +- OBB transfers to an already-running named container, SHA-256 check before + per-file rename. No claimed host sdcard mapping or device persistence. +- Backup/restore covers private internal/ only, requires a stopped + instance, uses podman unshare, validates archive paths/types/package/instance, + preserves numeric owners/modes, retains the prior data directory on restore. +- SideQuest adapter intentionally raises a page-only SourceError on search and + download; details gives a numeric listing page with unknown facts, images + schema and downloadable=False. Needs aggregate search to surface the error. +- docs/sidequest.md contains source links, feature comparison, command examples, + terms/robots findings and outstanding device/API questions. +- Fixture tests/fixtures/sidequest-policy.json records observed policy excerpts; + no API response is fabricated. +- `python3 -m unittest discover -s tests`: final run 181 tests, OK (exit 0). + Includes real local shell execution of the OBB checksum/publish sequence, + rejecting a changed input without replacing the previous file; archive + round-trip/retained previous save, 0600 backup, malformed archive rejection. + No test contacts the network or Frame. +- `ast.parse(..., feature_version=(3,9))`: four implementation files passed. + Runtime python3 is Xcode Python 3.9. +- `python3 ui/frame_android.py install-obb` and `... backup-data`: both exit 1 + with the intended required-arguments error, without contacting Frame. +- `git diff --check`: passed before commit. +- No SideQuest game downloaded and no `info ` run: terms blocked that + requested E2E. No Frame app was installed or launched; no live OBB, namespace + ownership or restore acceptance test. Independent review reserved for parent, + per this task's explicit no-delegation instruction. diff --git a/docs/apks.md b/docs/apks.md index 8bca60a..54d2fdf 100644 --- a/docs/apks.md +++ b/docs/apks.md @@ -325,3 +325,11 @@ because gamescope scales Lepton's surface to fit the same panel. Also unverified whether the settings survive the app or its Lepton instance relaunching. Lepton Development rebuilds its Android data on exit, so there they probably don't. + +## Expansion files and save backups + +SideQuest-inspired CLI helpers install local OBB files into an already-running +app instance and back up/restore a stopped instance's private app data. See +[SideQuest features and limits](sidequest.md) for commands, archive scope and +verification status. These paths have offline coverage; real Frame storage and +permissions remain unverified. They do not change APK install or launch behavior. diff --git a/docs/sidequest.md b/docs/sidequest.md new file mode 100644 index 0000000..eba0f02 --- /dev/null +++ b/docs/sidequest.md @@ -0,0 +1,142 @@ +# SideQuest and Frame Control + +Researched 2026-09-28. SideQuest is both a Quest discovery website and a desktop +sideloading/device-management app. Its Quest labels are **not** evidence that a +game works on Lepton: inspect the APK for arm64/OpenXR, Android API requirements, +VrApi and Meta services (see [VR APKs](vr-apks.md)). + +## Features worth borrowing + +Desktop evidence is the public [SideQuest source at af2ac70](https://github.com/SideQuestVR/SideQuest/tree/af2ac7043db122bca3c8db18f2b58f1660e9befb), +especially [ADB operations](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/adb-client.service.ts), +[drag and drop](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/drag-and-drop.service.ts), +and the [legacy repository index](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/packages/package.service.ts). +Website evidence: [SideQuest](https://sidequestvr.com/) and its public Angular +bundle `main-4MMXZRXL.js`, inspected locally without browser automation. +No SideQuest implementation code was copied. + +| SideQuest feature | Frame Control before this change | Borrow? / effort | +|---|---|---| +| Store descriptions, screenshots, banners, trailers, ratings | F-Droid names, icons, compatibility verdicts and reports; no equivalent rich VR store | Yes, from authorised sources; medium. Search and library workers own presentation/artwork. | +| OBB expansion-file install | APK-only install | **Implemented helper and CLI**, medium. Essential for games whose assets are separate from the APK. | +| App-data backup/restore | Persistent instances and optional keep-data uninstall, no portable save archive | **Implemented private-data helper and CLI**, medium. Back up before updates or experiments. | +| File manager (list, upload, download, remove) | General Send to Frame, no Android file browser | Useful later, medium; requires clear instance selection and scoped paths. | +| Installed-app management (launch, uninstall, backup) | List, launch, stop, remove, probe | Already mostly covered. Backup added here. | +| Update notices / account library | Compatible-version lookup; no source-aware installed update notices | Useful later, medium; needs original version code and source identity recorded on install. | +| Custom repositories | Built-in F-Droid catalogue and compatible-version indexes | Separate user-repos worker. Legacy SideQuest source has a fixed SideQuestRepos index; arbitrary current custom-repo support was not verified. | +| Drag-and-drop APK/OBB install | APK drag-and-drop already works | OBB backend added here; future UI can call it. UI drop wiring is not included. | +| Tags, price, headset filters, reviews | Text search and Lepton verdicts, not Quest headset metadata | Useful, medium; search worker owns filters. Keep source headset claims distinct from tested Frame compatibility. | +| Screenshot/video capture and streaming | Frame screenshots/VR capture already present | Reuse existing tools; do not port Quest capture commands. | +| Device settings and ADB utilities | Frame/Android display settings, SSH and own-instance tools | Borrow selectively; Quest CPU/GPU presets and wireless-ADB setup do not map directly to Lepton. | + +Priority: expansion files, then save backup/restore. Rich discovery and update +notices follow once a permitted metadata source and source/version persistence +are available. This patch deliberately exposes CLI/backend operations, leaving +shared UI, install(), Steam artwork and launch behavior to sibling work. + +## SideQuest as a source: page-only + +[Terms](https://sidequestvr.com/terms), “Prohibited Activities”, (i) prohibits +copying/distributing/disclosing the Service including automated or non-automated +“scraping”; (xi) prohibits content access through means other than those provided +or authorised by the Service; (xii) prohibits bypassing access restrictions. +The terms describe downloading developer-posted games through the Service, but +do not establish permission for this third-party API integration. + +[robots.txt](https://sidequestvr.com/robots.txt) requests a three-second crawl +delay and disallows `/search/`, `/user/*` and `/sideload/*`. Robots permission +would not override the terms. The API host's robots request returned HTTP 403; +a request for the first shared website JS chunk also returned 403. No bypass, +account token, cookies, browser session or private endpoint was used. + +The homepage publishes `https://api.sidequestvr.com` and +`https://cdn.sidequestvr.com`. The website bundle calls `searchApps(...)` and +`getApp(id, null)`; their actual HTTP search/detail routes could not be established +from the retrieved bundle. Do not invent endpoints. The open-source desktop +[install flow](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/electron/app.ts) +POSTs `{token: ...}` to `/install-from-key`. It consumes +`data.apps[].urls[]`, with `provider` values including `APK`, `OBB`, +`Github Release` and `Mod`, and `link_url`. This is a website-issued install-key +flow, not evidence of an anonymous download API. It is not implemented here. + +`ui/apk_sources/sidequest.py` implements the shared interface conservatively: + +- `sources()` marks SideQuest `page_only` and explains why. +- `search()` raises a user-readable `SourceError` with the browse URL (zero + limit returns no rows). It does not invent app results or report a false + “no matching games”. The aggregate search UI should surface this source error. +- `details()` accepts a numeric listing id and returns its canonical page link, + `downloadable: False`, empty versions/tags/headsets and the `images` shape + `{icon: None, banner: None, screenshots: []}`. Name is explicitly a listing id; + unknown facts, including free/VR status, stay `None`. +- `download()` refuses with that page link. Paid/external listings cannot be + downloaded by this adapter either. No downloads means no verification claim. + +The JSON fixture records policy evidence, **not a purported live app response**. +No listing metadata, artwork URLs, or OBB download URLs were scraped. +The requested real SideQuest → OpenXR APK → `frame_android.py info` test is +**blocked by the terms**, and was not performed. No alternate source is silently +substituted. A future integration needs SideQuest's permission or an expressly +supported third-party API, plus recorded search/detail/download fixtures, +free/direct-download classification, and size/hash verification. A calculated +local SHA-256 alone must not be called publisher verification. + +## OBB files + +```sh +python3 ui/frame_android.py install-obb org.example.game main.42.org.example.game.obb +python3 ui/frame_android.py install-obb org.example.game main.42.org.example.game.obb patch.42.org.example.game.obb +``` + +Install the APK first. The named instance must already be running; the helper +never launches an app or uses Lepton Development. It requires standard +`main|patch...obb` filenames and nonempty files, validates +the entire batch before transfer, streams each file through SSH into that +instance, checks its SHA-256 **inside Android**, then renames it into +`/sdcard/Android/obb//`. `verified: True` here means transfer integrity +against the local input, not publisher authentication. Publication is atomic per +file, not for the whole batch; retry after a partial batch failure. Existing OBBs +with different version codes remain. The filename version must match the game; +the current install metadata does not expose its version code for comparison. +Restart the game yourself after the transfer if it cached missing expansion data. + +Both read-only SSH attempts to the Frame timed out. Therefore the exact +host-side `/sdcard` mapping and persistence of expansion data were **not verified**. +`compatdata//internal/` is documented as `/data/data/`; +it must not be mistaken for `/sdcard`. Using Android's path avoids guessing a +host layout, but device verification across restart/update is still required. +No OBB file was installed on the Frame during this work. + +## Private app-data backups + +```sh +python3 ui/frame_android.py stop org.example.game +python3 ui/frame_android.py backup-data org.example.game ./game-save.tar.gz +python3 ui/frame_android.py restore-data org.example.game ./game-save.tar.gz +``` + +Keep the instance stopped throughout either operation; do not launch it from +Steam concurrently. The remote guard fails if Podman cannot enumerate containers +or reports that instance running. The helpers use `podman unshare` to read/write +Android's mapped ownership without changing the live data's permissions. + +The archive covers **only** `compatdata//internal/`, not the +APK, external `/sdcard/Android/data`, OBBs, keystore, or the full Android snapshot. +It contains a package/instance manifest and regular files/directories. Backups +are private (0600), validated before publication, and never overwrite an existing +backup. Keep them safe: app data can contain credentials and is not encrypted. + +Restore checks the package and instance, rejects absolute/traversing/duplicate +paths, links and devices, caps files at 100,000 and content at 20 GiB, and validates +again on the Frame. It extracts into a separate directory, preserves numeric +ownership, ordinary modes and timestamps, then swaps the private-data directory. +Setuid/setgid bits are not restored. The previous directory remains beside it as +`..before-restore-`; the returned `previous` path identifies +it. This is an additional recovery copy, not an automatic deletion policy. + +Locally verified: archive round trip including recovery copy, malformed archive +rejection, transfer command construction and failure handling. Not verified: +real Frame UID mappings/permissions, Android app-level recovery, live FUSE OBB +writes or persistence. Backups reject symlinks/special files; an app requiring +those needs a separately designed backup format. These CLI features still need +a real-device acceptance pass before being exposed as a polished UI workflow. diff --git a/frame/android/app-data.py b/frame/android/app-data.py new file mode 100644 index 0000000..1ecc141 --- /dev/null +++ b/frame/android/app-data.py @@ -0,0 +1,139 @@ +"""Private-data archives, run under podman unshare on the Frame. Stdlib only.""" +import json +import os +from pathlib import Path, PurePosixPath +import shutil +import sys +import tarfile +import tempfile +import time + +MAX_BYTES = 20 * 1024 ** 3 +MAX_FILES = 100000 + + +def inspect_archive(path, package, instance): + names, total, manifest = set(), 0, None + with tarfile.open(path, 'r:gz') as archive: + for member in archive: + name = member.name + parts = PurePosixPath(name).parts + if (not parts or name.startswith('/') or '..' in parts or + name != '/'.join(parts) or name in names or '\\' in name): + raise ValueError('unsafe or duplicate archive path') + if name == 'data' and not member.isdir(): + raise ValueError('data root must be a directory') + names.add(name) + if len(names) > MAX_FILES or not (member.isdir() or member.isfile()): + raise ValueError('archive has too many files, links or special files') + if member.uid < 0 or member.gid < 0 or member.uid > 65535 or member.gid > 65535: + raise ValueError('archive owner outside Android user namespace') + total += member.size + if total > MAX_BYTES: + raise ValueError('archive exceeds 20 GiB') + if name == 'manifest.json' and member.isfile() and member.size <= 4096: + manifest = json.load(archive.extractfile(member)) + elif parts[0] != 'data': + raise ValueError('unexpected archive member') + if (not isinstance(manifest, dict) or manifest.get('format') != 1 or + manifest.get('package') != package or manifest.get('instance') != instance or + 'data' not in names): + raise ValueError('backup does not match this package and instance') + return {'files': len(names) - 1, 'bytes': total, 'package': package, 'instance': instance} + + +def backup(root, package, instance, output): + import io + source = root / package + if source.is_symlink() or not source.is_dir(): + raise ValueError('private app data does not exist or is a symlink') + count, total = 0, 0 + + def checked(member): + nonlocal count, total + count += 1 + total += member.size + if not (member.isdir() or member.isfile()) or count > MAX_FILES or total > MAX_BYTES: + raise ValueError('private data contains links/special files or exceeds backup limits') + return member + + manifest = json.dumps({'format': 1, 'package': package, 'instance': instance}).encode() + with tarfile.open(fileobj=output, mode='w|gz', dereference=False) as archive: + member = tarfile.TarInfo('manifest.json') + member.size, member.mode = len(manifest), 0o600 + archive.addfile(member, io.BytesIO(manifest)) + archive.add(str(source), arcname='data', filter=checked) + + +def restore(root, package, instance, input_stream): + source = root / package + if source.is_symlink() or not source.is_dir(): + raise ValueError('private app data does not exist or is a symlink') + with tempfile.TemporaryDirectory(prefix='.frame-restore-', dir=str(root)) as work: + work = Path(work) + archive_path = work / 'backup.tar.gz' + with archive_path.open('wb') as output: + size = 0 + while True: + chunk = input_stream.read(1024 * 1024) + if not chunk: + break + size += len(chunk) + if size > MAX_BYTES: + raise ValueError('compressed backup exceeds 20 GiB') + output.write(chunk) + result = inspect_archive(archive_path, package, instance) + stage = work / 'stage' + stage.mkdir(mode=0o700) + with tarfile.open(archive_path, 'r:gz') as archive: + directories = [] + for member in archive: + if member.name == 'manifest.json': + continue + target = stage / member.name + if member.isdir(): + target.mkdir(parents=True, exist_ok=True) + directories.append((target, member)) + else: + target.parent.mkdir(parents=True, exist_ok=True) + with archive.extractfile(member) as src, target.open('xb') as dst: + shutil.copyfileobj(src, dst, 1024 * 1024) + apply_metadata(target, member) + for target, member in reversed(directories): + apply_metadata(target, member) + previous = root / ('.' + package + '.before-restore-' + str(time.time_ns())) + source.rename(previous) + try: + (stage / 'data').rename(source) + except BaseException: + previous.rename(source) + raise + result['previous'] = str(previous) + return result + + +def apply_metadata(path, member): + os.chown(str(path), member.uid, member.gid) + os.chmod(str(path), member.mode & 0o777) + os.utime(str(path), (member.mtime, member.mtime)) + + +def main(): + action, package, instance = sys.argv[1:] + instance = int(instance) + root = Path.home() / '.local/share/Steam/steamapps/compatdata' / str(instance) / 'internal' + if root.is_symlink() or root.resolve() != root.absolute(): + raise ValueError('private-data directory traverses a symlink') + if action == 'backup': + backup(root, package, instance, sys.stdout.buffer) + elif action == 'restore': + print(json.dumps(restore(root, package, instance, sys.stdin.buffer))) + else: + raise ValueError('unknown app-data action') + + +if __name__ == '__main__': + try: + main() + except (OSError, ValueError, tarfile.TarError) as error: + sys.exit(str(error)) diff --git a/tests/fixtures/sidequest-policy.json b/tests/fixtures/sidequest-policy.json new file mode 100644 index 0000000..dd08588 --- /dev/null +++ b/tests/fixtures/sidequest-policy.json @@ -0,0 +1,18 @@ +{ + "recorded": "2026-09-28", + "robots": { + "url": "https://sidequestvr.com/robots.txt", + "user_agent": "*", + "crawl_delay": 3, + "disallow": ["/search/", "/user/*", "/sideload/*"], + "sitemap": "https://sidequestvr.com/sitemap_index.xml" + }, + "api_robots": {"url": "https://api.sidequestvr.com/robots.txt", "status": 403}, + "terms": { + "url": "https://sidequestvr.com/terms", + "bundle": "https://sidequestvr.com/main-4MMXZRXL.js", + "prohibited_activities_i": "copy, distribute, or disclose any part of the Service in any medium, including without limitation by any automated or non-automated scraping", + "prohibited_activities_xi": "access any content on the Service through any technology or means other than those provided or authorized by the Service" + }, + "note": "Policy evidence, not a fabricated API response. No app metadata or download fixture was collected after discovering the restriction." +} diff --git a/tests/test_frame_android_data.py b/tests/test_frame_android_data.py new file mode 100644 index 0000000..3f4c3fe --- /dev/null +++ b/tests/test_frame_android_data.py @@ -0,0 +1,213 @@ +import io +import json +import os +from pathlib import Path +import runpy +import shlex +import shutil +import subprocess +import sys +import tarfile +import tempfile +import unittest +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / 'ui')) +import frame_android as android +import frame_android_data as data + +REMOTE = runpy.run_path(str(data.REMOTE)) +PKG = 'org.example.game' +META = {'package': PKG, 'instance': 2800000001} + + +class ObbTests(unittest.TestCase): + def test_invalid_files_never_contact_frame(self): + with tempfile.TemporaryDirectory() as tmp, patch.object(android, 'ssh') as ssh: + for name in ('game.obb', 'main.1.org.other.game.obb', 'main.x.' + PKG + '.obb'): + path = Path(tmp) / name + path.write_bytes(b'content') + with self.assertRaises(android.FrameError): + data.install_obb(PKG, [path]) + with self.assertRaises(android.FrameError): + data.install_obb('../game', []) + with self.assertRaises(android.FrameError): + data.install_obb(PKG, []) + ssh.assert_not_called() + + def test_streams_to_correct_instance_and_checks_hash_before_rename(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / ('main.7.' + PKG + '.obb') + path.write_bytes(b'expansion payload') + calls = [] + def stream(command, src=None, dst=None): + calls.append(command) + self.assertEqual(src.read(), b'expansion payload') + with patch.object(android, '_meta_or_fail', return_value=META), \ + patch.object(android, 'ssh', return_value='lepton-steamlaunch-2800000001\n'), \ + patch.object(data, '_stream', side_effect=stream): + result = data.install_obb(PKG, [path]) + self.assertTrue(result['verified']) + self.assertIn('podman exec -i lepton-steamlaunch-2800000001', calls[0]) + self.assertIn('/sdcard/Android/obb/' + PKG, calls[0]) + self.assertLess(calls[0].index('sha256sum'), calls[0].index('; mv')) + self.assertIn(result['obb'][0]['sha256'], calls[0]) + + def test_stopped_instance_and_failed_transfer(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / ('patch.7.' + PKG + '.obb') + path.write_bytes(b'patch') + with patch.object(android, '_meta_or_fail', return_value=META), \ + patch.object(android, 'ssh', return_value=''), patch.object(data, '_stream') as stream: + with self.assertRaisesRegex(android.FrameError, 'start this app'): + data.install_obb(PKG, [path]) + stream.assert_not_called() + with patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 1, b'', b'bad hash')): + with self.assertRaisesRegex(android.FrameError, 'bad hash'): + data._stream('command') + + + @unittest.skipUnless(shutil.which("sh") and shutil.which("shasum"), "shell checksum tools unavailable") + def test_android_shell_publish_and_hash_failure(self): + with tempfile.TemporaryDirectory() as tmp: + source = Path(tmp) / ('main.7.' + PKG + '.obb') + source.write_bytes(b'good expansion') + output = Path(tmp) / 'sdcard/Android/obb' / PKG / source.name + tools_dir = Path(tmp) / 'bin' + tools_dir.mkdir() + checksum = tools_dir / 'sha256sum' + checksum.write_text('#!/bin/sh\nexec shasum -a 256 "$@"\n') + checksum.chmod(0o700) + corrupt = False + def stream(command, src=None, dst=None): + script = shlex.split(command)[-1].replace('/sdcard/', tmp + '/sdcard/') + if corrupt: + source.write_bytes(b'corrupt expansion') + result = subprocess.run(['sh', '-c', script], stdin=src, capture_output=True, + env=dict(os.environ, PATH=str(tools_dir) + ':' + os.environ['PATH'])) + if result.returncode: + raise android.FrameError('checksum failed') + with patch.object(android, '_meta_or_fail', return_value=META), \ + patch.object(android, 'ssh', return_value='lepton-steamlaunch-2800000001'), \ + patch.object(data, '_stream', side_effect=stream): + data.install_obb(PKG, [source]) + self.assertEqual(output.read_bytes(), b'good expansion') + corrupt = True + with self.assertRaises(android.FrameError): + data.install_obb(PKG, [source]) + self.assertEqual(output.read_bytes(), b'good expansion') + self.assertEqual(list(output.parent.glob('*.part')), []) + + +class BackupTests(unittest.TestCase): + def test_roundtrip_and_retains_previous_data(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + source = root / PKG + (source / 'files').mkdir(parents=True) + (source / 'files/save').write_bytes(b'original save') + archive = io.BytesIO() + REMOTE['backup'](root, PKG, META['instance'], archive) + (source / 'files/save').write_bytes(b'new save') + archive.seek(0) + # Current user's uid/gid in this local test; no elevated execution. + result = REMOTE['restore'](root, PKG, META['instance'], archive) + self.assertEqual((source / 'files/save').read_bytes(), b'original save') + self.assertEqual((Path(result['previous']) / 'files/save').read_bytes(), b'new save') + + def make_archive(self, path, members, package=PKG): + with tarfile.open(path, 'w:gz') as archive: + payload = json.dumps({'format': 1, 'package': package, 'instance': META['instance']}).encode() + member = tarfile.TarInfo('manifest.json') + member.size = len(payload) + archive.addfile(member, io.BytesIO(payload)) + root = tarfile.TarInfo('data') + root.type = tarfile.DIRTYPE + archive.addfile(root) + for name, kind in members: + member = tarfile.TarInfo(name) + member.type = kind + member.linkname = '/tmp/escape' + archive.addfile(member) + + def test_rejects_wrong_package_traversal_links_devices_duplicates(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / 'bad.tar.gz' + cases = [('../escape', tarfile.REGTYPE), ('/absolute', tarfile.REGTYPE), + ('data/link', tarfile.SYMTYPE), ('data/link', tarfile.LNKTYPE), + ('data/device', tarfile.CHRTYPE), ('data', tarfile.DIRTYPE), + ('other/file', tarfile.REGTYPE), ('data/../escape', tarfile.REGTYPE)] + for member in cases: + self.make_archive(path, [member]) + with self.assertRaises(ValueError, msg=str(member)): + REMOTE['inspect_archive'](path, PKG, META['instance']) + self.make_archive(path, [], package='org.other.game') + with self.assertRaisesRegex(ValueError, 'does not match'): + REMOTE['inspect_archive'](path, PKG, META['instance']) + + def test_failed_backup_leaves_no_archive_and_existing_is_preserved(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / 'backup.tar.gz' + with patch.object(android, '_meta_or_fail', return_value=META), \ + patch.object(data, '_stream', side_effect=android.FrameError('offline')): + with self.assertRaises(android.FrameError): + data.backup_data(PKG, path) + self.assertEqual(list(Path(tmp).iterdir()), []) + path.write_bytes(b'keep') + with self.assertRaisesRegex(android.FrameError, 'already exists'): + data.backup_data(PKG, path) + self.assertEqual(path.read_bytes(), b'keep') + + def test_guard_does_not_hide_podman_failure(self): + command = data._data_command('backup', META) + self.assertIn('|| exit 1', command) + self.assertIn('stop the app', command) + self.assertIn('podman unshare python3', command) + self.assertNotIn('|| true', command) + + def test_bad_restore_is_rejected_before_transfer(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / 'bad.tar.gz' + self.make_archive(path, [('../escape', tarfile.REGTYPE)]) + with patch.object(android, '_meta_or_fail', return_value=META), patch.object(data, '_stream') as stream: + with self.assertRaises(android.FrameError): + data.restore_data(PKG, path) + stream.assert_not_called() + + + def test_successful_backup_is_private_and_inspectable(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / PKG).mkdir() + (root / PKG / 'save').write_bytes(b'checkpoint') + destination = root / 'backup.tar.gz' + def stream(command, src=None, dst=None): + REMOTE['backup'](root, PKG, META['instance'], dst) + with patch.object(android, '_meta_or_fail', return_value=META), \ + patch.object(data, '_stream', side_effect=stream): + result = data.backup_data(PKG, destination) + self.assertEqual(destination.stat().st_mode & 0o777, 0o600) + self.assertEqual(result['sha256'], data._sha256(destination)) + self.assertEqual(result['files'], 2) + + def test_rejected_restore_keeps_existing_data(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / PKG).mkdir() + (root / PKG / 'save').write_bytes(b'keep') + archive = root / 'bad.tar.gz' + self.make_archive(archive, [('data/link', tarfile.SYMTYPE)]) + with archive.open('rb') as source, self.assertRaises(ValueError): + REMOTE['restore'](root, PKG, META['instance'], source) + self.assertEqual((root / PKG / 'save').read_bytes(), b'keep') + self.assertFalse(list(root.glob('.frame-restore-*'))) + self.assertFalse(list(root.glob('.*.before-restore-*'))) + + def test_archive_root_must_be_a_directory(self): + with tempfile.TemporaryDirectory() as tmp: + archive = Path(tmp) / 'bad.tar.gz' + with tarfile.open(archive, 'w:gz') as target: + target.addfile(tarfile.TarInfo('data')) + with self.assertRaisesRegex(ValueError, 'directory'): + REMOTE['inspect_archive'](archive, PKG, META['instance']) diff --git a/tests/test_sidequest.py b/tests/test_sidequest.py new file mode 100644 index 0000000..495d685 --- /dev/null +++ b/tests/test_sidequest.py @@ -0,0 +1,36 @@ +import json +from pathlib import Path +import sys +import unittest +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui')) +from apk_sources import SourceError, sidequest + + +class SideQuestTests(unittest.TestCase): + def test_policy_is_recorded_and_source_is_page_only(self): + fixture = json.loads((Path(__file__).parent / 'fixtures/sidequest-policy.json').read_text()) + self.assertIn('/search/', fixture['robots']['disallow']) + self.assertIn('scraping', fixture['terms']['prohibited_activities_i']) + self.assertTrue(sidequest.sources()[0]['page_only']) + + @patch('urllib.request.urlopen', side_effect=AssertionError('network forbidden')) + def test_unknown_listing_never_claims_free_or_downloadable(self, _urlopen): + source = sidequest.sources()[0] + entry = sidequest.details(source, '123') + self.assertEqual(entry['page'], 'https://sidequestvr.com/app/123') + self.assertFalse(entry['downloadable']) + self.assertIsNone(entry['free']) + self.assertIsNone(entry['vr']) + self.assertEqual(entry['images']['screenshots'], []) + with self.assertRaisesRegex(SourceError, 'page-only'): + sidequest.download(source, '123') + with self.assertRaisesRegex(SourceError, 'page-only'): + sidequest.search(source, 'open saber') + self.assertEqual(sidequest.search(source, 'open saber', 0), []) + + def test_invalid_ids(self): + for value in ('../123', '1?paid=false', '１', '', '1/2', '1' * 13): + with self.assertRaises(SourceError): + sidequest.details(sidequest.sources()[0], value) diff --git a/ui/apk_sources/sidequest.py b/ui/apk_sources/sidequest.py new file mode 100644 index 0000000..dfed2ee --- /dev/null +++ b/ui/apk_sources/sidequest.py @@ -0,0 +1,41 @@ +"""SideQuest page links only: its terms do not authorise third-party scraping. + +No API calls, cached listings or automated downloads. See docs/sidequest.md. +""" +from . import SourceError + +KIND = 'sidequest' +URL = 'https://sidequestvr.com' +REASON = ('SideQuest is page-only: its terms restrict scraping and unauthorised ' + 'access. Browse and download with SideQuest, then import a developer-provided APK.') + + +def sources(): + return [{'id': KIND, 'kind': KIND, 'name': 'SideQuest', 'url': URL, + 'builtin': True, 'enabled': True, 'trust': 'community', + 'page_only': True, 'reason': REASON}] + + +def search(source, query, limit=50): + # Do not invent catalogue results or interpret a query as a verified free app. + if limit <= 0: + return [] + raise SourceError(REASON + ' ' + URL + '/apps') + + +def details(source, entry_id): + entry_id = str(entry_id) + if not entry_id.isascii() or not entry_id.isdecimal() or len(entry_id) > 12: + raise SourceError('SideQuest listing ids must be numeric') + return {'source': source['id'], 'id': entry_id, 'package': None, + 'name': 'SideQuest listing ' + entry_id, 'summary': REASON, + 'icon': None, 'page': URL + '/app/' + entry_id, 'version': None, + 'version_code': None, 'min_sdk': None, 'abis': None, 'vr': None, + 'size': None, 'free': None, 'license': None, 'updated': None, + 'downloadable': False, 'versions': [], 'tags': [], 'headsets': [], + 'images': {'icon': None, 'banner': None, 'screenshots': []}} + + +def download(source, entry_id, version_code=None): + entry = details(source, entry_id) + raise SourceError(REASON + ' ' + entry['page']) diff --git a/ui/frame_android.py b/ui/frame_android.py index 499e99e..80c39a0 100644 --- a/ui/frame_android.py +++ b/ui/frame_android.py @@ -8,6 +8,7 @@ Lepton Development, which wipes its apps on exit. See docs/apks.md. Python stdlib only. CLI: python3 ui/frame_android.py install APK [--vr|--flat] [--no-xr-compat] | info APK | versions APK-or-PKG + install-obb PKG OBB [OBB ...] | backup-data PKG ARCHIVE | restore-data PKG ARCHIVE patch SRC DST [--add NAME=PATH ...] | list | launch PKG | stop PKG | remove PKG | probe PKG """ import json, os, re, shlex, shutil, struct, subprocess, sys, threading, time, zlib @@ -333,6 +334,21 @@ def patch(src, dst, add=None): raise FrameError(str(e)) from e +def install_obb(pkg, paths): + import frame_android_data + return frame_android_data.install_obb(pkg, paths) + + +def backup_data(pkg, destination): + import frame_android_data + return frame_android_data.backup_data(pkg, destination) + + +def restore_data(pkg, archive): + import frame_android_data + return frame_android_data.restore_data(pkg, archive) + + def main(): cmd, *args = sys.argv[1:] or ['help'] try: @@ -368,6 +384,14 @@ def main(): with open(path, 'rb') as f: additions[entry] = f.read() r = patch(opts.src, opts.dst, additions) + elif cmd == 'install-obb': + if len(args) < 2: + raise FrameError('install-obb requires PACKAGE OBB [OBB ...]') + r = install_obb(args[0], args[1:]) + elif cmd in ('backup-data', 'restore-data'): + if len(args) != 2: + raise FrameError(cmd + ' requires PACKAGE ARCHIVE.tar.gz') + r = (backup_data if cmd == 'backup-data' else restore_data)(*args) elif cmd == 'list': r = list_apps() elif cmd in ('launch', 'stop', 'probe'): diff --git a/ui/frame_android_data.py b/ui/frame_android_data.py new file mode 100644 index 0000000..5a2e31a --- /dev/null +++ b/ui/frame_android_data.py @@ -0,0 +1,133 @@ +"""Expansion files and stopped-instance private-data backups. Python stdlib only.""" +import hashlib +import json +import os +from pathlib import Path +import re +import runpy +import shlex +import subprocess +import tempfile +import uuid + +import frame_android as android + +REMOTE = Path(android.ROOT) / 'frame/android/app-data.py' + + +def _stream(command, src=None, dst=None): + try: + result = subprocess.run(['ssh', *android.SSH_OPTS, android.FRAME, command], + stdin=src if src else subprocess.DEVNULL, + stdout=dst if dst else subprocess.PIPE, + stderr=subprocess.PIPE, timeout=1800) + except subprocess.TimeoutExpired: + raise android.FrameError('app-data transfer timed out') + except OSError as error: + raise android.FrameError('app-data transfer failed: ' + str(error)) + if result.returncode: + raise android.FrameError(result.stderr.decode(errors='replace').strip()[-600:] or + 'app-data transfer failed') + return result.stdout + + +def _sha256(path): + digest = hashlib.sha256() + with open(path, 'rb') as src: + for chunk in iter(lambda: src.read(1024 * 1024), b''): + digest.update(chunk) + return digest.hexdigest() + + +def _meta(package): + if not android.PKG_RE.fullmatch(package or ''): + raise android.FrameError('invalid package name') + meta = android._meta_or_fail(package) + if meta['package'] != package: + raise android.FrameError('installed app metadata has a different package') + return meta + + +def install_obb(package, paths): + if not android.PKG_RE.fullmatch(package or ''): + raise android.FrameError('invalid package name') + paths = [Path(p).resolve() for p in paths] + if not paths: + raise android.FrameError('select at least one OBB file') + names = set() + for path in paths: + if (not re.fullmatch(r'(main|patch)\.[0-9]+\.' + re.escape(package) + r'\.obb', path.name) + or not path.is_file() or path.stat().st_size == 0 or path.name in names): + raise android.FrameError('OBB must be a nonempty main/patch..' + package + '.obb file') + names.add(path.name) + with android._install_lock: + meta = _meta(package) + container = 'lepton-steamlaunch-' + str(int(meta['instance'])) + # No implicit launch. Android resolves /sdcard, never an assumed host path. + running = android.ssh('podman ps --format "{{.Names}}"').splitlines() + if container not in running: + raise android.FrameError('start this app instance before installing OBB data') + dest = '/sdcard/Android/obb/' + package + results = [] + for path in paths: + digest = _sha256(path) + part = dest + '/.frame-' + uuid.uuid4().hex + '.part' + target = dest + '/' + path.name + script = (f'set -eu; mkdir -p {dest}; umask 002; ' + f'trap "rm -f {part}" EXIT; cat > {part}; ' + f'test "$(sha256sum {part} | cut -d " " -f 1)" = {digest}; ' + f'chmod 664 {part}; mv {part} {target}') + command = shlex.join(['podman', 'exec', '-i', container, '/system/bin/sh', '-c', script]) + with path.open('rb') as src: + _stream(command, src=src) + results.append({'name': path.name, 'path': target, 'sha256': digest}) + return {'package': package, 'instance': meta['instance'], 'obb': results, + 'verified': True} + + +def _data_command(action, meta): + container = 'lepton-steamlaunch-' + str(int(meta['instance'])) + # Fail closed if podman cannot enumerate containers; don't mistake errors for stopped. + guard = ('running=$(podman ps --format "{{.Names}}") || exit 1; ' + f'if printf "%s\\n" "$running" | grep -Fxq {shlex.quote(container)}; then ' + 'echo "stop the app before backup or restore" >&2; exit 1; fi; ') + return guard + shlex.join(['podman', 'unshare', 'python3', '-c', REMOTE.read_text(), + action, meta['package'], str(int(meta['instance']))]) + + +def backup_data(package, destination): + destination = Path(destination).expanduser().absolute() + if destination.exists(): + raise android.FrameError('backup destination already exists') + with android._install_lock: + meta = _meta(package) + fd, temporary = tempfile.mkstemp(prefix='.frame-backup-', dir=str(destination.parent)) + try: + with os.fdopen(fd, 'wb') as dst: + _stream(_data_command('backup', meta), dst=dst) + result = _inspect(temporary, meta) + # Exclusive publication: a concurrently created backup is never overwritten. + os.link(temporary, str(destination)) + return dict(result, path=str(destination), sha256=_sha256(destination)) + finally: + os.unlink(temporary) + + +def _inspect(path, meta): + import tarfile + try: + return runpy.run_path(str(REMOTE))['inspect_archive'](path, meta['package'], int(meta['instance'])) + except (OSError, EOFError, ValueError, tarfile.TarError) as error: + raise android.FrameError('invalid app-data backup: ' + str(error)) + + +def restore_data(package, archive): + with android._install_lock: + meta = _meta(package) + _inspect(archive, meta) + with open(archive, 'rb') as src: + result = _stream(_data_command('restore', meta), src=src) + try: + return json.loads(result) + except (ValueError, TypeError): + raise android.FrameError('could not read restore result; inspect app data before retrying')