diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 4f6738f..6ac075f 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -27,7 +27,10 @@ jobs: esac done - name: Python compiles - run: python -m py_compile ui/*.py apk-catalog/*.py frame/android/*.py + run: | + python -m py_compile ui/*.py apk-catalog/*.py frame/android/*.py + # Valve's devkit-utils (vendored; run by the Frame's python3). Most have no .py suffix. + python -m py_compile $(find frame/devkit-utils -type f ! -name '*.*' ! -name LICENSE) frame/devkit-utils/devkit_utils/*.py - name: Server tests run: python -m unittest discover -s tests -v - name: App syntax diff --git a/README.md b/README.md index 4756d45..0d5f56c 100644 --- a/README.md +++ b/README.md @@ -58,8 +58,8 @@ About 4,500 F-Droid apps rated for the Frame. One click installs each as its own -**📁 Files and clipboard**
-Drag files onto the window to send them. Send text or your clipboard straight to the headset's desktop. +**📁 Files, games and clipboard**
+Drag files onto the window to send them. Drop a game's .zip, folder or .exe to add it to the Steam library, with Proton or the Linux runtime picked for you. Send text or your clipboard straight to the headset's desktop. @@ -86,7 +86,8 @@ SSH, SFTP, Steam Link, remote desktop, volume, sleep, restart and shut down. Nothing is installed on the Frame for any of this: the app uses what SteamOS -already ships. [How each feature works](docs/frame-control.md). +already ships (sideloading a game copies Valve's own devkit scripts to +`~/devkit-utils`, as Valve's Devkit Client does). [How each feature works](docs/frame-control.md). ## Install @@ -156,7 +157,8 @@ computer. same network. For anywhere else, see [Tailscale](docs/tailscale.md). **What it changes:** only what you click. Installs go to your user account on -the Frame (`--user` Flatpaks, Lepton instances, Steam downloads), and nothing +the Frame (`--user` Flatpaks, Lepton instances, Steam downloads, sideloaded +games in `~/devkit-game`), and nothing needs `sudo` except the power buttons. On your computer it adds a `Host frame` entry to `~/.ssh/config` and keys at `~/.ssh/id_ed25519_frame` and `~/.ssh/id_rsa_frame_devkit` (the pairing service only takes RSA keys). @@ -183,6 +185,7 @@ Frame's software fits together, all checked against a real headset and labelled | [Scripts and headset setup](docs/scripts.md) | The command-line helpers, minimum typing, streaming options, floating panels | | [How the Frame works](docs/how-the-frame-works.md) | SteamVR → gamescope → Plasma, verified facts, debugging | | [Android apps (Lepton)](docs/apks.md) | Sideloading, the rated F-Droid catalogue, per-app instances | +| [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection | | [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build | | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [Open questions](docs/open-questions.md) | What's still unchecked | diff --git a/app/package.json b/app/package.json index 944d660..a075c51 100644 --- a/app/package.json +++ b/app/package.json @@ -55,6 +55,14 @@ "*.py" ] }, + { + "from": "../frame/devkit-utils", + "to": "frame/devkit-utils", + "filter": [ + "**/*", + "!**/__pycache__/**" + ] + }, { "from": "../apk-catalog", "to": "apk-catalog", diff --git a/app/preload.js b/app/preload.js index b921279..d1686a1 100644 --- a/app/preload.js +++ b/app/preload.js @@ -1,7 +1,10 @@ // Lets the page read this computer's clipboard through Electron, so sending it -// to the Frame needs no pbpaste, PowerShell, xclip or wl-clipboard. -const { contextBridge, ipcRenderer } = require("electron"); +// to the Frame needs no pbpaste, PowerShell, xclip or wl-clipboard. Also tells +// the page where a dropped file or folder lives, so a folder can be sideloaded +// as a title without zipping it (the local server reads it from there). +const { contextBridge, ipcRenderer, webUtils } = require("electron"); contextBridge.exposeInMainWorld("frameApp", { readClipboard: () => ipcRenderer.invoke("clipboard:read"), + pathForFile: (file) => { try { return webUtils.getPathForFile(file) || ""; } catch { return ""; } }, }); diff --git a/docs/frame-control.md b/docs/frame-control.md index c97cb8e..e0e9922 100644 --- a/docs/frame-control.md +++ b/docs/frame-control.md @@ -56,7 +56,10 @@ python3 ui/server.py # anywhere: then open http://127.0.0.1:47810 (0.85–1.3×) over ADB (`wm size`, `wm density`, `font_scale`). Reset puts all three back. Whether the settings survive the app relaunching is untested. - **Transfer**: drag and drop files to `~/Downloads`; `.apk` files install as - their own Android app. Send typed text, or your computer's clipboard, to the + their own Android app. A game's `.zip`, folder or `.exe` becomes a title in + the Steam library (Valve's Devkit Game path, with Proton or the Steam Linux + Runtime picked from the program's header), listed under **Sideloaded titles** + with Launch and Remove; see [sideloading.md](sideloading.md). Send typed text, or your computer's clipboard, to the Frame clipboard. - **Flatpaks**: install and remove them (quick picks: Moonlight, Firefox, VLC, Remmina). @@ -69,7 +72,7 @@ python3 ui/server.py # anywhere: then open http://127.0.0.1:47810 `app/` is an Electron shell. It starts `ui/server.py` on a free loopback port and shows it in its own window; the server stops when you quit the app. The -app bundles `ui/`, `scripts/`, `frame/android/` and the rated catalogue from +app bundles `ui/`, `scripts/`, `frame/android/`, Valve's `frame/devkit-utils/` and the rated catalogue from `apk-catalog/`, plus a standalone Python ([python-build-standalone](https://github.com/astral-sh/python-build-standalone)) and `adb` from Google's platform-tools, so there's nothing else to install. It @@ -95,7 +98,7 @@ library capsules and green Play buttons. both capture modes (headset view while in use, and a blank frame in standby, which the UI labels), clipboard, volume, file push, and input validation. **Not yet exercised from the UI:** Launch, Flatpak install/remove, APK drop, -and the power buttons. Each of these calls a command that was verified +title sideloading (not yet run on a headset at all), and the power buttons. Each of these calls a command that was verified separately. ## Per-platform notes diff --git a/docs/sideloading.md b/docs/sideloading.md new file mode 100644 index 0000000..741a360 --- /dev/null +++ b/docs/sideloading.md @@ -0,0 +1,153 @@ +# Sideloading Linux and Windows games + +A game you have as files (an itch.io download, your own build, a DRM-free +release) can go into the Frame's Steam library without a Steam store page. +Frame Control uses the same path as Valve's +[SteamOS Devkit Client](https://gitlab.steamos.cloud/devkit/steamos-devkit): +the title becomes a Steam **Devkit Game**, with a runtime (Proton or a Steam +Linux Runtime) chosen from the program itself. + +For Android APKs, see [apks.md](apks.md) instead. + +**Status: nothing here has run on a headset yet.** Every device-side step is +**inferred from Valve's steamos-devkit source** (release v0.20260925.1). The +local steps (reading the zip, picking the program and runtime, building the +request) are covered by `tests/test_frame_titles.py`. + +## Using it + +Drop a game's `.zip`, folder or `.exe` on **Send to Frame**. (Folders need the +desktop app, which knows where a dropped folder lives; in a plain browser, zip +it.) A dialog shows: + +- **Name**: what Steam shows. Steam uses the title id as the name, so it's + limited to letters, digits, `_` and `-`; the dialog shows the result. +- **Launches**: the program picked to start the game, with the other + candidates in the list. +- **Runtime**: picked from the program, see below. Windows programs can switch + between Proton Experimental and Proton (stable). + +Install copies it to the Frame and registers it with Steam; progress shows in +the bar and the activity log. **Sideloaded titles** lists what's installed, +with Launch and Remove. **Copy to ~/Downloads instead** keeps the old +behaviour for a zip that isn't a game. + +From a terminal: + +```sh +python3 ui/frame_titles.py inspect Game.zip # what would be installed, no headset needed +python3 ui/frame_titles.py install Game.zip [--name N] [--exe REL] [--runtime R] +python3 ui/frame_titles.py list | launch ID | remove ID +``` + +## Choosing the runtime + +The program's header decides, not its file name: + +| Program | Runtime (Steam compat tool) | `steam_play` | Confidence | +|---|---|---|---| +| Windows `.exe`, x86-64 (PE machine `0x8664`) | `proton-experimental` | 1 | Inferred: ARM64 Proton runs x86-64 code through FEX | +| Windows `.exe`, 32-bit x86 (`0x14c`) or ARM64 (`0xaa64`) | `proton-experimental` | 1 | Inferred | +| Linux ELF, aarch64 (`e_machine` `0xB7`) | `SteamLinuxRuntime_4-arm64` | 0 | Inferred: native | +| Linux ELF, x86-64 (`0x3E`) | `SteamLinuxRuntime_4` | 0 | Inferred: runs through FEX; the least certain row | +| Shell script | the runtime of the Linux binary beside it, else `SteamLinuxRuntime_4-arm64` | 0 | Guess | +| Anything else (32-bit Linux, other CPUs, DLLs, data) | refused with a message | | | + +Proton Experimental is the default rather than stable because the Frame's +ARM64 Proton and FEX stack is new and Proton fixes reach Experimental first. +If a game misbehaves, reinstall it with Proton (stable). + +The aliases and settings are the ones Valve's client sends: `RUNTIME_ALIASES` +in `devkit_client/__init__.py`, and `gui2._update_game`, which sets +`steam_play=1, steam_play_debug=0, steam_play_debug_version=2019` for Proton +and `steam_play=0` otherwise, plus `compat_tool=`. Valve's client only +offers `SteamLinuxRuntime_4-arm64` and Lepton when the device reports itself +as Deckard (the Frame). + +## Picking the program + +`ui/frame_titles.py` reads every file's header: ELF executables (PIE ones are +told from shared libraries by their `PT_INTERP` segment), PE executables (not +DLLs) and scripts with `#!`. A zip with a single top-level folder is treated +as that folder. Candidates are ranked by: + +1. Not a helper: names like `UnityCrashHandler64`, `CrashReportClient`, + `*setup*`, `unins*`, `vc_redist*`, `dxsetup`, `*prereq*`, and anything under + `_CommonRedist`, `Redist`, `DirectX` or `Engine` go last. +2. Platform: native ARM64 Linux, then Windows x86-64, then x86-64 Linux, then + other Windows builds. +3. Name: a program named like the zip or folder (build words such as + `-linux-arm64` or `_v1.2` are dropped from the name). +4. Depth, then size: Unreal's top-level `Game.exe` beats + `Game/Binaries/Win64/Game-Win64-Shipping.exe`. + +A top-level shell script beats a Linux binary one folder down (`run.sh` + +`bin/game`); a binary next to a script wins. The list in the dialog lets you +pick another. + +## What happens on the Frame (inferred) + +1. **Tools.** `frame/devkit-utils/` (Valve's scripts, vendored unmodified, MIT) + is copied to `~/devkit-utils`, where Valve's client puts it, unless the + stamp file there already matches. Files are merged, not replaced, so a + newer copy from Valve's client keeps its extra files. +2. **Folder.** `python3 ~/devkit-utils/steamos-prepare-upload --gameid ID` + makes `~/devkit-game/ID` and prints `{user, directory}`. +3. **Copy.** The files go there with `rsync -a --delete` on macOS and Linux, + or `scp -r` into a fresh folder that then replaces it on Windows. Then + `chmod -R 755`, the modes Valve's client gives an upload. +4. **Register.** `python3 ~/devkit-utils/steam-client-create-shortcut --parms JSON` + with `{gameid, directory, argv: [target], env: {}, settings, clear_settings, + force_appid: "", lepton_args: ""}`. It writes `ID-argv.json`, + `ID-env.json` and `ID-settings.json` next to the folder, then sends + `create-shortcut` to the running Steam client over `~/.steam/steam.pipe` + (authenticated by `~/.steam/steam.token`) and waits up to 5 s for Steam's + answer file. Its `error`, for example "The Steam client is not running", + is shown as the install error. The files stay, so installing again with + Steam running finishes the job. +5. **Launch** is `steam-devkit-rpc run-game gameid=ID`. **Remove** is + `steamos-delete --delete-title ID`, which deletes the folder and has Steam + drop shortcuts with no folder. Frame Control then removes the `ID-*.json` + files that Valve's script leaves behind. + +Frame Control also writes `~/devkit-game/ID-framecontrol.json` (name, source +file, target, runtime, size). **Sideloaded titles** lists every folder in +`~/devkit-game`, including titles uploaded with Valve's client. + +`argv` is one string, as in Valve's client (the start command may carry +arguments), so a program path with spaces is sent in double quotes. How Steam +splits that string is **not checked**. + +## Safety + +- Zips are unpacked on your computer first. Entries with absolute paths, `..`, + drive letters or `:` anywhere in the path, or links that point outside the + zip (or at a folder they're in) are refused. So are zips over 64 GB + unpacked, over 200,000 entries, more than 200× compressed past 1 GB, or + bigger than the free space. +- No symlink is created while unpacking, so no write can be redirected + through one. A link to a file inside the zip (`libfoo.so.1 → libfoo.so.1.2`) + becomes a copy of that file, which also works on Windows. Links to folders, + loops and dangling links are left out. +- A dropped folder that contains symlinks (or Windows junctions) is copied on your computer first, + with the same rule, because `scp -r` would follow a link out of the folder + and upload whatever it points at. +- Installs run one at a time, and Remove is refused while one runs. +- The title id is limited to `[A-Za-z0-9_-]`, at most 64 characters. Valve's + scripts pass it to a shell (`steamos-delete` runs `rm -r` on it). Valve's + reserved sideload names (`steam`, `steamvr`, and their `deckard` forms, + which would replace the Steam client itself) get `-game` added. +- Nothing needs `sudo`; everything goes to your home folder on the Frame. +- In the app, a dropped folder is read from its local path by the app's own + server, which only accepts requests from its own page (see + [frame-control.md](frame-control.md#how-it-works)). + +## To check on a headset + +- [ ] `create-shortcut` registers a title and it shows in the library under its id. +- [ ] An aarch64 build launches in `SteamLinuxRuntime_4-arm64`. +- [ ] An x86-64 Windows `.exe` launches under Proton Experimental through FEX. +- [ ] An x86-64 Linux build launches in `SteamLinuxRuntime_4` through FEX. +- [ ] `steam-devkit-rpc run-game` starts it, and `steamos-delete` removes the shortcut. +- [ ] How Steam splits a start command with a quoted path. +- [ ] Whether these titles open as flat panels or need anything VR-specific. diff --git a/frame/devkit-utils/LICENSE b/frame/devkit-utils/LICENSE new file mode 100644 index 0000000..bd7a5ef --- /dev/null +++ b/frame/devkit-utils/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2017-2022 Valve Software inc., Collabora Ltd + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/frame/devkit-utils/README.md b/frame/devkit-utils/README.md new file mode 100644 index 0000000..f97cbbe --- /dev/null +++ b/frame/devkit-utils/README.md @@ -0,0 +1,18 @@ +# Valve's devkit-utils (vendored) + +Unmodified copy of `client/devkit-utils/` from Valve's +[SteamOS Devkit Client](https://gitlab.steamos.cloud/devkit/steamos-devkit), +MIT licensed (see `LICENSE`; Valve's own notes are in `VALVE-README.md`). + +- Source: steamos-devkit, commit `6f0711a` ("Code drop."), + release **v0.20260925.1** (ChangeLog entry dated 2026-09-25). + +`ui/frame_titles.py` copies this folder to `~/devkit-utils` on the Frame (where +Valve's own client puts it) and uses `steamos-prepare-upload`, +`steam-client-create-shortcut`, `steam-devkit-rpc` and `steamos-delete` to +register uploaded builds as Steam "Devkit Games". See `docs/sideloading.md`. + +To update: copy the folder from a newer checkout over this one, keep this +README, and update the version line above. The stamp Frame Control compares +on the headset is a hash of these files, so a changed copy is re-synced on the +next use. diff --git a/frame/devkit-utils/VALVE-README.md b/frame/devkit-utils/VALVE-README.md new file mode 100644 index 0000000..585a691 --- /dev/null +++ b/frame/devkit-utils/VALVE-README.md @@ -0,0 +1,4 @@ +These scripts and supporting utility module are uploaded to the devkit by the devkit client: + +- steamos-* : scripts that operate (mostly) at SteamOS level for devkit functionality purposes +- steam-client-* : scripts that relay commands to the local running Steam client diff --git a/frame/devkit-utils/deckard-capture b/frame/devkit-utils/deckard-capture new file mode 100755 index 0000000..6cae14f --- /dev/null +++ b/frame/devkit-utils/deckard-capture @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- + +import sys +import os +import time +import subprocess +import logging +import argparse +import json +import datetime +import io + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +def main(): + parser = argparse.ArgumentParser(description='Capture screenshot and videos on Steam Frame device') + parser.add_argument('--filename', '-f', + default='/tmp/screenshot.png', + help='Output') + parser.add_argument('--timestamp', action='store_true', + help='Add timestamp') + parser.add_argument('--json', action='store_true', + help='Output result as JSON') + + args = parser.parse_args() + + output_buffer = io.StringIO() + try: + steamvr_path = subprocess.check_output(['steamvr', 'path'], stderr=subprocess.STDOUT, universal_newlines=True).strip() + cdd = os.path.join(steamvr_path, 'bin/linuxarm64') + run_vrcmd = os.path.join(cdd, 'vrcmd') + assert os.path.exists(run_vrcmd), "vrcmd not found" + + # Enable recording + cmd = [run_vrcmd, '--mailboxcmd', 'vrcompositor_systemlayer', 'set_local_video_record?enabled=true'] + output_buffer.write(f"Command: {' '.join(cmd)}\n") + result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output_buffer.write(result.stdout) + if result.returncode != 0: + raise subprocess.CalledProcessError(result.returncode, cmd) + + # Wait for the video device to produce frames. + # Note that even when disabled it outputs roughly 2 blank frames per second. + cmd = ['timeout', '1', 'ffmpeg', '-f', 'v4l2', '-i', '/dev/video99', '-frames:v', '4', '-f', 'null', '-', '-v', 'error'] + output_buffer.write(f"Command: {' '.join(cmd)}\n") + retries = 2 + while True: + result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output_buffer.write(result.stdout) + if result.returncode == 0: + break + retries -= 1 + if retries <= 0: + raise Exception("Failed to get video frames from /dev/video99. Is VR active? Is the v4l2 configuration correct?") + + output_filename = args.filename + if args.timestamp: + timestamp = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S") + base, ext = os.path.splitext(output_filename) + output_filename = f"{base}-{timestamp}{ext}" + + # Capture screenshot + cmd = ['ffmpeg', '-f', 'v4l2', '-i', '/dev/video99', '-frames:v', '1', '-q:v', '1', '-y', output_filename] + output_buffer.write(f"Command: {' '.join(cmd)}\n") + result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output_buffer.write(result.stdout) + if result.returncode != 0: + raise subprocess.CalledProcessError(result.returncode, cmd) + + # Disable recording + cmd = [run_vrcmd, '--mailboxcmd', 'vrcompositor_systemlayer', 'set_local_video_record?enabled=false'] + output_buffer.write(f"Command: {' '.join(cmd)}\n") + result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output_buffer.write(result.stdout) + if result.returncode != 0: + raise subprocess.CalledProcessError(result.returncode, cmd) + + except Exception as e: + error_msg = str(e) + # Print collected output to stderr on error + print(output_buffer.getvalue(), file=sys.stderr) + logger.error(error_msg) + + if args.json: + result = { + 'success': False, + 'error': error_msg + } + print(json.dumps(result)) + else: + print(f"Error: {error_msg}", file=sys.stderr) + + return 1 + + if args.json: + result = { + 'success': True, + 'output': output_filename + } + print(json.dumps(result)) + else: + print(f"Screenshot saved to {output_filename}") + +if __name__ == '__main__': + sys.exit(main()) \ No newline at end of file diff --git a/frame/devkit-utils/devkit_utils/__init__.py b/frame/devkit-utils/devkit_utils/__init__.py new file mode 100644 index 0000000..f02d7d4 --- /dev/null +++ b/frame/devkit-utils/devkit_utils/__init__.py @@ -0,0 +1,300 @@ +#!/usr/bin/env python +# encoding: utf-8 +"""Utility functions for the Steam client hook scripts""" + +import sys +import os +import traceback +import tempfile +import json +import logging +import fcntl +import errno +import contextlib +import time +import fcntl + + +import logging as logging_module +logger = logging_module.getLogger(__name__) + + +@contextlib.contextmanager +def wrap_outputs(stderr_prefix): + # capture stderr to file to support debugging + stderr_fd = sys.stderr.fileno() + tf = tempfile.NamedTemporaryFile( + mode='w+', + prefix=stderr_prefix, + delete=True) + if sys.version_info >= (3, 4): + # this API in the os module is only available for python3 + # but it does not seem to work with subprocess anyway + os.set_inheritable(tf.file.fileno(), True) + assert os.get_inheritable(tf.file.fileno()) + sys.stderr = tf.file + + # we can only write out a json response to stdout, + # so redirect stdout to stderr, + # and keep a handle on the original stdout for the response + stdout_fd = os.dup(sys.stdout.fileno()) + os.dup2(sys.stderr.fileno(), sys.stdout.fileno()) + + ctx = {} + try: + yield ctx + except: + logger.error(traceback.format_exc()) + finally: + tf.flush() + tf.seek(0) + os.write(stderr_fd, tf.read().encode('utf-8')) + if 'ret' in ctx: + os.write(stdout_fd, json.dumps(ctx['ret']).encode('utf-8')) + + +class SteamClientNotRunningException(Exception): + def __init__(self, error_message): + self.error_message = error_message + + def __str__(self): + return self.error_message + + +def validate_steam_client(): + """Verify that the steam client is running, and permissions are adequate""" + pid_path = os.path.normpath( + os.path.realpath( + os.path.expanduser('~/.steam/steam.pid'))) + if not os.path.exists(pid_path): + raise SteamClientNotRunningException('{0} does not exist'.format(pid_path)) + try: + pid = int(open(pid_path, 'rt').read()) + except Exception: + raise SteamClientNotRunningException('{0} is invalid'.format(pid_path)) + try: + os.kill(pid, 0) + except OSError: + raise SteamClientNotRunningException('{0} does not refer to a valid process'.format(pid_path)) + logger.info('Found steam client pid %s', pid) + + +def execute_steam_client_command(cmd): + """Send a command to the steam client over the IPC pipe""" + pipe_path = os.path.normpath( + os.path.realpath( + os.path.expanduser('~/.steam/steam.pipe'))) + try: + pipe = open(pipe_path, 'wb+', 0) + except IOError: + raise Exception('cannot open steam client pipe') + session_token = open(os.path.expanduser('~/.steam/steam.token')).read() + #pipe_cmd = 'steam://{0}'.format(cmd) + # ^ hack to execute a normal command over the IPC directly - sometimes useful + pipe_cmd = 'devkit-1 steam://devkit-1/{0}/{1}'.format( + session_token, + cmd + ) + logger.debug('Sending command line:') + logger.debug(pipe_cmd) + pipe.write('{0}\n'.format(pipe_cmd).encode('utf-8')) + pipe.close() + + +def save_argv(gameid, argv): + """Save command line and arguments if provided""" + + if argv is None: + return + + argvfile = os.path.join(os.getenv("HOME"), "devkit-game", + gameid + "-argv.json") + try: + with open(argvfile, "w") as argvf: + fcntl.flock(argvf, fcntl.LOCK_EX) + json.dump(argv, argvf) + fcntl.flock(argvf, fcntl.LOCK_UN) + except IOError: + raise Exception( + "Unable to open argv file for writing: {0}".format(argvfile)) + + +def obtain_argv(gameid, argv): + """Obtain command line with arguments""" + + # If present and not None or [], just return the local arguments + if argv: + return argv + + # From here, expect arguments to have been saved previously + argvfile = os.path.join(os.getenv("HOME"), "devkit-game", + gameid + "-argv.json") + try: + with open(argvfile, "r") as argvf: + fcntl.flock(argvf, fcntl.LOCK_EX) + argv = json.load(argvf) + fcntl.flock(argvf, fcntl.LOCK_UN) + except IOError: + raise Exception( + "Unable to open argv file for reading: {0}".format(argvfile)) + return argv + + +def save_env(gameid, env): + """Save environment variables if provided""" + + if not env: + return + + envfile = os.path.join(os.getenv("HOME"), "devkit-game", + gameid + "-env.json") + try: + with open(envfile, "w") as envf: + fcntl.flock(envf, fcntl.LOCK_EX) + json.dump(env, envf) + fcntl.flock(envf, fcntl.LOCK_UN) + except IOError: + raise Exception( + "Unable to open env file for writing: {0}".format(envfile)) + + +def obtain_env(gameid): + """Obtain environment variables for a game, if any were saved""" + + envfile = os.path.join(os.getenv("HOME"), "devkit-game", + gameid + "-env.json") + try: + with open(envfile, "r") as envf: + fcntl.flock(envf, fcntl.LOCK_EX) + env = json.load(envf) + fcntl.flock(envf, fcntl.LOCK_UN) + except IOError: + return {} + return env + + +def save_settings(gameid, data): + """Save settings""" + settingsfile = os.path.join(os.getenv("HOME"), "devkit-game", + gameid + "-settings.json") + settings = dict() + + if data.get('clear_settings', False): + settings = {} + else: + try: + with open(settingsfile, "r") as f: + fcntl.flock(f, fcntl.LOCK_EX) + settings = json.load(f) + fcntl.flock(f, fcntl.LOCK_UN) + except IOError as e: + if (e.errno != errno.ENOENT): + raise + + # Merge settings from new json + if 'settings' in data: + settings.update(data['settings']) + + try: + with open(settingsfile, "w") as f: + fcntl.flock(f, fcntl.LOCK_EX) + json.dump(settings, f) + fcntl.flock(f, fcntl.LOCK_UN) + except (IOError): + raise Exception( + "Unable to open settings file for writing: {0}".format( + settingsfile + )) + + return settings + + +def load_settings(gameid): + settingsfile = os.path.join(os.getenv("HOME"), "devkit-game", gameid + '-settings.json') + + if not os.path.isfile(settingsfile): + return None + + with open(settingsfile, "r") as f: + fcntl.flock(f, fcntl.LOCK_EX) + settings = json.load(f) + fcntl.flock(f, fcntl.LOCK_UN) + + return settings + + +class SteamResponse_Timeout(Exception): + pass + + +class SteamResponse_Error(Exception): + def __init__(self, error_response): + self.error_response = error_response + + def __str__(self): + return self.error_response + + +@contextlib.contextmanager +def wait_on_file_response(path, timeout=5): + """ +The pipe to the Steam Client is one way. +Responses from the Steam Client are written to filesystem. +Protocol is as follows: +- Steam Client creates a 'path.lock' file +- Steam Client writes either 'path' or 'path.error' to indicate a problem +- Steam Client deletes 'path.lock' +- Caller (us) can then read the response + +NOTE 1: this function is used as a context manager and will block until a response comes in or timeout. + +NOTE 2: the files are created by Steam when responding to a command. If the files already exist the response protocol will break. + """ + lock_path = '{0}.lock'.format(path) + error_path = '{0}.error'.format(path) + max_count = timeout + while True: + time.sleep(1) + if os.path.exists(error_path) or os.path.exists(path) and not os.path.exists(lock_path): + if os.path.exists(error_path): + with open(error_path, 'r') as f: + fcntl.flock(f, fcntl.LOCK_EX) + error_response = f.read() + fcntl.flock(f, fcntl.LOCK_UN) + raise SteamResponse_Error(error_response) + with open(path, 'r') as f: + fcntl.flock(f, fcntl.LOCK_EX) + success_response = f.read() + yield success_response + fcntl.flock(f, fcntl.LOCK_UN) + return + max_count -= 1 + if max_count > 0: + continue + raise SteamResponse_Timeout() + + +# Setting up as a context manager so we never miss the deletion +# Creating a temporary .lock file to guard the create operation +@contextlib.contextmanager +def create_pid(pid_path): + os.makedirs(os.path.dirname(pid_path), exist_ok=True) + lock_path = '{0}.lock'.format(pid_path) + try: + lock_file = os.open(lock_path, os.O_CREAT | os.O_EXCL) + except IOError as e: + logger.error('cannot create lock file %s for pid file %s', lock_path, pid_path) + logger.error('remove the lock file manually and run again if you are confident no other instance is active') + raise + + pid_file = open(pid_path,'w') + pid_file.write(str(os.getpid())) + pid_file.flush() + os.close(lock_file) + os.unlink(lock_path) + try: + yield pid_file + finally: + pid_file.close() + # Assume that's atomic and all is well, no need for another .lock + os.unlink(pid_path) diff --git a/frame/devkit-utils/devkit_utils/resolve.py b/frame/devkit-utils/devkit_utils/resolve.py new file mode 100644 index 0000000..1329926 --- /dev/null +++ b/frame/devkit-utils/devkit_utils/resolve.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 + +import sys +import os +import logging +from urllib.parse import quote_plus as urllib_quote_plus +import json +import tempfile + +from . import validate_steam_client +from . import execute_steam_client_command +from . import wait_on_file_response + +import logging as logging_module +logger = logging_module.getLogger(__name__) + + +def resolve_shortcuts(): + # make sure there is a steam client online that we can talk to before doing anything + validate_steam_client() + + # scan the devkit games + installed_gameids = set([]) + devkit_game_path = os.path.expanduser('~/devkit-game') + if not os.path.exists(devkit_game_path): + logger.info('%r does not exist, creating', devkit_game_path) + os.mkdir(devkit_game_path) + entries = sorted(os.scandir(devkit_game_path), key=lambda entry: entry.name) + directories = [e for e in entries if e.is_dir()] + for d in directories: + gameid = d.name + file_names = [f.name for f in entries if f.is_file() and f.name.startswith(gameid)] + has_argv = '{0}-argv.json'.format(gameid) in file_names + has_settings = '{0}-settings.json'.format(gameid) in file_names + if (not has_argv and not has_settings): + logger.info('Subfolder %r in %r is not accompanied by devkit configuration files, ignoring', d.name, devkit_game_path) + continue + logger.info('Found installed Devkit Game: %r', gameid) + installed_gameids.add(gameid) + + # ask the Steam Client which Devkit Games are registered + with tempfile.TemporaryDirectory(prefix='list-shortcuts') as tempdir: + response = os.path.join(tempdir, 'shortcuts.json') + cmd = 'list-shortcuts?response={}'.format( + urllib_quote_plus(os.path.join(response)) + ) + # send the request + execute_steam_client_command(cmd) + with wait_on_file_response(response) as response: + client_shortcuts = json.loads(response) + logger.debug(client_shortcuts) + assert client_shortcuts['version'] == 2 + registered_gameids = set([]) + logger.info('Steam Client has %d registered devkit game(s)', len(client_shortcuts['gameids'])) + for gameid in client_shortcuts['gameids']: + logger.info('Found Devkit Game registered with Steam Client: %r', gameid) + registered_gameids.add(gameid) + + # any registered game that is not found installed on disk needs to be removed + for remove_gameid in registered_gameids - installed_gameids: + with tempfile.TemporaryDirectory(prefix='delete-shortcut') as tempdir: + logger.info('Removing stale registered Devkit Game: %r', remove_gameid) + response = os.path.join(tempdir, 'shortcut-deleted') + cmd = 'delete-shortcut?response={}&gameid={}'.format( + urllib_quote_plus(response), + remove_gameid + ) + execute_steam_client_command(cmd) + with wait_on_file_response(response) as response: + logger.info('from Steam Client: %s', response.strip()) + + # any installed game that is not found registered needs to be added + for add_gameid in installed_gameids - registered_gameids: + with tempfile.TemporaryDirectory(prefix='create-shortcut') as tempdir: + logger.info('Registering installed Dekit Game: %r', add_gameid) + response = os.path.join(tempdir, 'registered') + cmd = 'create-shortcut?response={}&gameid={}&directory={}'.format( + urllib_quote_plus(response), + add_gameid, + urllib_quote_plus(devkit_game_path) + ) + execute_steam_client_command(cmd) + with wait_on_file_response(response) as response: + logger.info('from Steam Client: %s', response.strip()) + +if __name__ == '__main__': + resolve_shortcuts() diff --git a/frame/devkit-utils/steam-client-create-shortcut b/frame/devkit-utils/steam-client-create-shortcut new file mode 100644 index 0000000..802edce --- /dev/null +++ b/frame/devkit-utils/steam-client-create-shortcut @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 + +import os +import logging +import argparse +import json +import platform +import tempfile +from urllib.parse import quote_plus as urllib_quote_plus + +import devkit_utils + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger() + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--parms', required=True, action='store') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + parms = json.loads(conf.parms) + gameid = parms['gameid'] + directory = parms['directory'] + assert os.path.isdir(directory) + + force_appid = parms['force_appid'] + steam_appid_path = os.path.join(directory, 'steam_appid.txt') + if force_appid: + with open(steam_appid_path, 'w') as f: + f.write(force_appid + '\n') + logger.info(f'Wrote {steam_appid_path} with AppID {force_appid}') + elif os.path.exists(steam_appid_path): + # don't overwrite an existing steam_appid.txt file from the content tree + # NOTE: if the user sets an AppID through the tool, then delete it, we may leave it in place .. + # (that's ok for now, do a clean upload if you want to get rid of it) + logger.info(f'{steam_appid_path} already exists, leaving it in place') + + # Lepton (Android runtime) titles: write UECommandLine.txt next to the .apk + is_lepton = parms['settings']['compat_tool'] == 'lepton' + uecommandline = parms['lepton_args'] if is_lepton else '' + uecommandline_path = os.path.join(directory, 'UECommandLine.txt') + if uecommandline: + with open(uecommandline_path, 'w') as f: + f.write(uecommandline + '\n') + logger.info(f'Wrote {uecommandline_path}') + elif os.path.exists(uecommandline_path): + # don't overwrite an existing UECommandLine.txt file from the content tree + # NOTE: if the user sets cmdline args through the tool, then clears them, we may leave it in place .. + # (that's ok for now, do a clean upload if you want to get rid of it) + logger.info(f'{uecommandline_path} already exists, leaving it in place') + + logger.info(f'Updating command line and runtime settings for {gameid} on {platform.node()}') + devkit_utils.save_argv(gameid, parms['argv']) + devkit_utils.save_env(gameid, parms['env']) + devkit_utils.save_settings(gameid, parms) + + ret = {} + + try: + devkit_utils.validate_steam_client() + except devkit_utils.SteamClientNotRunningException as e: + skipping = 'The Steam client is not running. Registration did not complete.' + logger.warning(skipping) + ret['error'] = skipping + else: + with tempfile.TemporaryDirectory(prefix='create-shortcut') as tempdir: + logger.info(f'Registering Devkit Game {gameid} with Steam Client') + response = os.path.join(tempdir, 'registered') + cmd = 'create-shortcut?response={}&gameid={}'.format( + urllib_quote_plus(response), + gameid, + ) + devkit_utils.execute_steam_client_command(cmd) + try: + with devkit_utils.wait_on_file_response(response) as success_response: + logger.debug(success_response) + ret['success'] = success_response + except devkit_utils.SteamResponse_Timeout: + ret['error'] = 'timeout - Steam client did not respond to registration request' + except devkit_utils.SteamResponse_Error as e: + ret['error'] = e.error_response + + # response gets written out to stdout + print(json.dumps(ret)) diff --git a/frame/devkit-utils/steam-devkit-rpc b/frame/devkit-utils/steam-devkit-rpc new file mode 100755 index 0000000..409542e --- /dev/null +++ b/frame/devkit-utils/steam-devkit-rpc @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 + +import sys +import os +import logging +import argparse +import tempfile +import urllib.parse +import re + +import devkit_utils + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger() + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('command') + parser.add_argument('args', nargs='*') + conf = parser.parse_args() + + try: + devkit_utils.validate_steam_client() + except devkit_utils.SteamClientNotRunningException as e: + logger.error(repr(e)) + sys.exit(-1) + else: + with tempfile.TemporaryDirectory(prefix='steam-devkit-rpc') as tempdir: + response = os.path.join(tempdir, 'steam-devkit-rpc') + parms = { + 'response' : response, + } + for arg in conf.args: + (k, v) = re.split('=', arg) + parms[k] = v + cmd = f'{conf.command}/?{urllib.parse.urlencode(parms)}' + devkit_utils.execute_steam_client_command(cmd) + try: + with devkit_utils.wait_on_file_response(response) as success_response: + logger.info('success') + sys.stdout.write(success_response) + sys.exit(0) + except devkit_utils.SteamResponse_Timeout: + logger.error('timeout') + except devkit_utils.SteamResponse_Error as e: + logger.error('failed') + sys.stdout.write(e.error_response) + sys.exit(-1) diff --git a/frame/devkit-utils/steamos-delete b/frame/devkit-utils/steamos-delete new file mode 100644 index 0000000..c73a1bd --- /dev/null +++ b/frame/devkit-utils/steamos-delete @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 + +import sys +import os +import shutil +import logging +import argparse +import subprocess + +import devkit_utils.resolve + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') + +def session_select_command(): + if shutil.which('holo-session-select'): + return 'holo-session-select' + return 'steamos-session-select' + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--delete-title', required=False, action='store', help='Delete a devkit title by name') + parser.add_argument('--delete-all-titles', required=False, action='store_true', default=False, help='Delete all devkit titles uploaded') + parser.add_argument('--reset-steam-client', required=False, action='store_true', default=False, help='Reset Steam client and delete all local Steam content') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + if conf.delete_all_titles: + subprocess.check_call('rm -rf ~/devkit-game/*', shell=True) + elif conf.delete_title: + gamepath = os.path.expanduser( os.path.join( '~/devkit-game', conf.delete_title ) ) + if not os.path.isdir(gamepath): + print(f'Not found: {gamepath}') + else: + subprocess.check_call(f'rm -r {gamepath}', shell=True) + + # synchronize the Steam client's view of the devkit games with the on disk state + try: + devkit_utils.resolve.resolve_shortcuts() + except Exception as e: + logger.warning(f'Steam client sync of devkit games failed: {e}') + + if conf.reset_steam_client: + # first make sure any sideloaded trampoline has been deleted + devkit_steam_trampoline_path = os.path.join(DEVKIT_TOOL_FOLDER, 'devkit-steam') + if os.path.exists(devkit_steam_trampoline_path): + os.unlink(devkit_steam_trampoline_path) + + # wipe the local Steam install + subprocess.check_call(f'rm -rf ~/.local/share/Steam', shell=True) + + # restart the session, which will initiate a reinstall of Steam from the OS client + subprocess.check_call([session_select_command(), 'gamescope']) diff --git a/frame/devkit-utils/steamos-dump-controller-config b/frame/devkit-utils/steamos-dump-controller-config new file mode 100644 index 0000000..4195119 --- /dev/null +++ b/frame/devkit-utils/steamos-dump-controller-config @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 + +import os +import logging +import argparse +import tempfile +import json +from urllib.parse import quote_plus as urllib_quote_plus + +import devkit_utils + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--appid', required=False, action='store') + parser.add_argument('--gameid', required=False, action='store') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + ret = {} + + try: + devkit_utils.validate_steam_client() + except devkit_utils.SteamClientNotRunningException as e: + skipping = 'The Steam client is not running.' + logger.warning(skipping) + ret['error'] = skipping + else: + with tempfile.TemporaryDirectory(prefix='controller-config') as tempdir: + response = os.path.join(tempdir, 'dumpcontrollerconfig') + cmd = f'dumpcontrollerconfig?response={urllib_quote_plus(response)}' + if conf.appid: + cmd += f'&appid={conf.appid}' + if conf.gameid: + cmd += f'&gameid={conf.gameid}' + logger.debug(f'command: {cmd}') + devkit_utils.execute_steam_client_command(cmd) + try: + with devkit_utils.wait_on_file_response(response) as success_response: + logger.debug(success_response) + ret['success'] = success_response + except devkit_utils.SteamResponse_Timeout: + ret['error'] = 'timeout - Steam did not respond to the command request' + except devkit_utils.SteamResponse_Error as e: + ret['error'] = e.error_response + + # response gets written out to stdout + print(json.dumps(ret)) diff --git a/frame/devkit-utils/steamos-get-status b/frame/devkit-utils/steamos-get-status new file mode 100755 index 0000000..caaa010 --- /dev/null +++ b/frame/devkit-utils/steamos-get-status @@ -0,0 +1,443 @@ +#!/usr/bin/env python3 + +import sys +import os +import re +import shutil +import subprocess +import logging +import enum +import argparse +import json +import shlex +import datetime +import pathlib +import socket + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') +STEAM_EXTRA_ARGS_FILE = os.path.expanduser('~/.config/systemd/user/steam.service.d/extra_args.conf') + +WIRELESS_DISABLE_POWER_MANAGEMENT = '/usr/bin/steamos-polkit-helpers/steamos-disable-wireless-power-management' + +# Must match in gui2.py +class SteamStatus(enum.Enum): + NOT_RUNNING = 0 + ERROR = 1 + OS = 2 + OS_DEV = 3 + SIDELOADED = 4 + + @classmethod + def from_string(cls, status_str): + if not status_str: + return cls.ERROR + try: + if '.' in status_str: + name = status_str.split('.')[-1] + else: + name = status_str + return cls[name] + except KeyError: + return cls.ERROR + + @property + def description(self): + DESCRIPTIONS = { + SteamStatus.NOT_RUNNING: 'not running', + SteamStatus.OS: 'OS client', + SteamStatus.OS_DEV: 'OS client dev mode', + SteamStatus.SIDELOADED: 'sideloaded client', + SteamStatus.ERROR: 'error', + } + return DESCRIPTIONS[self] + + +class SteamConfig(enum.Enum): + ERROR = 1 + # Matching the SteamStatus numeric values + OS = 2 + OS_DEV = 3 + SIDELOADED = 4 + + @property + def description(self): + DESCRIPTIONS = { + SteamConfig.OS: 'OS client', + SteamConfig.OS_DEV: 'OS client dev mode', + SteamConfig.SIDELOADED: 'sideloaded client', + SteamConfig.ERROR: 'error', + } + return DESCRIPTIONS[self] + + +SESSION_NAMES = ['gamescope', 'plasma-x11', 'plasma-x11-persistent', 'plasma-wayland', 'plasma-wayland-persistent'] + +class SessionConfig(enum.IntEnum): + # note: matches SESSION_NAMES indexes + GAMESCOPE = 0 + PLASMA_X11 = 1 + PLASMA_X11_PERSISTENT = 2 + PLASMA_WAYLAND = 3 + PLASMA_WAYLAND_PERSISTENT = 4 + ERROR = 5 + +def cef_debugging(): + '''Only sane way to check is to look for the listening port.''' + ret = subprocess.run('/usr/bin/ss -l -t -n -p | grep steamwebhelper | grep 8080 > /dev/null', shell=True) + return ( ret.returncode == 0 ) + +def steam_process_get_path_and_args(): + ret = subprocess.run(['pgrep', '-a', '-x', 'steam'], capture_output=True, text=True) + if ret.returncode != 0: + return None + # Proton may run a dummy 'steam' process that confused previous implementations of this logic + # look for a process who's real filename is 'steam' + for l in ret.stdout.splitlines(): + try: + pid = int(l.split(' ')[0]) + except: + continue + rp = os.path.realpath(f'/proc/{pid}/exe') + if os.path.basename(rp) == 'steam': + try: + with open(f'/proc/{pid}/cmdline', 'rb') as f: + cmdline = f.read() + argv = [a.decode('utf-8', errors='replace') for a in cmdline.split(b'\x00') if a] + if len(argv) >= 2: + path = argv[0] + args = argv[1:] + # strip -srt-logger-opened: injected by steam.sh at runtime + args = [a for a in args if a != '-srt-logger-opened'] + return (path, args) + return (argv[0], []) + except Exception as e: + logger.warning(f'Failed to read /proc/{pid}/cmdline: {e}') + return None + +def steam_process_get_path(): + try: + (path, _) = steam_process_get_path_and_args() + except: + return None + return path + +def steam_process_get_args(): + try: + (_, args) = steam_process_get_path_and_args() + except: + return '' + return args + +def steam_status(): + '''What is the status of the Steam client on the system?''' + s = steam_process_get_path() + if s is None: + return SteamStatus.NOT_RUNNING + if s.find('.local/share/Steam/') != -1: + if os.path.exists(os.path.expanduser('~/devkit-game/devkit-steam')): + return SteamStatus.OS_DEV + return SteamStatus.OS + if s.find('devkit-game/steam/') != -1 or s.find('devkit-game/steamdeckard/') != -1: + return SteamStatus.SIDELOADED + logger.warning(f'could not interpret pgrep result to determine steam client status: {s!r}') + return SteamStatus.ERROR + +def steam_configuration(): + '''How is the Steam client configured to run?''' + devkit_steam_trampoline_path = os.path.join(DEVKIT_TOOL_FOLDER, 'devkit-steam') + if not os.path.exists(devkit_steam_trampoline_path): + return SteamConfig.OS + t = open(devkit_steam_trampoline_path, 'rt').read() + if t.find('SteamStatus.OS_DEV') != -1: + return SteamConfig.OS_DEV + if t.find('SteamStatus.SIDELOADED') != -1: + return SteamConfig.SIDELOADED + logger.warning(f'could not determine what {devkit_steam_trampoline_path} means to do') + return SteamConfig.ERROR + + +def osclient_branch(is_deckard): + '''Which branch is the default Steam 'OS client' configured to use?''' + # makes more sense to return strings here + beta_path = os.path.expanduser('~/.steam/steam/package/beta') + if not os.path.exists(beta_path): + return 'default' # not sure that's valid actually - would be the desktop client, which will only run in desktop mode .. + t = open(beta_path, 'rt').readline().strip('\n') + try: + p = 'steamdeck_(.*)' + if re.match(p, t): + branch = re.split(p, t)[1] + return branch + # internal builds + p = 'steampal_(.*)_.*' + if re.match(p, t): + branch = re.split(p, t)[1] + return branch + if is_deckard: + p = 'linux_arm64_(.*)_.*' + if re.match(p, t): + branch = re.split(p, t)[1] + return branch + raise Exception('no match') + except: # noqa: E722 + logger.warning(f'could not determine the OS client branch config: {t!r}') + return 'error' + +def osclient_version(conf): + '''Which version is the Steam 'OS client'?''' + if conf.is_deckard: + # old Steam client was using linuxarm64/, which is now reserved for the SDK binaries + for folder in ('linuxarm64', 'steamrtarm64'): + fn = os.path.expanduser(f'~/.steam/steam/{folder}/builddate.txt') + if os.path.exists(fn): + return open(fn, 'rt').read() + return 'Unknown - no builddate.txt' + beta_path = os.path.expanduser('~/.steam/steam/package/beta') + if not os.path.exists(beta_path): + logger.warning(f'not found: {beta_path}') + return None + t = open(beta_path, 'rt').readline().strip('\n') + manifest = os.path.expanduser(f'~/.steam/steam/package/steam_client_{t}_ubuntu12.manifest') + if not os.path.exists(manifest): + logger.warning(f'not found: {manifest}') + return None + try: + version = int(re.search('"version".*"(.*)"', open(manifest,'rt').read()).group(1)) + return version + except: + logger.warning(f'could not parse version out of {manifest}') + return None + +def session_config(): + '''What is the graphics session configuration?''' + # RESTART_SESSION writes this file + conf_file = '/etc/sddm.conf.d/zz-steamos-autologin.conf' + if not os.path.exists(conf_file): + # fallback to the OS default + conf_file = '/etc/sddm.conf.d/steamos.conf' + if os.path.exists(conf_file): + s = open(conf_file, 'rt').read() + if s.find('plasmawayland.desktop') != -1: + return SessionConfig.PLASMA_WAYLAND_PERSISTENT + if s.find('plasma.desktop') != -1: + return SessionConfig.PLASMA_X11_PERSISTENT + if s.find('gamescope-wayland.desktop') != -1: + return SessionConfig.GAMESCOPE + if s.find('plasma-steamos-oneshot.desktop') != -1: + return SessionConfig.PLASMA_X11 + if s.find('plasma-steamos-wayland-oneshot.desktop') != -1: + return SessionConfig.PLASMA_WAYLAND + else: + # if the conf file doesn't exist we are likely in the default config + # check for a running gamescope for sanity + if subprocess.call('pgrep -a -x gamescope', shell=True, stdout=subprocess.DEVNULL) == 0: + return SessionConfig.GAMESCOPE + # couldn't figure it out, halp + return SessionConfig.ERROR + +def session_select_command(): + if shutil.which('holo-session-select'): + return 'holo-session-select' + return 'steamos-session-select' + +def get_os_info(): + os_info = {} + try: + for k, v in [ s.split('=') for s in open('/etc/os-release').read().split('\n') if len(s) > 0 ]: + os_info[k] = v.strip('"') + except Exception as e: + logger.error(e) + logger.error('Failed to parse OS release file') + return os_info + +def steam_default_args(conf): + if conf.is_deckard: + # Frame currently uses a different setup + return [] + + try: + if os.path.exists('/usr/lib/steamos/steam-launcher'): + output = subprocess.check_output('cat /usr/lib/steamos/steam-launcher | grep ^steamargs=', + shell=True, + universal_newlines=True) + ret = [ v.strip('"') for v in re.findall('\".*?\"', output) ] + return ret + except: + logger.warning('Failed to obtain steam default arguments from /usr/lib/steamos/steam-launcher') + + # Legacy SteamOS + try: + output = subprocess.check_output('cat /usr/bin/gamescope-session | grep ^steamargs', + shell=True, + universal_newlines=True) + ret = [ v.strip('"') for v in re.findall('\".*?\"', output) ] + except: + logger.warning('Failed to obtain steam default arguments from /usr/bin/gamescope-session') + + # Hardcoded fallback + return ['-steamos3', '-steampal', '-steamdeck', '-gamepadui'] + +def frame_osclient_extra_args(conf, steam_status): + if not conf.is_deckard or steam_status != SteamStatus.OS: + return None + if os.path.exists(STEAM_EXTRA_ARGS_FILE): + try: + content = open(STEAM_EXTRA_ARGS_FILE, 'rt').read() + match = re.search(r'Environment="STEAM_EXTRA_ARGS=(.*)"', content) + if match: + return match.group(1).replace('\\"', '"') + except Exception as e: + logger.warning(f'Failed to parse steam extra args: {e}') + return None + +def user_password_is_set(): + ret = subprocess.run('passwd', stdin=subprocess.DEVNULL, shell=True, capture_output=True, universal_newlines=True) + logger.debug(repr(ret)) + return (ret.stderr.find('Current password:') != -1) + +def steam_launch_flags(): + '''Pull various steam flags that affect title execution.''' + ret = {} + if not 'XDG_RUNTIME_DIR' in os.environ: + logger.warning('XDK_RUNTIME_DIR is not set') + return ret + env_folder = os.path.join(os.environ['XDG_RUNTIME_DIR'], 'steam/env') + if not os.path.isdir(env_folder): + return ret + for fn in os.listdir(env_folder): + filepath = os.path.join(env_folder, fn) + content = open(filepath, 'rt').read() + # Check if this is a declaration file with key=value pairs + if content.count('\n') > 1 or '=' in content: + # Parse key=value format with comments + for line in content.splitlines(): + line = line.strip() + # Skip comments and empty lines + if not line or line.startswith('#'): + continue + # Parse key=value pairs + if '=' in line: + key, value = line.split('=', 1) + ret[key.strip()] = value.strip() + else: + # Legacy format: filename is the key, file content is the value + ret[fn] = content.strip('\n') + return ret + +def renderdoc_replay_server_running(): + ret = subprocess.run(['pgrep', '-x', 'renderdoccmd'], capture_output=True) + return ret.returncode == 0 + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--json', required=False, action='store_true') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + os_info = get_os_info() + assert os_info is not None + conf.is_deckard = os_info.get('VARIANT_ID', None) == 'vr' + os_name = os_info.get('PRETTY_NAME', None) + os_version = os_info.get('BUILD_ID', None) + + _steam_launch_flags = steam_launch_flags() + + if not conf.is_deckard: + # this bit of cargo cult is Steam Deck only + try: + # disable wireless power management for devkit usage: less latency on commands + subprocess.check_call(WIRELESS_DISABLE_POWER_MANAGEMENT) + except subprocess.CalledProcessError as e: + logger.warning(e) + + session_config = session_config() + # enum -> human readable + session_status = SESSION_NAMES[session_config] if session_config != SessionConfig.ERROR else 'error' + + steam_status = steam_status() + cef_debugging_enabled = False + if steam_status != SteamStatus.NOT_RUNNING: + cef_debugging_enabled = cef_debugging() + steam_configuration = steam_configuration() + osclient_branch = osclient_branch(conf.is_deckard) + osclient_version = osclient_version(conf) + + steam_status_description = steam_status.description + if steam_status in (SteamStatus.OS, SteamStatus.OS_DEV) : + steam_status_description += f', on branch {osclient_branch!r}' + if osclient_version is not None: + if conf.is_deckard: + # we get builddate.txt + steam_status_description += f', {osclient_version}' + else: + utc_date_string = datetime.datetime.fromtimestamp(osclient_version, datetime.UTC).isoformat() + steam_status_description += f', version {osclient_version} {utc_date_string}' + + has_side_loaded_client = os.path.exists( + os.path.join( + DEVKIT_TOOL_FOLDER, + 'steam' + ) + ) + + _user_password_is_set = user_password_is_set() + + _renderdoc_replay_server_running = renderdoc_replay_server_running() + _renderdoc_layer_enabled = _steam_launch_flags.get('ENABLE_VULKAN_RENDERDOC_CAPTURE', '0') == '1' + + _hostname = socket.gethostname() + + if conf.json: + ret = { + 'is_deckard': conf.is_deckard, + 'hostname': _hostname, + 'os_name': os_name, + 'os_version': os_version, + 'os_info': os_info, + 'session_status': session_status, + 'session_options': SESSION_NAMES, + 'session_select': session_select_command(), + 'steam_status': str(steam_status), + 'cef_debugging_enabled': cef_debugging_enabled, + 'steam_status_description': steam_status_description, + 'steam_configuration': str(steam_configuration), + 'steam_osclient_branch': osclient_branch, + 'steam_osclient_version': osclient_version, + 'has_side_loaded_client': has_side_loaded_client, + 'steam_default_args': steam_default_args(conf), + 'steam_current_args': steam_process_get_args(), + 'frame_osclient_extra_args': frame_osclient_extra_args(conf, steam_status), + 'user_password_is_set': _user_password_is_set, + 'steam_launch_flags': _steam_launch_flags, + 'renderdoc_layer_enabled': _renderdoc_layer_enabled, + 'renderdoc_replay_server_running': _renderdoc_replay_server_running, + } + json.dump(ret, sys.stdout, sort_keys=True, indent=4) + else: + logger.info(f'Hostname : {_hostname}') + logger.info(f'OS : {os_name}') + logger.info(f'OS version : {os_version}') + logger.info(f'Session mode is : {session_status}') + logger.info(f'Session select command : {session_select_command()}') + logger.info(f'Steam client status : {steam_status_description}') + logger.info(f'Steam client args : {steam_process_get_args()!r}') + logger.info(f"Steam extra args (Frame) : {frame_osclient_extra_args(conf, steam_status)!r}") + logger.info(f"Steam CEF debug : {'enabled' if cef_debugging_enabled else 'disabled'}") + logger.info(f'Steam client config : {steam_configuration.description}') + logger.info(f'Steam OS client branch : {osclient_branch}') + logger.info(f'Steam OS client version : {osclient_version}') + logger.info(f"Sideloaded client : {'available' if has_side_loaded_client else 'not installed'}") + logger.info(f'OS client arguments : {steam_default_args(conf)!r}') + logger.info(f"User password is set : {'yes' if _user_password_is_set else 'no'}") + logger.info(f"Steam launch flags : {_steam_launch_flags}") + logger.info(f"RenderDoc layer enabled : {'yes' if _renderdoc_layer_enabled else 'no'}") + logger.info(f"RenderDoc replay running : {'yes' if _renderdoc_replay_server_running else 'no'}") diff --git a/frame/devkit-utils/steamos-list-games b/frame/devkit-utils/steamos-list-games new file mode 100644 index 0000000..31cc305 --- /dev/null +++ b/frame/devkit-utils/steamos-list-games @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 + +import os +import logging +import argparse +import getpass +import json +from subprocess import DEVNULL + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + ret = [] + if os.path.isdir(DEVKIT_TOOL_FOLDER): + for filename in os.listdir(DEVKIT_TOOL_FOLDER): + gamefolder = os.path.join(DEVKIT_TOOL_FOLDER, filename) + if os.path.isdir(gamefolder): + ret.append( { + 'gameid': filename, + } ) + + print(json.dumps(ret)) diff --git a/frame/devkit-utils/steamos-prepare-upload b/frame/devkit-utils/steamos-prepare-upload new file mode 100644 index 0000000..8d59874 --- /dev/null +++ b/frame/devkit-utils/steamos-prepare-upload @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 + +import sys +import os +import logging +import argparse +import getpass +import json +import shutil +import subprocess + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--gameid', required=True, action='store') + parser.add_argument('--restart-steam', required=False, default='0', action='store') + parser.add_argument('--use-mask-unmask', required=False, default='0', action='store') + parser.add_argument('--prevent-auto-repair', required=False, default='0', action='store') + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + directory = os.path.join( + os.path.expanduser(DEVKIT_TOOL_FOLDER), + conf.gameid + ) + os.makedirs(directory, exist_ok=True) + + INHIBIT_SENTINEL = os.path.expanduser('~/.config/inhibit-short-session-tracker') + if int(conf.prevent_auto_repair) == 1: + open(INHIBIT_SENTINEL, 'w').close() + logger.info(f'Created sentinel file: {INHIBIT_SENTINEL}') + elif os.path.exists(INHIBIT_SENTINEL): + os.remove(INHIBIT_SENTINEL) + logger.info(f'Removed sentinel file: {INHIBIT_SENTINEL}') + + + ret = { + 'user': getpass.getuser(), + 'directory': directory, + } + print(json.dumps(ret)) diff --git a/frame/devkit-utils/steamos-set-password.sh b/frame/devkit-utils/steamos-set-password.sh new file mode 100644 index 0000000..26b4bc3 --- /dev/null +++ b/frame/devkit-utils/steamos-set-password.sh @@ -0,0 +1,7 @@ +#!/bin/bash +# meant to be executed remotely/interactively for password prompts + +# there's some annoying trash at the top of the remote ssh screen +clear +passwd +sleep 2 diff --git a/frame/devkit-utils/steamos-set-steam-client b/frame/devkit-utils/steamos-set-steam-client new file mode 100644 index 0000000..7d463af --- /dev/null +++ b/frame/devkit-utils/steamos-set-steam-client @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 + +import sys +import os +import logging +import argparse +import enum +import subprocess +import shutil +import pathlib + +logging.basicConfig(format='%(message)s', level=logging.DEBUG) +logger = logging.getLogger(__name__) + +DEVKIT_TOOL_FOLDER = os.path.expanduser('~/devkit-game') +# NOTE: only relevant to Frame + OS client with extra arguments +# sideloaded client on Frame supports full command line edit instead +STEAM_EXTRA_ARGS_FILE = os.path.expanduser('~/.config/systemd/user/steam.service.d/extra_args.conf') + +class SteamStatus(enum.Enum): + # Supported values from steamos-get-status + OS = 2 + OS_DEV = 3 + SIDELOADED = 4 + +STATUS_STRINGS = [ + ( SteamStatus.OS, 'SteamStatus.OS' ), + ( SteamStatus.OS_DEV, 'SteamStatus.OS_DEV' ), + ( SteamStatus.SIDELOADED, 'SteamStatus.SIDELOADED' ), +] + +# gamescope-session passes the execution to this script if it exists rather than start steam itself +DEVKIT_STEAM_TRAMPOLINE = os.path.expanduser('~/devkit-game/devkit-steam') + +# this script executes the sideloaded Steam client (part of the steamos-devkit-service package) +SIDE_LOADED_STEAM_CLIENT = '/usr/share/steamos-devkit/bin/devkit-standalone.py' + +def write_trampoline(text): + with open(DEVKIT_STEAM_TRAMPOLINE, 'w') as devkit_steam: + devkit_steam.write(text) + devkit_steam.flush() + os.chmod(DEVKIT_STEAM_TRAMPOLINE, 0o770) + # trying really hard to avoid leaving a zero sized trampoline if the deck is about to hang on the session restart coming next + subprocess.run(['/usr/bin/sync', DEVKIT_TOOL_FOLDER]) + +def get_os_info(): + os_info = {} + try: + for k, v in [ s.split('=') for s in open('/etc/os-release').read().split('\n') if len(s) > 0 ]: + os_info[k] = v.strip('"') + except Exception as e: + logger.error(e) + logger.error('Failed to parse OS release file') + return os_info + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--verbose', required=False, action='store_true') + parser.add_argument('--client', action='store', required=True, choices=[ v[1] for v in STATUS_STRINGS ]) + parser.add_argument('--args', action='store', required=False, help='steam client command line arguments') + parser.add_argument('--gameid', required=True, action='store') + parser.add_argument('--gdbserver', action='store_true', required=False) + conf = parser.parse_args() + + if conf.verbose: + logger.setLevel(logging.DEBUG) + else: + logger.setLevel(logging.INFO) + + target = [ v for v in STATUS_STRINGS if v[1] == conf.client ][0][0] + logging.info(f'Set steam client on device to {target}') + + if os.path.exists(DEVKIT_STEAM_TRAMPOLINE): + os.unlink(DEVKIT_STEAM_TRAMPOLINE) + + os_info = get_os_info() + assert os_info is not None + is_deckard = os_info.get('VARIANT_ID', None) == 'vr' + + if target == SteamStatus.OS: + if is_deckard: + # conf.args is the extra arguments for the normal Steam 'OS client', update it now + if conf.args is None or conf.args == '': + logger.info('Clearning extra arguments for normal Steam client') + if os.path.exists(STEAM_EXTRA_ARGS_FILE): + os.unlink(STEAM_EXTRA_ARGS_FILE) + subprocess.run(['systemctl', '--user', 'daemon-reload'], check=True) + else: + logger.info(f'Setting extra arguments for normal Steam client: {conf.args}') + os.makedirs(os.path.dirname(STEAM_EXTRA_ARGS_FILE), exist_ok=True) + with open(STEAM_EXTRA_ARGS_FILE, 'wt') as extra_args_file: + escaped_args = conf.args.replace('"', '\\"') + extra_args_file.write(f'[Service]\nEnvironment="STEAM_EXTRA_ARGS={escaped_args}"') + subprocess.run(['systemctl', '--user', 'daemon-reload'], check=True) + + # When disabling a sideloaded client, also delete the ~/.steam symlinks: + # They will be re-created by the OS client when starting, + # this prevents SteamVR trying to use the sideloaded binaries that are still there for the steam API. + # (this may happen because SteamVR starts before Steam starts and has a chance to set those symlinks correctly) + for path in pathlib.Path(os.path.expanduser('~/.steam')).glob('*'): + if path.is_symlink(): + try: + lnk = path.resolve() + if 'devkit-game' in str(lnk): + path.unlink() + print(f'Deleted: {path} -> {lnk}') + except Exception as e: + print(f'Error processing {path}: {e}') + logger.info('Devkit Steam client override is disabled - default OS client execution will resume.') + sys.exit(0) + + os.makedirs(os.path.dirname(DEVKIT_STEAM_TRAMPOLINE), exist_ok=True) + + # OS client + steam_client = '$HOME/.local/share/Steam/steam.sh' + if target == SteamStatus.SIDELOADED: + steam_client = '$HOME/devkit-game/steam/steam.sh' + + if is_deckard: + # RUNSTEAM.sh checks for SIDELOADED_STEAMROOT="${HOME}/devkit-game/steam" + # this is consistent with sideload on Steam Deck, but we use a different name 'steamdeckard' + # will be addressed when reworking the sideload and debug strategy, for now just drop in a symlink + steam_symlink = os.path.expanduser('~/devkit-game/steam') + if os.path.lexists(steam_symlink): + if os.path.islink(steam_symlink): + os.unlink(steam_symlink) + else: + # this happens if an upload in Steam Deck mode was attempted against a Steam Frame for instance + # was an easy mistake to make before recent changes + logger.warning('warning: ~/devkit-game/steam exists but is not a symlink. Removing anyway.') + shutil.rmtree(steam_symlink) + os.symlink( + os.path.expanduser('~/devkit-game/steamdeckard'), + steam_symlink, + ) + + args = '"$@"' + if conf.args is not None: + args = conf.args + + gdbserver = '' + if conf.gdbserver: + logger.info('Configuring for remote debugging via gdbserver') + gdbserver = 'export DEBUGGER="gdbserver 0.0.0.0:2345"' + + write_trampoline('''#!/bin/bash +# Generated by steamos-set-steam-client, do not edit! +# configuration tag (do not delete): {} +{} +mkdir -p $HOME/.steam/steam/logs +exec {} {} +'''.format( + conf.client, + gdbserver, + steam_client, + args +)) diff --git a/tests/test_frame_titles.py b/tests/test_frame_titles.py new file mode 100644 index 0000000..8ca91af --- /dev/null +++ b/tests/test_frame_titles.py @@ -0,0 +1,382 @@ +"""frame_titles without a headset: executable headers, launch targets, zips, runtimes.""" +import json +import os +import shutil +import struct +import sys +import tempfile +import unittest +import zipfile + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'ui')) +import frame_titles # noqa: E402 +from frame_titles import FrameError # noqa: E402 + + +def elf(machine, e_type=3, interp=True, pad=0): + """A 64-bit little-endian ELF header plus one program header (PT_INTERP or PT_LOAD).""" + ident = b'\x7fELF' + bytes([2, 1, 1]) + b'\0' * 9 + header = ident + struct.pack(' . ; alias/alias/escape -> ../.. ; escape/victim would land outside if links were real. + p = self.link_zip([('alias', '.', True), ('alias/alias/escape', '../..', True), ('escape/victim', b'x', False)]) + out = tempfile.mkdtemp(dir=self.dir) + frame_titles.extract_zip(p, out) + self.assertTrue(os.path.isfile(os.path.join(out, 'escape', 'victim'))) # stayed inside + self.assertFalse(os.path.exists(os.path.join(self.dir, 'victim'))) + self.assertFalse(os.path.exists(os.path.join(os.path.dirname(self.dir), 'victim'))) + for root, dirs, files in os.walk(out): + self.assertFalse([n for n in dirs + files if os.path.islink(os.path.join(root, n))]) + + def test_folder_links_are_dropped_and_order_does_not_matter(self): + # b -> a/file listed before a -> dir; and a folder link that would contain itself. + p = self.link_zip([('dir/file', b'data', False), ('b', 'a/file', True), ('a', 'dir', True), + ('dir/sub/loop', '../../a', True)]) + out = tempfile.mkdtemp(dir=self.dir) + frame_titles.extract_zip(p, out) + with open(os.path.join(out, 'b'), 'rb') as f: + self.assertEqual(f.read(), b'data') + self.assertFalse(os.path.lexists(os.path.join(out, 'a'))) + self.assertFalse(os.path.lexists(os.path.join(out, 'dir', 'sub', 'loop'))) + + def test_link_components_resolve_before_parent_steps(self): + # alias -> dirlink/../game.exe, dirlink -> deep/subdir: that's deep/game.exe, not game.exe. + p = self.link_zip([('deep/subdir/x', b'', False), ('deep/game.exe', b'deep one', False), + ('game.exe', b'top one', False), ('dirlink', 'deep/subdir', True), + ('alias', 'dirlink/../game.exe', True)]) + out = tempfile.mkdtemp(dir=self.dir) + frame_titles.extract_zip(p, out) + with open(os.path.join(out, 'alias'), 'rb') as f: + self.assertEqual(f.read(), b'deep one') + + def test_many_links_to_one_file_count_against_the_limit(self): + # The zip (1 KB) and the copies (15 KB) each fit under the limit; together they don't. + members = [('big', b'x' * 1000, False)] + [(f'alias{i}', 'big', True) for i in range(15)] + old = frame_titles.MAX_UNPACKED + frame_titles.MAX_UNPACKED = 15500 + out = tempfile.mkdtemp(dir=self.dir) + try: + with self.assertRaisesRegex(FrameError, 'links would copy'): + frame_titles.extract_zip(self.link_zip(members), out) + finally: + frame_titles.MAX_UNPACKED = old + self.assertEqual(os.listdir(out), ['big']) # refused before copying any link + + def test_oversized_link_is_refused(self): + p = self.link_zip([('big', 'x' * 5000, True)]) + with self.assertRaisesRegex(FrameError, 'oversized link'): + frame_titles.extract_zip(p, tempfile.mkdtemp(dir=self.dir)) + + @unittest.skipIf(os.name == 'nt', 'needs symlinks') + def test_unwrap_never_steps_through_a_link(self): + # A folder whose only entry links elsewhere (a junction on Windows) stays the boundary. + outside, game = os.path.join(self.dir, 'outside'), os.path.join(self.dir, 'Game') + os.makedirs(outside) + os.makedirs(game) + with open(os.path.join(outside, 'Other.exe'), 'wb') as f: + f.write(pe(0x8664)) + os.symlink(outside, os.path.join(game, 'inner')) + with self.assertRaisesRegex(FrameError, 'no Linux or Windows program'): + frame_titles.inspect(game) + + @unittest.skipIf(os.name == 'nt', 'needs symlinks') + def test_folder_with_outside_link_is_staged_without_it(self): + game, secret = os.path.join(self.dir, 'Game'), os.path.join(self.dir, 'secret') + os.makedirs(game) + os.makedirs(secret) + with open(os.path.join(secret, 'key'), 'wb') as f: + f.write(b'private') + with open(os.path.join(game, 'Game.exe'), 'wb') as f: + f.write(pe(0x8664)) + os.symlink(secret, os.path.join(game, 'leak')) + os.symlink(os.path.join(secret, 'key'), os.path.join(game, 'leak-file')) + os.symlink('Game.exe', os.path.join(game, 'Alias.exe')) + plan = frame_titles.inspect(game) + try: + self.assertNotEqual(os.path.realpath(plan['root']), os.path.realpath(game)) + self.assertEqual(sorted(os.listdir(plan['root'])), ['Alias.exe', 'Game.exe']) + self.assertFalse(os.path.islink(os.path.join(plan['root'], 'Alias.exe'))) + finally: + frame_titles.discard(plan) + + def test_links_become_copies(self): + # No symlinks on disk (Windows may not allow them); the library a link names is still there. + p = self.link_zip([('game/lib/libfoo.so.1.2', b'ELF-ish', False), ('game/lib/libfoo.so.1', 'libfoo.so.1.2', True), + ('game/lib/libfoo.so', 'libfoo.so.1', True), ('game/dangling', 'nowhere', True)]) + out = tempfile.mkdtemp(dir=self.dir) + frame_titles.extract_zip(p, out) + for name in ('libfoo.so.1', 'libfoo.so'): + path = os.path.join(out, 'game', 'lib', name) + self.assertFalse(os.path.islink(path)) + with open(path, 'rb') as f: + self.assertEqual(f.read(), b'ELF-ish') + self.assertFalse(os.path.lexists(os.path.join(out, 'game', 'dangling'))) + + def test_drive_qualified_parts_are_refused(self): + for bad in ('sub/C:../C:../victim.txt', 'game/file.exe:stream'): + with self.subTest(bad=bad): + with self.assertRaisesRegex(FrameError, 'drive or stream'): + frame_titles.extract_zip(self.zip({bad: b'x'}, 'drive.zip'), tempfile.mkdtemp(dir=self.dir)) + + def test_absurd_size_is_refused(self): + p = self.zip({'game.exe': pe(0x8664)}, 'bomb.zip') + old = frame_titles.MAX_UNPACKED + frame_titles.MAX_UNPACKED = 10 + try: + with self.assertRaisesRegex(FrameError, 'looks wrong'): + frame_titles.extract_zip(p, tempfile.mkdtemp(dir=self.dir)) + finally: + frame_titles.MAX_UNPACKED = old + + def test_not_a_zip(self): + p = os.path.join(self.dir, 'x.zip') + with open(p, 'wb') as f: + f.write(b'nope') + with self.assertRaisesRegex(FrameError, 'not a readable zip'): + frame_titles.inspect(p) + + +class Names(unittest.TestCase): + def test_title_id(self): + self.assertEqual(frame_titles.title_id('Hollow Knight: Silksong!'), 'Hollow_Knight_Silksong') + self.assertEqual(frame_titles.title_id('steam'), 'steam-game') # Valve's reserved sideload names + self.assertEqual(frame_titles.title_id('--rm -rf /'), 'rm_-rf') + self.assertEqual(len(frame_titles.title_id('x' * 200)), 64) + with self.assertRaises(FrameError): + frame_titles.title_id('!!!') + + def test_display_name(self): + self.assertEqual(frame_titles.display_name('MyGame-linux-arm64.zip'), 'MyGame') + self.assertEqual(frame_titles.display_name('Portal 2.zip'), 'Portal 2') + self.assertEqual(frame_titles.display_name('Game_v1.0.3_Win64.zip'), 'Game') + + +class Parms(unittest.TestCase): + def test_proton_parms(self): + p = frame_titles.shortcut_parms('Cool_Game', '/home/steamos/devkit-game/Cool_Game', + 'Cool Game.exe', 'proton-experimental') + self.assertEqual(p, {'gameid': 'Cool_Game', 'directory': '/home/steamos/devkit-game/Cool_Game', + 'argv': ['"Cool Game.exe"'], 'env': {}, + 'settings': {'steam_play': '1', 'steam_play_debug': '0', + 'steam_play_debug_version': '2019', + 'compat_tool': 'proton-experimental'}, + 'clear_settings': True, 'force_appid': '', 'lepton_args': ''}) + json.dumps(p) + + def test_linux_parms(self): + p = frame_titles.shortcut_parms('g', '/home/steamos/devkit-game/g', 'bin/game', 'SteamLinuxRuntime_4-arm64') + self.assertEqual(p['argv'], ['bin/game']) + self.assertEqual(p['settings'], {'steam_play': '0', 'compat_tool': 'SteamLinuxRuntime_4-arm64'}) + + def test_cleanup_names_only_this_title(self): + # A glob like Game-*.json would also delete Game-Deluxe's files. + self.assertEqual(frame_titles._json_files('Game').split(), + ['devkit-game/Game-argv.json', 'devkit-game/Game-env.json', + 'devkit-game/Game-settings.json', 'devkit-game/Game-framecontrol.json']) + + def test_launch_needs_steam_to_answer(self): + # steam-devkit-rpc exits 0 after a timeout; only its 'success' line means Steam took it. + calls = [] + old = frame_titles.ssh, frame_titles._check_id, frame_titles.ensure_utils + frame_titles._check_id, frame_titles.ensure_utils = (lambda g: g), (lambda: False) + try: + frame_titles.ssh = lambda cmd, **kw: calls.append(cmd) or 'Found steam client pid 1\ntimeout\n' + with self.assertRaisesRegex(FrameError, "didn't confirm"): + frame_titles.launch('Game') + frame_titles.ssh = lambda cmd, **kw: 'Found steam client pid 1\nsuccess\n{}' + self.assertEqual(frame_titles.launch('Game'), {'id': 'Game'}) + finally: + frame_titles.ssh, frame_titles._check_id, frame_titles.ensure_utils = old + self.assertIn('steam-devkit-rpc run-game gameid=Game', calls[0]) + + def test_remove_waits_for_installs(self): + with frame_titles._install_lock: + with self.assertRaisesRegex(FrameError, 'install is running'): + frame_titles.remove('Game') + + def test_vendored_utils_are_present(self): + for name in ('steamos-prepare-upload', 'steam-client-create-shortcut', 'steam-devkit-rpc', + 'steamos-delete', 'devkit_utils/__init__.py', 'LICENSE'): + self.assertTrue(os.path.isfile(os.path.join(frame_titles.UTILS_LOCAL, name)), name) + self.assertEqual(len(frame_titles.utils_stamp()), 20) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_server.py b/tests/test_server.py index 5511184..a75c0b0 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -6,14 +6,17 @@ request guards and input validation, which all run before any SSH call. Run: python3 -m unittest discover -s tests """ import http.client +import io import json import os import socket +import struct import subprocess import sys import tempfile import time import unittest +import zipfile from pathlib import Path from urllib.parse import quote @@ -54,7 +57,7 @@ class ServerGuards(unittest.TestCase): @classmethod def request(cls, method, path, body=None, headers=None): conn = http.client.HTTPConnection("127.0.0.1", cls.port, timeout=10) - data = json.dumps(body).encode() if body is not None else None + data = body if isinstance(body, bytes) else json.dumps(body).encode() if body is not None else None conn.request(method, path, body=data, headers=headers or {}) r = conn.getresponse() payload = r.read() @@ -130,6 +133,37 @@ class ServerGuards(unittest.TestCase): status, _ = self.post("/api/launch", ["not", "an", "object"]) self.assertEqual(status, 400) + def test_title_upload_is_inspected_then_discarded(self): + # A zip holding a Windows x86-64 program: inspected locally, no SSH until install. + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as z: + z.writestr("Tiny Game/Tiny Game.exe", + b"MZ" + b"\0" * 0x3A + struct.pack(" {payload}") + self.assertEqual(self.request("GET", "/api/titles/job?token=nope", headers={"X-Frame-UI": "1"})[0], 404) + self.assertEqual(self.request("POST", "/api/titles", {"action": "list"})[0], 403) + def test_unknown_routes(self): self.assertEqual(self.request("GET", "/nope")[0], 404) self.assertEqual(self.post("/api/nope", {})[0], 404) diff --git a/ui/frame_titles.py b/ui/frame_titles.py new file mode 100644 index 0000000..0ab0192 --- /dev/null +++ b/ui/frame_titles.py @@ -0,0 +1,797 @@ +"""Linux and Windows builds on the Frame as Steam "Devkit Games". + +A .zip, a folder or a single executable becomes a title in the Steam library, +through the same path as Valve's SteamOS Devkit Client: its devkit-utils +(vendored in frame/devkit-utils, synced to ~/devkit-utils on the Frame) make +~/devkit-game//, the files are copied there, and steam-client-create-shortcut +asks the running Steam client to register it with a runtime: + + Windows .exe -> Proton Experimental (steam_play=1; x86-64 runs through FEX) + aarch64 ELF -> Steam Linux Runtime 4.0 ARM64 (steam_play=0) + x86-64 ELF -> Steam Linux Runtime 4.0 (steam_play=0; runs through FEX) + +Everything device-side is inferred from Valve's steamos-devkit source until +checked on a headset; see docs/sideloading.md. + +Python stdlib only. CLI: + python3 ui/frame_titles.py inspect PATH + python3 ui/frame_titles.py install PATH [--name N] [--exe REL] [--runtime R] + python3 ui/frame_titles.py list | launch ID | remove ID +""" +import hashlib, json, os, posixpath, re, shlex, shutil, stat, struct, subprocess, sys, tempfile, threading, time, zipfile + +import frame_android +import frame_host +from frame_android import FrameError + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +UTILS_LOCAL = os.path.join(ROOT, 'frame', 'devkit-utils') +UTILS = 'devkit-utils' # on the Frame, relative to $HOME (where Valve's client puts it) +GAMES = 'devkit-game' # ditto; steamos-prepare-upload makes / in here +STAMP = '.frame-control-stamp' +PY = 'python3 ~/' + UTILS + '/' + +# Valve's reserved sideload names: uploading one of these replaces the Steam client itself. +RESERVED_IDS = ('steam', 'steamdeckard', 'steamvr', 'steamvrdeckard') +ID_RE = re.compile(r'^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$') +DIR_RE = re.compile(r'^/[A-Za-z0-9_./-]+$') + +# Zip limits: well above any real game, well below a zip bomb. +MAX_UNPACKED = 64 * 1024**3 +MAX_ENTRIES = 200000 +MAX_RATIO = 200 # uncompressed / compressed, once past 1 GB + +# The Steam compat tool aliases Valve's client uses (devkit_client RUNTIME_ALIASES). +RUNTIMES = { + 'proton-experimental': {'label': 'Proton Experimental', 'steam_play': True}, + 'proton-stable': {'label': 'Proton (stable)', 'steam_play': True}, + 'SteamLinuxRuntime_4-arm64': {'label': 'Steam Linux Runtime 4.0 (ARM64)', 'steam_play': False}, + 'SteamLinuxRuntime_4': {'label': 'Steam Linux Runtime 4.0 (x86-64, through FEX)', 'steam_play': False}, +} +# Experimental rather than stable: the Frame's ARM64 Proton + FEX stack is new, +# and Proton fixes reach Experimental first. --runtime proton-stable switches. +DEFAULT_PROTON = 'proton-experimental' + +ELF_MACHINES = {0xB7: 'arm64', 0x3E: 'x86_64', 0x03: 'x86', 0x28: 'arm'} +PE_MACHINES = {0x8664: 'x86_64', 0xAA64: 'arm64', 0x14C: 'x86', 0x1C4: 'arm'} +# Executables that are never the game: crash reporters, installers, redistributables. +SKIP_RE = re.compile(r'crash|unins|setup|install|redist|dxsetup|dxwebsetup|dotnet|prereq|' + r'easyanticheat|eac_|updater|uploader|report|sandbox|helper', re.I) +SKIP_DIRS = re.compile(r'^(_*commonredist|redist|redistributables?|directx|vcredist|__installer|' + r'installers?|prereqs?|support|engine|__macosx)$', re.I) +# Trailing words of an archive name that describe the build, not the game. +BUILD_WORDS = re.compile(r'([ ._-]+(win(dows)?(32|64)?|linux(32|64)?|x64|x86(_64)?|amd64|arm64|aarch64|' + r'build|release|portable|steamos|v\d+([._]\d+)*|\d+([._]\d+)+))+$', re.I) + + +def classify(path): + """{'format': 'elf'|'pe'|'script', 'arch', 'exe'} for an executable file, else None.""" + try: + with open(path, 'rb') as f: + head = f.read(4096) + if head[:4] == b'\x7fELF': + return _elf(f, head) + if head[:2] == b'MZ': + return _pe(f, head) + except (OSError, struct.error, ValueError, OverflowError): + return None # unreadable, or a header that lies about its sizes + if head[:2] == b'#!' or path.lower().endswith('.sh'): + return {'format': 'script', 'arch': None, 'exe': True} + return None + + +def _elf(f, head): + if len(head) < 64 or head[4] not in (1, 2) or head[5] not in (1, 2): + return None + wide, end = head[4] == 2, '<' if head[5] == 1 else '>' + e_type, machine = struct.unpack_from(end + 'HH', head, 16) + arch = ELF_MACHINES.get(machine, f'elf-0x{machine:x}') + if e_type == 2: # ET_EXEC + return {'format': 'elf', 'arch': arch, 'exe': True} + if e_type != 3: # not ET_DYN either: object file, core dump + return {'format': 'elf', 'arch': arch, 'exe': False} + # ET_DYN is a PIE executable or a shared library; only executables ask for an interpreter. + if wide: + phoff, = struct.unpack_from(end + 'Q', head, 32) + phentsize, phnum = struct.unpack_from(end + 'HH', head, 54) + else: + phoff, = struct.unpack_from(end + 'I', head, 28) + phentsize, phnum = struct.unpack_from(end + 'HH', head, 42) + if phentsize < (56 if wide else 32) or phnum > 256: + return {'format': 'elf', 'arch': arch, 'exe': False} + f.seek(phoff) + table = f.read(phentsize * phnum) + interp = any(struct.unpack_from(end + 'I', table, i * phentsize)[0] == 3 # PT_INTERP + for i in range(len(table) // phentsize)) + return {'format': 'elf', 'arch': arch, 'exe': interp} + + +def _pe(f, head): + if len(head) < 0x40: + return None + lfanew, = struct.unpack_from(' a local tree to upload ---------- + +def extract_zip(zpath, dest): + """Unpack a zip into dest, refusing paths that escape it and absurd sizes.""" + try: + z = zipfile.ZipFile(zpath) + except (zipfile.BadZipFile, OSError) as e: + raise FrameError(f'{os.path.basename(zpath)} is not a readable zip: {e}') + with z: + infos = z.infolist() + if len(infos) > MAX_ENTRIES: + raise FrameError(f'the zip has {len(infos)} entries; the limit is {MAX_ENTRIES}') + total = sum(i.file_size for i in infos) + packed = max(1, os.path.getsize(zpath)) + if total > MAX_UNPACKED or (total > 1024**3 and total > packed * MAX_RATIO): + raise FrameError(f'the zip would unpack to {total / 1024**3:.1f} GB, which looks wrong') + free = shutil.disk_usage(dest).free + if total + 256 * 1024**2 > free: + raise FrameError(f'not enough space on this computer to unpack the zip ' + f'({total / 1024**3:.1f} GB needed, {free / 1024**3:.1f} GB free)') + try: + _extract_members(z, infos, os.path.realpath(dest)) + except FrameError: + raise # a RuntimeError too, but already worded + except (zipfile.BadZipFile, RuntimeError, NotImplementedError, EOFError, OSError) as e: + # Encrypted or corrupt members, unsupported compression, clashing names, a full disk. + raise FrameError(f'could not unpack {os.path.basename(zpath)}: {e}') + + +def _extract_members(z, infos, root): + # No symlink is ever created here, so no write can be redirected through one + # (chained links, Windows without the privilege). Links inside the zip are + # resolved on paper and materialised as copies once the real files are out. + links = {} + for info in infos: + rel = _safe_member(info.filename) + if rel is None: + continue + target = _inside(root, rel, info.filename) + mode = info.external_attr >> 16 + if info.is_dir(): + os.makedirs(target, exist_ok=True) + continue + os.makedirs(os.path.dirname(target), exist_ok=True) + if stat.S_ISLNK(mode): + if info.file_size > 4096: # a link's content is a path, never this big + raise FrameError(f'the zip has an oversized link: {info.filename}') + link = z.read(info).decode('utf-8', 'replace').replace('\\', '/') + dest = posixpath.normpath(posixpath.join(posixpath.dirname(rel), link)) + if link.startswith('/') or ':' in link or dest == '..' or dest.startswith('../'): + raise FrameError(f'the zip has a link that points outside it: {info.filename}') + links[rel] = link # as written: resolved later, one component at a time + continue + with z.open(info) as src, open(target, 'wb') as out: + shutil.copyfileobj(src, out, 1 << 20) + if mode & 0o111: + os.chmod(target, 0o755) + _materialise_links(root, links) + + +def _inside(root, rel, name): + """rel's path under root, refusing anything that resolves outside it.""" + target = os.path.join(root, *rel.split('/')) + if not (os.path.realpath(target) + os.sep).startswith(root + os.sep): + raise FrameError(f'the zip has a path that climbs out of it: {name}') + return target + + +def _resolve_link(path, links): + """path with every link in it followed, on paper; None if it loops or leaves the zip. + + Like the kernel: each component in turn, so 'dirlink/..' is the parent of + where dirlink points, not the folder dirlink sits in. + """ + todo, done, hops = path.split('/'), [], 0 + while todo: + part = todo.pop(0) + if part in ('', '.'): + continue + if part == '..': + if not done: + return None + done.pop() + continue + done.append(part) + text = links.get('/'.join(done)) + if text is not None: + hops += 1 + if hops > 40: + return None + done.pop() # link text is relative to the link's folder + todo = text.split('/') + todo + return '/'.join(done) + + +def _materialise_links(root, links): + """Copy the file each link names into its place (lib.so.1 -> lib.so.1.2.3 and the like). + + Only links to files: a folder link could hold itself, and game builds link + libraries, not folders. Folder, looping and dangling links are dropped. + """ + copies = [] + for rel in sorted(links): + dest = _resolve_link(rel, links) + if not dest: + continue # loops, escapes, or the zip's own top folder + src, target = _inside(root, dest, rel), _inside(root, rel, rel) + if os.path.isfile(src) and not os.path.lexists(target): # a real entry may have the name + copies.append((src, target)) + # Many links to one big file could fill the disk: the same limits as the zip itself. + need = sum(os.path.getsize(src) for src, _ in copies) + if need + _tree_size(root) > MAX_UNPACKED: + raise FrameError('the zip\'s links would copy more than it holds, which looks wrong') + if need + 256 * 1024**2 > shutil.disk_usage(root).free: + raise FrameError(f'not enough space on this computer for the zip\'s linked files ({need / 1024**3:.1f} GB)') + for src, target in copies: + os.makedirs(os.path.dirname(target), exist_ok=True) + shutil.copy2(src, target) + + +def _safe_member(name): + """The member's relative path with / separators, None to skip it; raises if it escapes.""" + rel = name.replace('\\', '/') + if rel.startswith('/') or re.match(r'^[A-Za-z]:', rel): + raise FrameError(f'the zip has an absolute path: {name}') + parts = [p for p in rel.split('/') if p not in ('', '.')] + if any(p == '..' for p in parts): + raise FrameError(f'the zip has a path that climbs out of it: {name}') + if any(':' in p for p in parts): + # Drive-qualified parts ('C:..') climb out on Windows; ':' is an NTFS stream elsewhere. + raise FrameError(f'the zip has a path with a drive or stream name: {name}') + if not parts or parts[0] == '__MACOSX' or parts[-1] in ('.DS_Store', 'Thumbs.db'): + return None + return '/'.join(parts) + + +def _redirected(path, expected): + """True if path is a symlink, or resolves somewhere else (a Windows junction isn't islink).""" + return os.path.islink(path) or os.path.normcase(os.path.realpath(path)) != os.path.normcase(expected) + + +def _has_links(root): + real = os.path.realpath(root) + for dirpath, dirnames, filenames in os.walk(real): + for n in dirnames + filenames: + if _redirected(os.path.join(dirpath, n), os.path.join(dirpath, n)): + return True + return False + + +def _stage_folder(src, dest): + """Copy src to dest; links to files inside src become copies, all other links are left out.""" + real = os.path.realpath(src) + inside = lambda p: os.path.normcase(p).startswith(os.path.normcase(real) + os.sep) # noqa: E731 + folders, copies = [], [] # decide everything first, so the space check sees the same files + for dirpath, dirnames, filenames in os.walk(real): + out = os.path.join(dest, os.path.relpath(dirpath, real)) + folders.append(out) + # Only descend into real folders: not symlinked ones, not junctions (os.walk follows those). + linked = [d for d in dirnames if _redirected(os.path.join(dirpath, d), os.path.join(dirpath, d))] + dirnames[:] = [d for d in dirnames if d not in linked] + for fn in filenames + linked: + target = os.path.realpath(os.path.join(dirpath, fn)) + if inside(target) and os.path.isfile(target): + copies.append((target, os.path.join(out, fn))) + if shutil.disk_usage(os.path.dirname(dest)).free < sum(os.path.getsize(t) for t, _ in copies) + 256 * 1024**2: + raise FrameError('not enough space on this computer to stage the folder') + for out in folders: + os.makedirs(out, exist_ok=True) + for target, out in copies: + shutil.copy2(target, out) + return dest + + +def _unwrap(root): + """Step into a single top-level folder, the usual shape of a zipped build. + + Never through a link or junction: the folder you chose (or unpacked) stays the boundary. + """ + root = os.path.realpath(root) + for _ in range(4): + entries = [e for e in os.listdir(root) if e not in ('__MACOSX', '.DS_Store', 'Thumbs.db')] + if len(entries) != 1: + break + only = os.path.join(root, entries[0]) + if not os.path.isdir(only) or _redirected(only, only): + break + root = only + return root + + +def candidates(root, title=''): + """Executables under root, best launch target first.""" + found = [] + want = _norm(title) + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = sorted(d for d in dirnames if not os.path.islink(os.path.join(dirpath, d))) + rel_dir = os.path.relpath(dirpath, root) + parts = [] if rel_dir == '.' else rel_dir.split(os.sep) + for fn in sorted(filenames): + full = os.path.join(dirpath, fn) + if os.path.islink(full) or not os.path.isfile(full): + continue + c = classify(full) + if not c or not c['exe']: + continue + stem = _norm(os.path.splitext(fn)[0]) + skip = bool(SKIP_RE.search(fn)) or any(SKIP_DIRS.match(p) for p in parts) + match = 2 if want and stem == want else 1 if want and stem and (want in stem or stem in want) else 0 + found.append({'path': '/'.join(parts + [fn]), 'format': c['format'], 'arch': c['arch'], + 'size': os.path.getsize(full), 'depth': len(parts), 'skip': skip, 'match': match}) + found.sort(key=_rank) + _prefer_launcher_script(found) + return found + + +def _platform_rank(c): + # Native ARM64 first, then Proton, then x86-64 Linux through FEX; scripts are placed separately. + order = {('elf', 'arm64'): 0, ('pe', 'x86_64'): 1, ('elf', 'x86_64'): 2, ('pe', 'x86'): 3, ('pe', 'arm64'): 3} + return order.get((c['format'], c['arch']), 5 if c['format'] == 'script' else 6) + + +def _rank(c): + return (c['skip'], _platform_rank(c), -c['match'], c['depth'], -c['size'], c['path']) + + +def _prefer_launcher_script(found): + """A top-level shell script beats a Linux binary one folder down (run.sh + bin/game).""" + if not found or found[0]['format'] != 'elf' or found[0]['depth'] == 0: + return + for i, c in enumerate(found): + if c['format'] == 'script' and c['depth'] == 0 and not c['skip']: + found.insert(0, found.pop(i)) + return + + +def runtime_for(target, found=()): + """(compat tool alias, note) for a launch target, or raise FrameError if it can't run.""" + fmt, arch = target['format'], target['arch'] + if fmt == 'script': + # A script runs in the runtime of the binaries next to it; alone, natively. + elf = next((c for c in found if c['format'] == 'elf' and not c['skip']), None) + if elf: + return runtime_for(elf)[0], 'A shell script; runtime chosen from the Linux binary next to it.' + return 'SteamLinuxRuntime_4-arm64', 'A shell script with no Linux binary beside it; run natively.' + if fmt == 'pe': + if arch not in ('x86_64', 'x86', 'arm64'): + raise FrameError(f"{target['path']} is a Windows program for {arch}, which Proton can't run") + note = 'Windows x86-64 build: Proton runs it through FEX.' if arch == 'x86_64' else \ + f'Windows {arch} build under Proton.' + return DEFAULT_PROTON, note + if fmt == 'elf' and arch == 'arm64': + return 'SteamLinuxRuntime_4-arm64', 'Native ARM64 Linux build.' + if fmt == 'elf' and arch == 'x86_64': + return 'SteamLinuxRuntime_4', 'x86-64 Linux build: runs through FEX (inferred, not yet checked).' + raise FrameError(f"{target['path']} is a {arch} Linux program; the Frame runs ARM64 and x86-64 (through FEX) only") + + +def allowed_runtimes(target, found=()): + alias, _ = runtime_for(target, found) + return ['proton-experimental', 'proton-stable'] if RUNTIMES[alias]['steam_play'] else [alias] + + +def inspect(path, name=None): + """Read a .zip, folder or executable into an install plan (a JSON-safe dict). + + A zip is unpacked into a temporary folder, plan['work']; pass the plan to + discard() when done with it. Raises FrameError if nothing in it can run. + """ + path = os.path.abspath(path) + if not os.path.exists(path): + raise FrameError(f'{path} does not exist') + work = None + try: + if os.path.isdir(path): + root = _unwrap(path) + if _has_links(root): + # scp -r follows links, so a link out of the folder could upload + # anything; copy the folder with its links made safe first. + work = tempfile.mkdtemp(prefix='frame-title-') + root = _stage_folder(root, os.path.join(work, os.path.basename(root))) + elif path.lower().endswith('.zip'): + work = tempfile.mkdtemp(prefix='frame-title-') + extract_zip(path, work) + root = _unwrap(work) + elif classify(path): + # A single executable is uploaded on its own; don't copy a whole Downloads folder. + work = tempfile.mkdtemp(prefix='frame-title-') + shutil.copy2(path, os.path.join(work, os.path.basename(path))) + root = work + else: + raise FrameError(f'{os.path.basename(path)} is not a .zip, a folder or a program') + title = name or display_name(os.path.basename(path.rstrip('/\\'))) + found = candidates(root, title) + if not found: + raise FrameError(f'no Linux or Windows program found in {os.path.basename(path)}') + plan = {'source': os.path.basename(path.rstrip('/\\')), 'name': title, 'id': title_id(title), + 'root': root, 'work': work, 'candidates': found, + 'size': _tree_size(root), 'warnings': []} + _choose(plan, found[0]['path']) + return plan + except BaseException: + if work: + shutil.rmtree(work, ignore_errors=True) + raise + + +def _tree_size(root): + total = 0 + for dirpath, _, filenames in os.walk(root): + for fn in filenames: + full = os.path.join(dirpath, fn) + if os.path.isfile(full) and not os.path.islink(full): + total += os.path.getsize(full) + return total + + +def _choose(plan, rel, runtime=None): + """Set plan's launch target (a path relative to root) and its runtime.""" + target = next((c for c in plan['candidates'] if c['path'] == rel), None) + if target is None: + full = os.path.realpath(os.path.join(plan['root'], *rel.replace('\\', '/').split('/'))) + root = os.path.realpath(plan['root']) + if not (full + os.sep).startswith(root + os.sep) or not os.path.isfile(full): + raise FrameError(f'{rel} is not a file in the title') + c = classify(full) + if not c or not c['exe']: + raise FrameError(f"{rel} isn't a program the Frame can start") + target = {'path': os.path.relpath(full, root).replace(os.sep, '/'), 'format': c['format'], + 'arch': c['arch'], 'size': os.path.getsize(full), 'depth': rel.count('/'), + 'skip': False, 'match': 0} + alias, note = runtime_for(target, plan['candidates']) + allowed = allowed_runtimes(target, plan['candidates']) + if runtime: + if runtime not in allowed: + raise FrameError(f"{target['path']} can't use {runtime}; choose one of {', '.join(allowed)}") + alias = runtime + plan.update(target=target['path'], format=target['format'], arch=target['arch'], runtime=alias, + runtime_label=RUNTIMES[alias]['label'], runtimes=allowed, note=note) + plan['warnings'] = (['This looks like an installer or helper, not the game itself.'] if target['skip'] else []) + return plan + + +def discard(plan): + if plan and plan.get('work'): + shutil.rmtree(plan['work'], ignore_errors=True) + + +def argv_for(rel): + # Valve's client sends the start command as one string (it may carry arguments), + # so a path with spaces is quoted. How Steam splits it is inferred. + return ['"' + rel + '"' if re.search(r'\s', rel) else rel] + + +def shortcut_parms(gameid, directory, rel, runtime): + """The JSON steam-client-create-shortcut takes, as devkit_client.new_or_ensure_game builds it.""" + settings = {'steam_play': '1' if RUNTIMES[runtime]['steam_play'] else '0'} + if RUNTIMES[runtime]['steam_play']: + # gui2._update_game sends these with every Proton title; debugging stays off. + settings.update(steam_play_debug='0', steam_play_debug_version='2019') + settings['compat_tool'] = runtime + return {'gameid': gameid, 'directory': directory, 'argv': argv_for(rel), 'env': {}, + 'settings': settings, 'clear_settings': True, 'force_appid': '', 'lepton_args': ''} + + +# ---- the Frame side ---------------------------------------------------------- + +def ssh(cmd, input=None, timeout=120): + return frame_android.ssh(cmd, input=input, timeout=timeout) + + +def _json_out(out, what): + """The JSON object a devkit-utils script prints last (its logging goes to stderr).""" + for line in reversed(out.strip().splitlines()): + line = line.strip() + if line.startswith('{') or line.startswith('['): + try: + return json.loads(line) + except ValueError: + break + raise FrameError(f'{what} gave no usable answer: {out.strip()[-300:]!r}') + + +def utils_stamp(): + """Hash of the vendored devkit-utils, compared with the copy on the Frame.""" + h = hashlib.sha256() + for dirpath, dirnames, filenames in os.walk(UTILS_LOCAL): + dirnames[:] = sorted(d for d in dirnames if d != '__pycache__') + for fn in sorted(filenames): + if fn.endswith('.pyc'): + continue + full = os.path.join(dirpath, fn) + h.update(os.path.relpath(full, UTILS_LOCAL).replace(os.sep, '/').encode() + b'\0') + with open(full, 'rb') as f: + h.update(f.read()) + return h.hexdigest()[:20] + + +def _json_files(gid): + # Exact names: a glob like Game-*.json would also match another title called Game-Deluxe. + return ' '.join(f'{GAMES}/{gid}-{k}.json' for k in ('argv', 'env', 'settings', 'framecontrol')) + + +_utils_lock = threading.Lock() + + +def ensure_utils(): + """Copy frame/devkit-utils to ~/devkit-utils on the Frame unless it's already this version.""" + with _utils_lock: # one sync at a time: they share a staging folder + return _ensure_utils() + + +def _ensure_utils(): + stamp = utils_stamp() + have = ssh(f'cat {UTILS}/{STAMP} 2>/dev/null || true', timeout=30).strip() + if have == stamp: + return False + # Merge rather than replace: Valve's own client may have put newer files there. + tmp = f'.{UTILS}.frame-control' + ssh(f'rm -rf {tmp}', timeout=30) + _copy_tree(UTILS_LOCAL, tmp, timeout=300) + ssh(f'mkdir -p {UTILS} && cp -R {tmp}/. {UTILS}/ && rm -rf {tmp} {UTILS}/__pycache__ ' + f'&& echo {stamp} > {UTILS}/{STAMP}', timeout=60) + return True + + +def _copy_tree(src, dest, timeout=3 * 3600, delete=False): + """Copy a local folder's contents to dest on the Frame (dest ends up a copy of src). + + rsync where installed (not on Windows; see server.push_file), else scp -r + into a fresh dest, which is what Windows has. dest must be a plain path. + """ + name = os.path.basename(src.rstrip('/\\')) + opts = frame_android.SSH_OPTS # read now: the server swaps in its multiplexed options + if _rsync(): + cmd = ['rsync', '-a', *(['--delete'] if delete else []), '-e', shlex.join(['ssh', *opts]), + src.rstrip('/') + '/', f'{frame_android.FRAME}:{dest.rstrip("/")}/'] + else: + # scp -r copies src *into* dest when dest exists, so dest must not. + ssh(f'rm -rf {shlex.quote(dest)}', timeout=60) + cmd = ['scp', *opts, '-r', src, f'{frame_android.FRAME}:{dest}'] + try: + subprocess.run(cmd, check=True, capture_output=True, stdin=subprocess.DEVNULL, text=True, + errors='replace', timeout=timeout) + except subprocess.TimeoutExpired: + raise FrameError(f'copying {name} to the Frame timed out') + except subprocess.CalledProcessError as e: + raise FrameError(f'copying {name} to the Frame failed: {(e.stderr or "").strip()[-300:]}') + + +def _rsync(): + # Not on Windows: a Windows rsync (cwRsync, MSYS2) wouldn't take the POSIX -e quoting. + return not frame_host.WINDOWS and bool(shutil.which('rsync')) + + +_install_lock = threading.Lock() + + +def install(path, name=None, exe=None, runtime=None, progress=None): + """Sideload a .zip, folder or executable as a Devkit Game; returns the title dict. + + name: the Steam name (sanitised to the title id), default from the file name. + exe: launch target relative to the title's root, default the best candidate. + runtime: a compat tool alias from RUNTIMES that suits the target (e.g. + 'proton-stable' instead of the default Proton Experimental). + progress: optional callable(stage_text, fraction 0..1). + """ + plan = inspect(path, name) + try: + return install_plan(plan, name=name, exe=exe, runtime=runtime, progress=progress) + finally: + discard(plan) + + +def install_plan(plan, name=None, exe=None, runtime=None, progress=None): + """Install an inspect() plan, optionally with another name, target or runtime.""" + if name: + plan['name'], plan['id'] = name, title_id(name) + if exe or runtime: + _choose(plan, exe or plan['target'], runtime) + step = progress or (lambda *a: None) + with _install_lock: + return _install(plan, step) + + +def _install(plan, step): + gid = plan['id'] + if not ID_RE.match(gid) or gid.lower() in RESERVED_IDS: + raise FrameError(f'bad title id {gid!r}') + step("Syncing Valve's devkit tools to the Frame", 0.02) + ensure_utils() + existed = ssh(f'test -d {GAMES}/{gid} && echo yes || true', timeout=30).strip() == 'yes' + step('Preparing the title folder', 0.05) + ready = _json_out(ssh(f'{PY}steamos-prepare-upload --gameid {gid}', timeout=60), 'steamos-prepare-upload') + directory = str(ready.get('directory') or '') + if not DIR_RE.match(directory) or not directory.endswith(f'/{GAMES}/{gid}'): + raise FrameError(f'steamos-prepare-upload returned an unexpected folder {directory!r}') + registered = False + try: + step(f"Copying {plan['size'] / 1e6:.0f} MB to the Frame", 0.1) + if _rsync(): + _copy_tree(plan['root'], directory, delete=True) + else: + part = f"{directory.rsplit('/', 1)[0]}/.{gid}.upload" + _copy_tree(plan['root'], part) + ssh(f'rm -rf {directory} && mv {part} {directory}', timeout=120) + # Same modes Valve's client gives an upload (rsync --chmod=Du=rwx,Dgo=rx,Fu=rwx,Fog=rx). + ssh(f'chmod -R 755 {directory}', timeout=300) + step('Registering with Steam', 0.9) + parms = shortcut_parms(gid, directory, plan['target'], plan['runtime']) + reply = _json_out(ssh(f'{PY}steam-client-create-shortcut --parms {shlex.quote(json.dumps(parms))}', + timeout=90), 'steam-client-create-shortcut') + meta = {'id': gid, 'name': plan['name'], 'target': plan['target'], 'runtime': plan['runtime'], + 'source': plan['source'], 'size': plan['size'], 'installed': time.strftime('%Y-%m-%dT%H:%M:%S')} + ssh(f'cat > {GAMES}/{gid}-framecontrol.json', input=json.dumps(meta, indent=1), timeout=30) + registered = True # the files stay: Steam registers them once it's running + if 'error' in reply: + raise FrameError(f"Uploaded, but Steam didn't register it: {reply['error']}. " + "With Steam running on the Frame, install it again.") + step('Done', 1.0) + meta.update(runtime_label=RUNTIMES[plan['runtime']]['label'], steam=str(reply.get('success', '')).strip()) + return meta + finally: + if not registered and not existed: + # A first install that failed part-way: don't leave an orphan folder behind. + try: + ssh(f'rm -rf {GAMES}/{gid} {GAMES}/.{gid}.upload {_json_files(gid)}', timeout=60) + except FrameError: + pass + + +LIST_SCRIPT = r''' +import json, os +root = os.path.expanduser('~/devkit-game') +reserved = %r +out = [] +for d in sorted(os.listdir(root)) if os.path.isdir(root) else []: + if d.startswith('.') or d.lower() in reserved or not os.path.isdir(os.path.join(root, d)): + continue + t = {'id': d} + for key, suffix in (('settings', '-settings.json'), ('argv', '-argv.json'), ('meta', '-framecontrol.json')): + try: + with open(os.path.join(root, d + suffix)) as f: + t[key] = json.load(f) + except (OSError, ValueError): + t[key] = None + out.append(t) +print(json.dumps(out)) +''' % (RESERVED_IDS,) + + +def list_titles(): + """The Devkit Games on the Frame (any uploaded by Valve's client too).""" + raw = _json_out(ssh('python3 -', input=LIST_SCRIPT, timeout=30), 'the title list') + titles = [] + for t in raw if isinstance(raw, list) else []: + if not ID_RE.match(str(t.get('id', ''))): + continue + settings, meta = t.get('settings') or {}, t.get('meta') or {} + argv = t.get('argv') if isinstance(t.get('argv'), list) else [] + alias = str(settings.get('compat_tool') or '') + titles.append({'id': t['id'], 'name': str(meta.get('name') or t['id']), + 'target': str(meta.get('target') or (argv[0] if argv else '')), + 'runtime': alias, 'runtime_label': RUNTIMES.get(alias, {}).get('label', alias or 'not set'), + 'source': str(meta.get('source') or ''), 'size': meta.get('size'), + 'installed': meta.get('installed'), 'registered': t.get('settings') is not None, + 'frame_control': bool(meta)}) + return titles + + +def _check_id(gid): + gid = str(gid or '') + if not ID_RE.match(gid) or gid.lower() in RESERVED_IDS: + raise FrameError(f'bad title id {gid!r}') + if ssh(f'test -d {GAMES}/{gid} && echo yes || true', timeout=30).strip() != 'yes': + raise FrameError(f'{gid} is not installed') + return gid + + +def launch(gid): + gid = _check_id(gid) + ensure_utils() + # steam-devkit-rpc logs 'success' when Steam answers, but exits 0 after a + # 5 s timeout too, so read its log (stderr) rather than trust the exit code. + out = ssh(f'{PY}steam-devkit-rpc run-game gameid={gid} 2>&1', timeout=60) + if 'success' not in [line.strip() for line in out.splitlines()]: + raise FrameError(f"Steam didn't confirm the launch: {out.strip()[-300:] or 'no answer'}") + return {'id': gid} + + +def remove(gid): + # Not while an install runs: it could be this title, half copied or about to register. + if not _install_lock.acquire(blocking=False): + raise FrameError('an install is running; remove the title when it has finished') + try: + gid = _check_id(gid) + ensure_utils() + # steamos-delete removes the folder and syncs Steam's shortcuts; its json files stay, so clear them too. + ssh(f'{PY}steamos-delete --delete-title {gid}', timeout=120) + ssh(f'rm -f {_json_files(gid)}', timeout=30) + return {'id': gid} + finally: + _install_lock.release() + + +def public(plan): + """A plan without its local paths, for the UI, with the runtimes each candidate may use.""" + out = {k: v for k, v in plan.items() if k not in ('root', 'work', 'candidates')} + out['candidates'] = [] + for c in plan['candidates']: + c = dict(c) + try: + c['runtimes'] = allowed_runtimes(c, plan['candidates']) + except FrameError as e: + c['runtimes'], c['blocked'] = [], str(e) + out['candidates'].append(c) + out['runtime_labels'] = {k: v['label'] for k, v in RUNTIMES.items()} + return out + + +def main(): + cmd, *args = sys.argv[1:] or ['help'] + + def opt(flag): + return args[args.index(flag) + 1] if flag in args and args.index(flag) + 1 < len(args) else None + + try: + if cmd == 'inspect' and args: + plan = inspect(args[0], opt('--name')) + try: + if opt('--exe') or opt('--runtime'): + _choose(plan, opt('--exe') or plan['target'], opt('--runtime')) + r = public(plan) + finally: + discard(plan) + elif cmd == 'install' and args: + r = install(args[0], name=opt('--name'), exe=opt('--exe'), runtime=opt('--runtime'), + progress=lambda text, _: print(text + '…', file=sys.stderr)) + elif cmd == 'list': + r = list_titles() + elif cmd in ('launch', 'remove') and args: + r = globals()[cmd](args[0]) + else: + sys.exit(__doc__) + except FrameError as e: + sys.exit(f'error: {e}') + print(json.dumps(r, indent=1)) + + +if __name__ == '__main__': + main() diff --git a/ui/index.html b/ui/index.html index 320088c..b2fb580 100644 --- a/ui/index.html +++ b/ui/index.html @@ -221,12 +221,18 @@ .and-grid { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 2fr); gap: 22px; align-items: start; } .and-col { display: grid; gap: 22px; align-content: start; } .rep-item .s { white-space: normal; } - #repDlg { background: #1e2329; color: var(--text); border: 1px solid rgba(255,255,255,.1); border-radius: 4px; + #repDlg, #titleDlg { background: #1e2329; color: var(--text); border: 1px solid rgba(255,255,255,.1); border-radius: 4px; padding: 22px; width: min(560px, 92vw); box-shadow: 0 20px 60px rgba(0,0,0,.6); } - #repDlg::backdrop { background: rgba(0,0,0,.55); } - #repDlg h2 { margin: 0 0 14px; font-size: 15px; letter-spacing: 1.5px; text-transform: uppercase; color: var(--bright); } - #repForm label { display: block; font-size: 12.5px; color: var(--muted); margin-top: 10px; } - #repForm label input[type=text], #repForm textarea { margin-top: 5px; } + #repDlg::backdrop, #titleDlg::backdrop { background: rgba(0,0,0,.55); } + #repDlg h2, #titleDlg h2 { margin: 0 0 14px; font-size: 15px; letter-spacing: 1.5px; text-transform: uppercase; color: var(--bright); } + #repForm label, #titleForm label { display: block; font-size: 12.5px; color: var(--muted); margin-top: 10px; } + #repForm label input[type=text], #repForm textarea, #titleForm label input, #titleForm label select { margin-top: 5px; } + #titleForm select { width: 100%; background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; + border-radius: 3px; padding: 8px 10px; font: inherit; } + #titleForm select:focus { outline: none; border-color: var(--blue); } + #titleForm .note { font-size: 12.5px; color: var(--muted); margin-top: 10px; } + #titleForm .note.warn { color: #d9a23a; } + #titleList { margin-top: 8px; } #repForm fieldset { border: 0; padding: 0; margin: 12px 0 0; } #repForm legend { font-size: 12.5px; color: var(--muted); padding: 0; margin-bottom: 4px; } #repForm label.opt { display: inline-flex; align-items: center; gap: 6px; margin: 4px 14px 0 0; color: var(--text); font-size: 13.5px; } @@ -458,10 +464,13 @@

Send to Frame

Drop files here - Files land in ~/Downloads. .apk files install as their own Android app. + Files land in ~/Downloads. .apk files install as their own Android app; + a game's .zip, folder or .exe becomes a title in the Steam library.
+

Sideloaded titles

+
Loading…
@@ -567,6 +576,22 @@
+ +
+

Add to the Steam library

+
+ +
+ + +
+
+
+ + +
+
+