Real regression from c9d721f: pinning CURLOPT_PROTOCOLS_STR /
REDIR_PROTOCOLS_STR to "https" on the piece downloader (and the simple
file streamer) broke real-world Sony patch downloads. Banishers reliably
aborted after ~73 MB and Last of Us Part I after ~127 MB — the Sony CDN
appears to 302 the piece URL to an http:// signed edge inside its own
infrastructure, and refusing those redirects killed the transfer mid-piece.
The smaller patchdl_http_get path (version.xml + manifest JSON) keeps the
HTTPS pin since those payloads always come back from the public https
endpoints. The defence still holds elsewhere: host_allowed gates every
URL to the Sony CDN allowlist, TLS verifies against the pinned SCEI root
even on a 302, and NOSIGNAL stays so a connection RST can't smuggle a
SIGPIPE back into the worker thread.
For future regressions of this class /api/downloads now exposes the last
CURLcode + HTTP status per job (last_curl_rc, last_http_code) so we don't
have to instrument the binary again to find out what curl returned.
Manifest JSON parser used substring scans with no per-piece scope; a key
defined in a later piece could be misattributed to the current one, and
the escape handling silently dropped the byte after a backslash even for
unknown escapes. json_string_after/json_u64_after now take an optional
limit pointer (NULL = legacy unbounded), and the manifest loops pass
obj_end so per-piece reads can't leak across pieces. JSON escapes are
decoded properly: \" \\ \/ \n \r \t \b \f; unknown \X drops the
backslash and keeps the payload byte.
Sanity caps on assembled manifests: PATCHDL_MAX_PIECES (4096),
PATCHDL_MAX_PIECE_BYTES (8 GiB), PATCHDL_MAX_TOTAL_BYTES (200 GiB).
A malformed manifest with a single multi-TB piece or millions of entries
is now rejected before any disk activity.
patchdl_buf_t gains an optional `max` field; write_cb fails the transfer
when growth would exceed it. patchdl_http_get preserves the caller-set
max across its internal memset(). verxml_query caps at 16 MiB,
fetch_manifest and download_manifest at 64 MiB.
host_allowed switches to strcasecmp (DNS is case-insensitive; an upstream
redirect could otherwise drop out of the list). CURLOPT_PROTOCOLS_STR /
REDIR_PROTOCOLS_STR pin all traffic and redirects to HTTPS.
CURLOPT_NOSIGNAL=1 prevents libcurl from raising SIGPIPE in a worker.
DNS label parser bounds-checks the length byte before incrementing pos,
so a malformed response with a 0xFF label near the end can no longer
read past the receive buffer.
extract_title_id used `p[8]` as the loop guard, which crossed the NUL
terminator on strings shorter than 9 chars (UB). Replaced with a
strlen-based bound.
Three read-only endpoints (no install, no writes) to inspect a downloaded
package on-device:
- GET /api/manifest/<title_id> — re-fetch the patch manifest (PatchDL bypasses
the DNS block) and dump each piece's offset/size/SHA-256.
- GET /api/pkgverify/<title_id> — SHA-256 every piece of the assembled .pkg
against the manifest hashes, on-device (SSD, no multi-GB transfer), and report
per-piece pass/fail. Proves whether the file is byte-correct.
- GET /api/pkgmeta/<title_id> — read the pkg's embedded content id + title id
(GetContentIdFromPkg) vs the target ids, to expose cross-region linkage.
Supporting code: patchdl_sha256_fd_region() (pread + OpenSSL EVP) in the net
layer, and bind sceAppInstUtilGetContentIdFromPkg in the install backend.
Used to diagnose the Dead Island 2 install: the 61.6 GB package verifies
byte-perfect (17/17 pieces) and its content id matches the target, so the
0x80B2116F install rejection is a Sony install-method limitation, not the data.
Replace the single sequential transfer with a pool of N worker threads
that pull pieces of one manifest in parallel, lifting the per-connection
~7 MB/s ceiling. One job runs at a time; the rest queue. The connection
count is configurable (1-16, default 4) and applies on the next start.
Resume is tracked per piece in a sidecar bitmap that survives a reboot,
and a one-time migration recognises a partial written by the old
sequential build (a piece-aligned contiguous prefix on disk) and marks
those pieces done so an in-progress download is not restarted from zero.
Pause keeps the partial; Cancel deletes it. Both, plus Resume, are
available at any point in a download's life.
Concurrency review fixes folded in:
- a job is published as the active (claimable) job only after its
manifest/state/fd are attached, so a half-built job can no longer be
settled to "done" before any bytes are fetched
- cancel/pause during the admit I/O window only flag the job; admit_next
is the sole finalizer, closing a use-after-free and a lost-pause race
- resuming a paused job frees the stale per-job buffers and zeroes the
committed counters before re-seeding, fixing a leak and a double-count
- the background version.xml thread is joined on shutdown before the
title list is freed
- verify_downloads is snapshotted under its own lock before the pool lock
- the web UI keeps Resume/Cancel after a failed transfer and bounds the
local "downloading" bridge flag so a card cannot wedge
An interrupted download (cancel excepted) now keeps its partial package on
disk instead of deleting it, and records the manifest it belongs to in a
sidecar (state.json). On the next start the title is flagged resumable and
the UI shows a "Paused — X downloaded" note with a Resume button.
Resume refetches the manifest, skips every piece already fully on disk, and
re-fetches only the one partially-written piece (piece-granular, no HTTP
range needed), appending the rest. The fresh-download path is unchanged. A
partial belonging to a different/older manifest is dropped and the download
starts clean; a corrupt (failed SHA-256) download is not kept.
Survives a reboot: a killed payload runs no cleanup, so the partial and its
sidecar persist under /data/patchdl until resumed, completed, or deleted.
Each Sony manifest piece carries a SHA-256 (hashValue). When the new
"Verify downloaded pieces" setting is on, every piece is hashed while it
streams to disk (OpenSSL EVP, already linked) and compared against the
manifest value; a mismatch aborts the download, deletes the partial, and
reports piece_verify_failed instead of handing a corrupt 60 GB package to
the installer.
Off by default: TLS already protects the bytes in transit and the PS5
installer verifies the whole packageDigest before applying, so this is a
fail-fast belt-and-suspenders check. It is also unverified on hardware
yet, so it stays opt-in (persisted in config.json) until confirmed
on-device; the hex compare is case-insensitive since Sony mixes cases.
Cancel a running download (the worker aborts mid-piece and the partial
file is removed) or delete a finished package, from the queue or the
title card. The progress callback now returns an abort signal that
reaches libcurl and the manifest merge loop.
Manifest merge: bound the piece scan to the "pieces" array so a later
"url" key (e.g. playgoChunkCrcUrl) can't be appended as a bogus piece,
and require each piece's fileOffset to match the bytes written so far so
an out-of-order manifest fails instead of silently producing a corrupt
package.
Report real free space on the download partition via statvfs; it was a
hardcoded 0.
Fixes found in review:
- scan: bound the SFO entry table to the bytes actually read and require
the key to be NUL-terminated before strcmp (OOB read on a crafted
param.sfo from a shadow-mounted dir).
- proc: bound the kinfo_proc walk and the name compare to the record and
the buffer.
- install: publish the API probe under the lock (data race with the MHD
worker thread) and initialize rc2.
- verxml: reject a truncated attribute value instead of returning it as
valid.
- web: keep download/install/downloaded flags across a refresh, stop the
queue poll only after repeated empty results, coerce the progress
number, and treat a cancelled download (HTTP 200, ok:false) as
not-downloaded.
Standalone PS5 payload with an embedded web UI on :12880, no etaHEN.
- Scans installed titles and classifies the source (genuine install,
ShadowMountPlus mount, preinstall, unknown) from the on-disk layout.
- Reads name, installed version and the Sony version.xml URL from the PS5
app database (vendored SQLite).
- Resolves version.xml past nanoDNS via a raw DNS query, TLS pinned to the
SCEI DNAS root.
- Filters patches to the newest one compatible with the current firmware.
- Downloads the patch and installs it through AppInstUtil (sysmodule loaded
at runtime), gated to genuine installs with a package title-id match guard.
- Action-based UI filters and an on-screen startup notification with the URL.
Knutwurst