The old pin ("running progress bars sweep, capacity bars do not") only read
animationName — but getComputedStyle is readable on a display:none subtree, and
the element it found was the .track.pulse inside #view-tasks, hidden by default.
So it was green while nothing on screen was animating. That is exactly the
defect the user reported ("demo5 has no flashing progress bar").
ui_demos_check.mjs
- strength: require a non-zero layout box AND animationName !== "none"; switch to
#view-tasks inside the assertion and switch back (synchronous, so it stays
atomic under Promise.all). The negative half only checks the track's
visibility, not the fill's — a queued row's fill is legitimately 0 wide.
- scope: also cover the task table's progress column (.row .bar[data-p]), which
is the same task as the big card; classifying .bar by class had filed it under
"capacity meter, never animate".
- print the total at the end: the "N assertions" figure in the docs had been
obtained by grepping the output and was one too high (claimed 99, actually 98).
- comment: 8888 -> 2026 (the default port changed, the comment did not).
proposal_check.mjs
- keyword 99 项 -> 98 项, matching the measured count.
Style A and E are view-switched pages (mutually exclusive `.view` containers).
The runner only ever measured the default view, so the other four were
unguarded — and the switch itself was never asserted. Style E's first version
had navigation that only moved the `aria-current` highlight while the content
stayed identical, which is exactly the failure this now catches.
Both gaps paid for themselves immediately:
- style A: `view-tasks` overflowed 65px at 390px. `.task` is a grid item whose
`min-width:auto` resolved to min-content (371px) and burst the single column
out of a 288px content area. Fixed with an explicit `minmax(0,1fr)` track,
plus `overflow-wrap:anywhere` on the mono path line (a long
`Media/Streaming/…/pak` segment has no break opportunity).
- style E: the fake navigation had to be replaced by real view switching.
Also tightened two things that were measuring the wrong element:
- style E's contrast targets moved into the default view — reading
`getComputedStyle` on a `display:none` node returns values, but "measuring an
element nobody can see" is not a check.
- the focus-ring group now switches to a view that actually contains a
focusable row; otherwise the "inline inset ring" convention was silently
skipped and the nav button was measured instead.
140 assertions, all passing.
Enumerated every independent save-writeback implementation by fingerprinting the one
system call a write-back cannot avoid — sceFsCreatePfsSaveDataImage. 25 code hits,
minus SDK stubs and verbatim derived copies leaves 5 distinct projects; read each
write path. Result: 0 of 5 force a snapshot before write-back.
garlic-savemgr copy_file(local, ORIGINAL) with O_TRUNC -> the original is
truncated in place; no copy at all when it already lives on
/data/. (src/main.c:419 + :692)
garlic-worker no write-back step at all, and save_periodic_cleanup()
unlinks every leftover copy -> using it as a backup loses
the save. (src/ps5/savedata.c:287-298)
elf-arsenal family verbatim copy of the above
ps5-sd-tool tmp image -> copy_recursive -> unmount; 0 backup keywords
apollo-ps4 real backup UI, but _addBackupCommands() lists "Apply Changes
& Resign" BEFORE the "File Backup" section -> edit first
savescum (MIT) best backup UX (timestamped), but restore never forces a
backup first, and FTP sees only plaintext
vsh-utils/trophies 0 hits
So this is the domain's only vacuum, and it is a live data-loss path in the shipped
competitor rather than merely a missing feature. Recorded as the domain's admission
criterion, not an optional extra.
proposal_check.mjs:
- tables 14 -> 15 (new section 2-2-bis evidence table)
- NEW named assertion: no table may exceed the 960px content column. The generic
page-overflow check did catch this, but pointed at the page instead of the cause.
TRAP: the sheet's global `td:first-child{white-space:nowrap}` also matches a
`td[colspan]` full of prose (it IS the first child of its row), so one note row
rendered as a single 2265px line and blew the table out to 2085px. Fix was to
remove the conflict — move the prose out of the table — not to raise specificity.
- keyword assertions for the four hard rules and the evidence
- quotedOnly guard: the mis-stated install-layer AuthID 0x3800000000000010 (zero hits
in etaHEN; section 6 carried a stale copy of it until now) may only survive inside
a correction, never as a live claim
The four UI demos and the proposal spelled the auto-created folder as
`PPSA28495-app.rar/` -- the archive's full filename including its
extension. The folder must be `PPSA28495-app/`.
- demos 1-4: the subdir target is now `.../PPSA28495-app/`
- proposal: add the derivation spec -- strip the WHOLE archive suffix,
not just the last extension (`a.part01.rar` -> `a`, not `a.part01`;
`a.7z.001` -> `a`, not `a.7z`), reusing the regexes already in
assets/main.js:816-869 -- plus a worked strip table covering .rar /
.zip / .7z / RAR5 volumes / 7z byte-split sets
- proposal_check: table count 13 -> 14, assert the spec text survives,
and quotedOnly("-app.rar/") so the wrong target cannot return as a
live claim
- ui_demos_check: assert no `.(zip|rar|7z)/` path appears in any demo
Four UI style demos for the rewrite project now share one information
architecture, so the review criteria that keep them honest are worth
pinning as assertions rather than eyeballing:
- horizontal overflow at 1920 / 1280 / 390, including *after* opening
the notes drawer and the extract dialog — a transform-translated
drawer still extends the scrollable area, and `visibility:hidden`
does not stop it
- WCAG-AA contrast on 20 sampled text/background pairs, measured from
computed styles with an ancestor walk for the effective background.
This is how a 4.43:1 secondary colour was found (threshold 4.5:1)
- focus ring actually renders, and the focused row stays inside its
parent box (the inline-inset-ring convention for 46-52px list rows)
- no `button[disabled][title]` — `disabled` implies
`pointer-events:none`, so the tooltip explaining *why* a button is
disabled can never appear; the demos use `aria-disabled` instead
- content baseline: the four nav domains, the kstuff platform chip, the
average-speed progress wording, the save-mount singleton notice, the
pick-a-directory extract target, and "make subdirectory" defaulting
to unchecked
The script also screenshots each demo and inlines them into the
workspace-root comparison page as data URIs (kept single-file, no
external requests), marking placeholders with a `data-shot` attribute
so re-runs are idempotent.
104 assertions pass.
Round 7 of the rewrite proposal added section 2-1-bis, which pins the install
path to a self-built installer that needs kstuff only (no etaHEN). Guard the
load-bearing facts so a later edit cannot silently drop them:
- GPL-3.0-or-later reuse right + the NOTICE attribution obligation
- the AuthID that actually works: DEBUG_AUTHID 0x4800000000000006
(the documented 0x3800000000000010 has zero hits in the etaHEN source)
- self-elevation (kernel_set_ucred_authid / -lkernel_sys) and status polling
(sceAppInstUtilGetInstallStatus), both of which this repo currently lacks
- the MetaInfo 0x38 -> 0x30 ABI correction: slot / is_playgo_enabled came from
the Mono managed signature, not from the native struct
- DPI v2 being browser-reachable (Access-Control-Allow-Origin)
Also generalise the falsified-claim guard into quotedOnly() and add a second
case for "the SDK grants the permission" (rebutted by singleDPI's explicit
kernel_set_ucred_authid call, which is what actually runs on the console).
Tables 10 -> 13; 37 assertions, all passing.
`.build/proposal_check.mjs` verifies `ps5-nas-rewrite-proposal.html` -- a
document that deliberately lives outside the repo -- by loading it in real
Chromium and asserting: zero external requests, no horizontal overflow at
1100px and 390px, and the structural counts its sections depend on.
It was the last harness with no copy anywhere but this machine, so clearing
`.build` would have lost it. Adds it to the .gitignore allowlist next to the
other render checks and extends its assertions for the fourth capability
domain (save management):
- tables 8 -> 9, phase cards 5 -> 6
- six content assertions (garlic-savemgr, /dev/pfsmgr, sceFsMountSaveData,
save-management heading, Phase 5, re-sign) so a later edit cannot silently
drop that whole section
Follow-up to the .gitignore whitelist. Both READMEs described the frontend
harnesses as living "outside the gitignore whitelist", which stopped being true
one commit ago. Reword both, and add .build/ to the project-layout tree now
that it holds tracked files.
The eight validation scripts live under .build/, which is an otherwise
whitelisted scratch directory. They were not on the whitelist, so they were
tracked nowhere -- a scratch-tree cleanup deletes them permanently, and there
is no copy on the release page either.
- ui_retry_test.mjs, ui_upload_menu_test.mjs -- headless-DOM tests against
assets/main.js (extract password retry flow; upload menu, i18n key coverage,
CSS cascade scoping, extract-button state)
- preview_build.py, preview_stub.html, preview_check.mjs -- serve the real
index.html against a stub API and drive it in headless Chromium (hidden and
focus state, layout overflow, computed highlight values, wrap thresholds)
- readme_header_render.mjs, compare_html_check.mjs, compare_simple_check.mjs --
render checks for the README header badges and the comparison pages
Scripts themselves are unchanged; this only stops .gitignore from hiding them.
Three "only this fork has it" claims contradicted upstream's own source:
* Encrypted archives are not fork-only. Upstream's frontend carries a full
password flow (archive_password_required, prompt(), and it persists the
password to localStorage), and its helper IS 7-Zip. Moved to "shared".
* Split volumes are not fork-only either: upstream's extension regex already
matches .001 and its README lists .part01.rar.
* "The helper is ~100 MB" was wrong -- the released asset is 1,017,616 B, and
the correction reverses the verdict: our single file is 33.7% smaller than
upstream's two files combined.
Neither claim could have been caught from the READMEs: upstream's never says
the word "password". Same for the mixed-encoding note, which now says what the
fork actually adds (decoding the error text) instead of claiming the mechanism.
Also restored the inherited "rename" feature to both READMEs -- upstream lists
it, we have it, the fork's feature list had dropped it.
Release notes are now a flat "What's Changed" bullet list instead of an essay;
.github/release-notes-template.md records the shape so they stay short.
Both READMEs had grown into an upstream-style append log ("What's new in
v1.7 / v1.8 / v1.8.1 / v1.8.2 / v1.9 / v1.9.2 / v1.9.3M") that also dropped
several features the upstream README still documents. Rewrite both as
feature-oriented descriptions with one shared outline.
- README.md, README.zh-CN.md: feature-oriented rewrite. New sections for
the archive support matrix, limits and safety, conflict policy,
password handling, what is deliberately not built, and an
upstream-vs-us comparison table that includes the rows where upstream
wins.
- Add header badges (release / license / target / downloads) and two
for-the-badge buttons (Download, beginner's guide). The release badge
reads github/v/release so it tracks the latest release instead of
hard-coding a version; colours are pinned so they do not clash.
- CHANGELOG.md: write the missing [v1.9.1] and [v1.9.2] body sections
before that prose was removed from the README, so nothing was lost.
- Fix 4 dead links to third_party/unrar/ (renamed unrar7/ in v1.9) and one
reference to a README section that no longer exists.
- HANDOVER.md: update the handover commit and README structure note.
Documentation only. No binary rebuild, and tag v1.9.3M with its release
asset is untouched.
Verified: link + anchor audit across the tree (now including raw HTML
href/src, not just Markdown syntax) reports 0 problems; table column
counts consistent; bold pairing clean; the new header renders with 6/6
badges loaded at both 1280px and 420px viewports and no horizontal
overflow.
The release is out, so every "unreleased / not yet committed" statement in the
docs became false. Updated:
- CHANGELOG: the artifact header says published and links the release; the
[v1.9.3M] section carries the release date; the sentence claiming the
GitHub release was "still v1.9.2" is gone
- README (both languages): the version line reads v1.9.3M, the "what's new"
heading loses its "(unreleased)" qualifier, and the note around it links
the release instead of saying the published binary lacks all of it
- HANDOVER: current commit / tag / release, the artifact table row (no longer
"worktree, uncommitted"), the test counts (27 -> 40 retry checks, plus the
40 menu checks and the 12 headless assertions), and the two section headers
that said "not committed"
- DEVICE-TEST: the checklist has now been run and passed, so it is an
acceptance record rather than a to-do; the rollback pointer also offers the
v1.9.2 release page
- FORUM-POST-v1.9.2: the note about two gaps closed in the worktree but not
yet released now says they shipped in v1.9.3M
Left alone deliberately: the single remaining "the then-unreleased worktree"
phrase in DEVICE-TEST, which states a historical fact about the copy that was
named v1.9.2.
First release under the fork-marker convention: VERSION_TAG carries a trailing
`M`, so /api/version, the PS5 start-up notification, the stdout banner, the UI
footer and the ELF file name all read "v1.9.3M" in one move -- and a fork build
can no longer collide with an upstream artifact of the same version, a mix-up
that already happened twice. The footer carries a tooltip spelling the marker
out.
Two bodies of work.
1. Encrypted archives (ZIP / RAR / 7z)
- ZIP: ZipCrypto (traditional PKWARE) and WinZip AES-256, through
minizip-ng plus a vendored crypto layer (mz_crypt_wfm.c,
mz_strm_pkcrypt.c, mz_strm_wzaes.c).
- RAR: RARSetPassword, wired after RAROpenArchiveEx and before the first
RARReadHeaderEx. The ordering is load-bearing, not stylistic.
- 7z: 7zAES including -mhe=on encrypted headers, via a virtual
ISeekInStream that splices a pseudo-header + the real archive + the
decrypted header, so no offset stored inside the archive has to move.
A wrong password is reported as ZIPX_ERR_PASSWORD, and a failed attempt
leaves no staging directory behind.
2. Reporting, and the UI round that on-device testing produced
- A RAR whose dictionary exceeds what the build supports now gets its own
extract_dict_too_large code instead of being mis-reported as "entry too
large"; the message names both the required and the supported size. The
behaviour is deliberately unchanged -- such archives are still refused,
because admitting one means allocating the whole window up front, which
is why rarlab's own CLI refuses them by default.
- The upload entry is a menu again: one "Upload" button opening "Upload
files / Upload folder". The previous main-button-plus-small-arrow made
"upload folder" effectively undiscoverable.
- A drag-and-drop hint sits in the footer (hidden on the console browser,
where drag is not how anyone uploads).
- The extract button is now always present and merely disabled until
exactly one archive is selected, instead of appearing out of nowhere.
- Upload-and-extract on an encrypted archive now prompts for the password
directly. The retry table used to be keyed by PATH, and for a non-ASCII
directory the string the page holds and the string the server reports
are not the same bytes -- the lookup missed, so the user got a bare
error box and had to press Extract by hand before the prompt appeared.
It is now keyed by task id, which the server assigns and echoes back
verbatim.
- Error text passes through decodeFsText(), so a GBK entry name no longer
surfaces as `â®…ç§.psd`.
- Local names are encoded with encodeFsText() before being joined onto a
server-side path. fs_path_value() declines to rewrite a path if ANY code
point exceeds 0xFF, so concatenating a local name onto a server directory
produced a mixed representation and a silently dead path.
- The menu row highlight was losing the cascade to the generic button rule
(identical specificity, later in the file) while inheriting the toolbar's
3px focus ring, which overflowed a 46px row. Both rules are now scoped to
the panel and the keyboard cue is an inset ring, so it cannot escape the
row at any line height.
- The footer status line is clamped to a single line; a long
"uploading 3/12: some-name.zip" used to wrap out of the 46px footer.
- A first failed password attempt now says the archive is encrypted,
instead of blaming a password the user was never asked for.
Artifact
web-file-mgr-v1.9.3M.elf
903,448 B
sha256 8ca47d5aaca75085b32641300cce30fadb7df7749cb6b53d04f129bcecc286b7
e_machine 0x003e (x86-64 / PS5)
The file size is identical to the four builds before it, and every one of
them carries a different sha256: only .rodata moved, and by less than the
16 KiB section alignment absorbs. Compare sections with `readelf -SW` --
never infer "nothing changed" from the byte count.
Verification
- host suites: 140 ZIP + 37 RAR = 177 checks, 0 failures
- 7z suite: 27 cases, 0 failures. The `aeshe` entry that used to sit in
KNOWN_GAPS is gone -- the -mhe=on fixture now passes both the folder
decoder and the extraction facade
- frontend: .build/ui_retry_test.mjs (40 checks), .build/ui_upload_menu_test.mjs
(40 checks), .build/preview_check.mjs (12 assertions in headless Chromium
against the real page and a fixture API). Two layout regressions and the
highlight cascade bug were caught by the last one and by nothing else --
reading the source, both CSS rules "look correct"
- built twice from this tree: byte-identical (cmp clean). rsync refreshes
every asset mtime, so this is a genuine recompile, not make short-circuiting
on unchanged sources
- embedded assets verified in place with .build/check-elf-gzip.py, because
gen-asset-module.py gzips them and plain `strings` finds none of their text
- exercised end to end on a real PS5; the checklist is
docs/DEVICE-TEST-v1.9.3M.md
Docs
- docs/USER-GUIDE-zh-CN.md (new, simplified Chinese user guide)
- docs/DEVICE-TEST-v1.9.3M.md (new, on-device acceptance checklist)
- docs/REAL-CONSOLE-PROFILE.md (new, measured console behaviour)
- docs/archive/HANDOVER-v1.8-planning.md (superseded v1.8 design notes)
- CHANGELOG / README (both languages) / HANDOVER updated with the artifact
fingerprint, the section deltas and the new test counts
The v1.9.1 tag was a lightweight tag sitting on 9a36c3c, four commits BEHIND
the tree that produced the released ELF, so `git clone --branch v1.9.1` could
not rebuild the published artifact. v1.9.2 is that same tree with the version
literal bumped, which puts the tag back on the commit the binary came from.
Source delta vs the v1.9.1 tag (all of it already shipped inside the v1.9.1
binary -- the tag was just wrong, nothing was missing from the download):
- src/demangle_stub.c + -Wl,--icf=all: 1,034,328 -> 870,488 B (-15.8%)
The demangler was only reachable from the uncaught-exception path.
- upload entry: single-click file/folder pick, plus drag-and-drop upload
- docs: language switcher in both READMEs
Artifact
web-file-mgr-v1.9.2.elf
870,488 B
sha256 177e90fecf93a0251e83f67884fba4551051be248330b0d70fda8ea732f88e84
e_machine 0x003e (x86-64 / PS5)
Verification
- 163 checks (ZIP 108 + RAR 27 + 7z 28), 0 failures; `aeshe` remains the
known -mhe=on gap
- the build is reproducible: reverting the two version literals and
rebuilding reproduces the v1.9.1 ELF byte for byte, and re-applying them
reproduces this one. That is what pins the byte-level difference between
the two artifacts down to the version string alone
- docs/SIZE-OPTIMIZATION.md gains appendix A (artifact-consistency
verification, incl. the 55,280-byte raw diff explained as linker string
pool relayout) and appendix B (per-symbol accounting: 628 symbols
removed, 0 added, 76 ICF fold groups)
Not yet validated on retail hardware.
Most 7z folders are a single plain LZMA2 coder (7-Zip default -m0=lzma2
-ms=on). For that shape the facade now drives the SDK's own parallel
decoder (Lzma2DecMt, vendored with MtDec/Threads) instead of the
single-threaded chain walk; the chain stays responsible for every other
shape (BCJ2 stacks, encrypted folders, exotic methods) and as the
fallback when the platform cannot provide threads (SZ_ERROR_THREAD
downgrades, never fails).
Adapters: ISeqInStream over the read_at callback, ISeqOutStream into the
staging sink (runs on the calling thread, so the sink contract is
unchanged), ICompressProgress polling the cancel hook. The folder CRC is
taken over the decoded stream as before.
329 MiB fixture, internal timing: 1050 ms (asm, 1 thread) -> 930 ms
(4 threads) -> 765 ms (8 threads + 1 MiB inBufSize_MT), i.e. 1.37x,
now at parity with 7za -mmt=off (898 ms); 7za -mmt=8 is 485 ms. The PS5
count is 8 Zen 2 cores like the dev host; SZX_MT_THREADS=8.
Test matrices: 7z 28 + ZIP 108 + RAR 27 checks green.
The per-entry flush made entry count the dominant cost: an 8000-file
fixture spent most of its wall time in fsync (the extraction phase ran
past 200 s with it, 14.5 s without, same machine). Nothing in the
design needs that durability: a crash mid-extract leaves the staging
tree, which the next run discards, and publish is a rename-only phase.
The RAR and 7z engines never flushed per entry; all three now share
the same policy.
Measured on the same fixture (8000 small files, 1.9 MiB packed):
- with per-entry fsync: >200 s, not finished
- without: 14.5 s, extract phase 14.47 s CPU (scan 0.01 s, publish ~0)
- official 7-Zip on this host: >400 s (real-time AV scans every file
it creates; the PS5 has no such factor)
ZIP/RAR matrices green (108 + 27 checks).
The handover doc had grown to 349 lines through accretion: three separate
7z sections that repeated each other's facts, a full Frostpunk 2 debugging
transcript that has been resolved since e0bc4a6, and a "current status" line
still pointing at 00b750d. Anyone picking this up would have to read the
whole thing to work out what is actually left to do.
Rewritten around the questions a new maintainer actually asks:
* what is the state, in one table, with the exact ELF hash
* what do I run to build / test (the one-click script, plus the
/usr/bin/bash caveat and the PIPESTATUS/od-encoding traps)
* what are the design constraints I must not break (7z coder out_size,
volstream struct layout, AesOpt.c flags, *at() on PS5)
* what is left (only -mhe=on, plus real-hardware verification)
History is compressed to the facts that still constrain future work, not
the sequence in which they were discovered. New sections cover the version
plumbing (2.2/2.3), the license/ownership audit result, and the two
trap files sitting untracked in the repo root.
No code changes.
The footer string in the browser was a hard-coded `const APP_VERSION = "v1.9"`
in assets/main.js, completely disconnected from the Makefile's VERSION_TAG.
It had already drifted: the UI kept showing "v1.9" through the entire v1.9.1
release, while the startup notification and the ELF name both said v1.9.1.
Two sources of truth for one fact is one too many.
Backend:
* new src/version.c -- GET /api/version -> {"ok":true,"version":"v1.9.1",
"titleId":"FMGR88888"}, built straight from the -DVERSION_TAG /
-DTITLE_ID macros the Makefile already passes, so there is nothing left
to forget when bumping a release.
* registered in filemgr_api_request() next to /api/space and declared in
filemgr_internal.h; added to COMMON_SRCS so both the PS5 and the linux
targets link it.
Frontend:
* the literal becomes APP_VERSION_FALLBACK and is rendered first so the
footer is never empty, then loadVersion() refreshes it from the API in
the background. A failed request is deliberately swallowed -- a wrong
version string is cosmetic and should not raise an error toast.
* both the host i18n path and the PlayStation browser branch are
untouched; the footer element itself (#versionText) does not move.
Note the earlier answer to "does the version show in the PS5 UI?" was that it
does -- the bottom-right footer -- but it was showing the stale literal, not
the build's real tag. That is what this fixes.
Verified:
* WSL prospero-clang 18.1.8 -- ELF 1017864 B, e_machine=0x003e, contains
both the string "v1.9.1" and the "/api/version" route.
* MinGW gcc 16.2.0 host tests -- ZIP 108 + RAR 27 + 7z 28 = 163 checks,
0 failures.
Every build overwrote the same `web-file-mgr.elf`, so the only way to tell two
builds apart was the sha256 -- and the one thing you cannot see from a
filename on a USB stick is which firmware you are about to run. Derive the
binary name from VERSION_TAG:
BIN := web-file-mgr-$(VERSION_TAG).elf
LINUX_BIN := web-file-mgr-linux-$(VERSION_TAG)
Because VERSION_TAG also feeds -DVERSION_TAG, bumping it now updates the
embedded version string, the PS5 notification and the filename in one place.
The build scripts must not hardcode the name or they break the moment the
version changes, so build-elf-wsl.sh now reads it back out of the Makefile:
VERSION=$(sed -n 's/^VERSION_TAG *[?:]*= *//p' Makefile | head -1)
BIN_NAME="web-file-mgr-${VERSION}.elf"
rsync exclude widened to /web-file-mgr-*.elf, and build-win.sh now always
re-pushes the WSL script (it only copied it when missing, so editing
build-elf-wsl.sh silently kept running the stale copy).
Verified: same sources as the previous commit produce a byte-identical binary
under the new name -- sha256 94851c26ebe1a3b296fb78748aa8de90ea0c022aa4004d64e700d4156a0cc64a
both before and after, so this commit is pure plumbing.
The ELF we tagged v1.9.1 still reported "Version: v1.9" -- Makefile line 16
pinned VERSION_TAG by hand and nobody updated it when the tag was cut, so the
PS5 boot notification and `web-file-mgr.elf --version` both mislabeled the
binary. Bump it to v1.9.1 and switch := to ?= so a one-off build can override
it (`make VERSION_TAG=v1.9.2`) without touching the file.
Rebuilt ELF:
1017864 B, sha256 94851c26ebe1a3b296fb78748aa8de90ea0c022aa4004d64e700d4156a0cc64a
e_machine=0x003e, `strings` confirms the embedded literal is now v1.9.1
.gitignore: the per-directory deny list for .build/ never kept up -- every
debugging session drops a new probe directory (aespw/, bigzip/, chaincheck.py,
dbg/, engine-e2e/, ...) and they all showed up as untracked. Flip to a
whitelist: ignore .build/* and re-allow only the files that are genuinely part
of the repo. Also ignore .workbuddy/ (session data: never commit, never delete).
Two scripts, so building for PS5 is one command:
/usr/bin/bash .build/build-win.sh
build-win.sh (Windows side, Git Bash)
Just pipes the WSL script into wsl.exe via stdin and propagates the
exit code. Uses `wsl.exe -- bash < script` rather than `-- bash -c '...'`
because the repo path contains spaces and -c's argument gets split.
Copies the WSL script over first (.build/ is excluded from rsync so
the WSL side may not have it yet).
build-elf-wsl.sh (WSL side)
5 stages, each failing loudly:
1. env check -- SDK present, prospero-clang runnable,
libmicrohttpd in the sysroot
2. rsync -- Windows repo -> /home/song/ps5-web-file-manager,
incremental, skipping ps5-obj/ gen/ tests/ docs/,
then a smoke check that this session's key files
landed (sevenz_*.c, zipx_common.c, Makefile, ...)
3. make all
4. verify -- size / sha256 / e_machine, with an explicit
aarch64 guard (183) since PS5 is x86-64 (62)
5. copy back -- ELF -> Windows project root
make's exit status is read through PIPESTATUS (the `| tail -120`
would otherwise swallow it), so a failed compile actually fails the
script instead of sailing on to the verify stage.
One gotcha worth recording: `od -An -tx2 -j 18 -N 2` prints the
2-byte little-endian *value*, so the on-disk bytes "3e 00" come out
as "003e", not "3e00". Matching on the byte order fails; convert with
$((16#$EM)) and compare numerically (62 = x86-64, 183 = aarch64).
Verified end to end: 1017864 bytes, sha256 2eb04581...473a157,
e_machine 0x003e (62), copied back to the Windows project root.
PS5 (Zen 2) has every AES-NI / AVX / VAES instruction set, but prospero-clang
defaults to a generic x86_64 target that does NOT pre-declare
_mm256_aesenc_epi128 -- AesOpt.c relies on a compiler-version macro that lets
clang 18 in unconditionally, so it blew up with 20 errors and stopped at
-ferror-limit= before the rest of the build could even run.
You can't drop AesOpt.c either: Aes.c references the
AesCbc_{Encode,Decode}_HW / AesCtr_Code_HW[256] symbols via AesGenTables,
so without AesOpt.c the link fails (five undefined-symbol errors).
Fix: add a path-filtered CFLAGS_7Z = -maes -mavx2 -mvaes and apply it only to
third_party/7z/*.c -- zlib and minizip-ng don't need it, the project sources
don't need it, and it's a no-op at runtime since the symbols exist on PS5.
Verified:
* WSL prospero-clang++ 18.1.8 (FreeBSD sysroot) -- success
web-file-mgr.elf 1017864 B, sha256 2eb04581...473a157, e_machine=0x003e
* MinGW gcc 16.2.0 host tests -- 163 checks (ZIP 108 + RAR 27 + 7z 28) all pass
UI regression: a 7z with fewer than 4096 entries spun a "scanning archive"
label and never replaced it until extraction had been running long enough
for the byte-level callback to fire, on a slow PS5 that meant "looks hung".
Lower the per-entry barrier in scan_entries to 256 entries, force a final
report after the scan so entries_total / bytes_total always reach the UI,
and force a report at the start of precheck_folders so the throttled
"between scan end and first extraction byte" window is not silent either.
Error regression: when the password was wrong the engine returned
ZIPX_ERR_PASSWORD with an empty detail field; extract_set_error copied it
into task->error_arg verbatim, so the i18n template's {arg} expanded to ""
and the user saw "Wrong password: ". Carry the original archive path into
the error detail so the toast reads "Wrong password: myfile.7z".
Languages: drop the stale "only ZIP/RAR" line from err_extract_unsupported
(7z has been supported since v1.9), and add the new err_extract_password
key in both en/zh.
Tests: lock down both regressions.
* progress recorder: bytes_done monotonic, total > 0, final byte count
reaches >= 90% of the declared total.
* wrong password: r.detail contains the archive basename.
7z 28 + ZIP 108 + RAR 27 = 163 checks, 0 failures.
With the extraction engine in place (021c9cb), the /api/extract task
finally picks .7z files up.
* src/filemgr_internal.h -- file_task_t gains extract_password (256
bytes, UTF-8, the size a sane user will never exceed but enough for the
encrypted RAR / 7z passwords the engine actually accepts).
* src/extract.c -- extract_dispatch() routes a single .7z to sevenz_extract()
and a byte-split set (name.7z.001, .002, ...) to the same call (the
volume detector classifies by extension). api_extract() reads the
optional "password" form field and copies it into the task; the error
code map gains ZIPX_ERR_PASSWORD -> "extract_password".
* Makefile -- the sevenz_extract / sevenz_chain / sevenz_volstream sources
join COMMON_SRCS, and every third_party/7z/*.c (LZMA SDK 26.03, public
domain, decoder subset) joins THIRD_PARTY_C_SRCS. The C flag set gains
-Ithird_party/7z and -DZ7_PPMD_SUPPORT, the latter so PPMd is built
in (the SDK guards PPMd sources behind a macro that defaults to off).
* assets/main.js -- isSevenZipArchive + isSevenZipSplitVolume; both feed
isExtractableArchive. actionExtract() now prompts for a password on
7z archives (encrypted-stream support is opt-in -- the engine rejects
unencrypted archives with no password just fine, but the prompt saves
a wasted scan when an archive is encrypted). startExtractTask() takes
the password and sends it as a form field.
* assets/lang-{en,zh}.js -- extractPasswordAsk string for the new prompt.
ZIP 108 / RAR 27 / 7z 26 still all green on the host.
No real-device build attempted in this commit: the WSL runner is the
one that knows the PS5 toolchain, and the user has to be the one to
push it through. The Makefile changes are the wiring the cross
compiler needs; a `make linux` on a Linux host with libmicrohttpd-dev
should link end-to-end.
The 7z chain decoder (commit 2748b38) handles bytes; what the /api/extract
task needed was a third sibling of zip_extract.c and rar_extract.c that
runs that chain over a real archive, mirrors their staging / publish /
rollback / name-validation machinery, and fills in zipx_result_t the same
way so the dispatcher and the web UI can treat every format alike.
* src/sevenz_extract.[ch] -- one pass over the archive (scan: validate every
name, sum bytes, drop archives the chain cannot drive), then folder by
folder through the chain. The publish phase recurses with OVERWRITE and
MERGE alike for matching directories, and only differs at the file
leaves -- a strict non-recursive OVERWRITE would have made the policy
useless for any archive that overlaps a directory already on disk.
* src/zipx_common.c -- the limits profiles and zipx_status_string() that
the three engines share. Pulled out of zip_extract.c so every consumer
gets the same error text and the same MAX_* numbers.
* src/zip_extract.[ch] -- ZIPX_ERR_PASSWORD joined the contract (7zAES).
* tests/run-sevenz-tests.sh -- the engine matrix now also runs the facade
over every fixture, plus 22 error and policy checks against
test_sevenz_extract. KNOWN_GAPS holds only "aeshe" (encrypted header);
the plain 7zAES fixture is in the matrix.
* tests/test_sevenz_extract.c -- end-to-end driver for the facade.
The POSIX shim the host test runner injects needed three more pieces so
the tests pass on a CP936 host:
* lstat/stat -> wfm_stat (the MinGW ANSI entry points can't see a UTF-8
filename in CP936; without the redirect a non-ASCII entry causes the
publish phase to lstat() a mangled path and fail with ENOENT).
* opendir/readdir/closedir -> the wide variants, so readdir() hands us
real UTF-8 names instead of CP936 bytes that nothing can round-trip.
* fopen -> _wfopen, for the test helpers that read a non-ASCII file
back to verify it.
ZIP 108 checks / RAR 27 checks / 7z 26 checks, all green; the AES
fixture extracts with the password "Secret123" and rejects wrong / absent
passwords with "password required or wrong" / "wrong password, or the
archive is damaged".
tests/fixtures/ gains the volume-set fixtures (broken.zip.001, gap.zip.*,
disks.*, parts.part*.zip, plain.zip.*, split_single.zip) that were
generated by tests/make_split_fixtures.py in earlier sessions but never
made it into the index.
Next commit wires sevenz_extract() into the dispatch in src/extract.c and
recognises 7z archives in assets/main.js.
7-Zip wraps the packed stream of a password protected archive in a 7zAES
coder (method 0x06F10701). The bundled LZMA SDK has no C implementation of
it, so add one to the folder chain:
* aes_props_layout() splits the property bytes into numCyclesPower, salt
and IV exactly as CDecoder::SetDecoderProperties2() does, and rejects a
block whose declared lengths do not match its size;
* aes_derive_key() runs the 7-Zip KDF: SHA-256 over
(salt || password_utf16le || counter_le64) repeated 1 << numCyclesPower
times, with the 0x3F power meaning "no derivation", the key being the
salt and password copied into 32 bytes;
* utf8_to_utf16le() converts the caller's password, since that is the
encoding 7-Zip hashes;
* the SZ_N_AES node decrypts one block at a time with Aes_SetKey_Dec() /
AesCbc_Init() / g_AesCbc_Decode(), keeping the trailing partial block in
the node so the declared (unpadded) unpack size is what gets delivered.
sz_chain_decode() gains a `password` argument, and sz_chain_needs_password()
lets a caller ask for one before it touches the filesystem. A folder that
needs a password and did not get one fails as SZ_CHAIN_ERR_PASSWORD with a
message naming 7zAES, rather than as a generic corrupt archive.
Because a wrong password decrypts to plausible looking rubbish, a failure
from a folder that carried a 7zAES coder gets "(a wrong password looks like
this)" appended, so the UI can offer a retry instead of calling the file
damaged.
numCyclesPower comes from the archive and drives 2^n SHA-256 passes, so it is
capped by the limits profile (max_aes_cycles, 2^24 for both the default and
the large profile).
An encrypted *header* (-mhe=on) is a different problem: the archive header is
encrypted with the same coder and has to be decrypted before any folder
exists. That stays out of scope and is reported as such.
tests/run-sevenz-tests.sh: aes leaves KNOWN_GAPS and passes; aeshe stays
there with the reason spelled out.
A split 7z (`name.7z.001`, `.002`, ...) only contains the first slice of the
archive, so SzArEx_Open() followed the header offsets off the end of the file
and gave up with SZ_ERROR_INPUT_EOF -- the set was unusable even though the
file listing of each part looked fine.
src/sevenz_volstream.c backs ISeekInStream with the ordered part list, so the
SDK sees one continuous archive and never learns it was split. Nothing is
merged on disk: a 160 GiB set would otherwise need a second 160 GiB scratch
copy. A 7z split is a plain byte split, so offsets stored inside the archive
are already absolute and the Stream/Seek callbacks stay trivial.
The ordered list comes from zipx_volume, which already understands
`name.7z.001` naming, so an incomplete set fails before any decoding starts
and names the missing part. Two further checks report a set that could not
have worked anyway: a lone first volume, and a part whose size differs from
the earlier ones (7-Zip cuts equal sized parts and lets only the last one be
short).
To reuse the split-layout constants without dragging the ZIP stream header
into the 7z build, ZIPX_VOL_MODE_* moved to zipx_volume.h: they describe the
set, and every consumer of zipx_volume_t needs them.
The e2e driver now opens its input through the volume stream, so the packed
data reads go through it too.
Matrix: all 10 single-volume fixtures plus vol.7z.001 are byte-identical
(13 passed, 0 failed). Remaining known gaps: 7zAES (aes, aeshe).
Regression: ZIP 108 / RAR 27 checks, 0 failures.
The LZMA SDK's CSzFolder caps a folder at four coders, so anything 7-Zip
writes for BCJ2 (4 packed streams + 5 coders) comes back as
SZ_ERROR_UNSUPPORTED from SzAr_DecodeFolder. The header scanner has no such
limit (64 coders), which makes the failure look like corrupt data instead.
src/sevenz_chain.c parses the folder descriptor itself and drives the codec
chain as a pull pipeline: every decoder hands up to N bytes to its consumer
and asks upstream for more when it needs it, so a folder is decoded straight
into the caller's sink without staging the whole thing. Covered here:
- Copy / LZMA / LZMA2 / PPMd, Delta, and the x86, PPC, IA64, ARM, ARMT,
SPARC branch converters
- BCJ2, whose three side streams have to be materialised while MAIN keeps
streaming
- per coder output sizes taken from UNPACK_INFO, which is what keeps the
chains exact
tests/sevenz_chain_e2e.c decodes each folder once and splits the byte stream
across the entries that share it, mirroring the real extraction path, and
checks the per-entry and per-folder CRCs on the way past.
tests/make_sevenz_fixtures.py builds the fixture matrix (including
non-solid `-ms=off` folders), and tests/run-sevenz-tests.sh runs it while
keeping a KNOWN_GAPS list that fails loudly once a gap closes.
Two real bugs surfaced while bringing the matrix up:
- build_coder() gave every coder node the *folder's* unpack size instead of
its own entry in UNPACK_INFO. In a BCJ2 folder MAIN is regularly larger
than the folder's final size (300066 vs 300000 in the fixture), so the
node silently truncated its own output and the chain came up short by 21
bytes per LZMA2 layer. Only archives where a coder is bigger than the
folder tripped it, which is why single-folder BCJ2 passed and the same
archive split into per-file folders did not.
- the e2e driver kept `written` across folders, so a folder that failed
mid-entry made every later entry look half-written and the failures
cascaded.
Scripts no longer delete anything: objects are overwritten and each case gets
a fresh output directory, so the suite runs under a guarded shell.
Matrix: 10/10 single-volume fixtures byte-identical (store, lzma2, lzma,
ppmd, bcj, delta, utf8, bcj2, solidoff, bcj2off). Still open and listed as
known gaps: 7zAES (aes, aeshe) and multi-volume input (vol.7z.001).
Groundwork for 7z support (the second of the six format x volume
combinations). No engine code yet: this lands the vendor subset, real
fixtures and the end-to-end driver, so the engine has something to be
built and checked against.
Vendor (third_party/7z, LZMA SDK 26.03, public domain):
- decoder-only subset: 7zArcIn/7zDec container, Lzma/Lzma2/Ppmd7/Bcj2/
Bra/Delta codecs, Aes/Sha256 for the password channel to come
- the encoder half (LzmaEnc, Xz*, Sort, Threads, ...) is deliberately
not vendored; the engine never encodes
- README.md records two SDK limitations that shape the engine design:
* CSzFolder caps a folder at 4 coders / 3 bonds, so 7-Zip's own BCJ2
chain (BCJ2 + 4xLZMA2 = 5 coders) is refused by SzAr_DecodeFolder
even though SzArEx_Open parses it happily - the engine therefore
parses folder blobs and drives the codec chain itself
* the C decoder ships no 7zAES coder at all, so both -p and -mhe=on
archives are rejected until the engine implements it on top of the
vendored Aes.c / Sha256.c
Tests:
- make_sevenz_fixtures.py builds real archives with an actual 7-Zip
binary (7za from the Extra package; falls back to the reduced 7zr,
which has no PPMd encoder): store/lzma/lzma2/ppmd/bcj/delta/bcj2,
AES with plain and with encrypted headers, a 7-part -v100k volume
set, and UTF-8 entry names
- sevenz_e2e.c extracts an archive and validates every entry against
the stored CRC; it calls the wide-character file APIs on Windows, so
non-ASCII entry names are really created instead of silently failing
- run-sevenz-tests.sh builds the subset and the driver, then runs the
fixture matrix with an explicit KNOWN_GAPS list that fails loudly
when one of the gaps starts passing
Status: store, lzma, lzma2, ppmd, bcj, delta and utf8 all extract
byte-identical to the source tree (7/10). bcj2, aes, aeshe and
vol.7z.001 are the known gaps, each mapped to a specific piece of
engine work still to come.
A split ZIP is several files that only form an archive together. Until now a
`.zip.001` was rejected as an unsupported format and a `.z01`/`.partN.zip`
member could not be resolved, so these downloads were unusable.
Three naming conventions are recognised, from any member of the set:
name.zip.001, name.zip.002, ... byte split (7-Zip)
name.part1.zip, name.part2.zip byte split (WinRAR)
name.z01, ..., name.zip zip split disks (Info-ZIP / PKZIP)
The parts are served through a new stream that makes them look like one
archive, so nothing is copied to disk first (a 160 GiB set would otherwise
need a second 160 GiB scratch copy):
- src/zipx_volstream.c implements an mz_stream over the ordered part list.
Byte splits keep absolute offsets; zip split disks carry offsets relative to
the disk an entry starts on, so that mode honours
MZ_STREAM_PROP_DISK_NUMBER the way mz_zip_entry_seek_local_header drives it
and starts on the last volume, where the central directory lives.
- src/zipx_volume.c groups the set, verifies the numbering is complete and
reports exactly which volume is missing.
- zip_extract.c tries the layout implied by the naming first and the other one
as a fallback, so tools that disagree about offset bases still work.
- extract.c dispatches volume members to the engines and, when the caller
asked for the source to be deleted, removes every volume instead of leaving
orphaned parts behind.
Error messages are specific on purpose: an incomplete set reports the missing
file name (e.g. "gap.zip.002 is missing"), a lone first volume says the
remaining volumes are missing, `.rar.001` sets explain the rename that makes
them work, and 7z volumes say 7z is not supported yet.
Tests: tests/make_split_fixtures.py builds all three layouts (the disk set is
written with per-disk offsets exactly as APPNOTE 4.4.11 describes) plus two
broken sets; test_zip_extract.c extracts each layout from a different member
and byte-compares the nested binary entries. 108 ZIP + 27 RAR checks, 0
failures.
Two issues found in real-machine testing of v1.9:
1. RAR multi-volume extraction never advanced the progress bar. The engine
only accounted bytes_done after RARProcessFile() returned for a whole
entry, so a multi-GB entry spanning volumes froze the UI for its entire
duration. Wire RARSetCallback/UCM_PROCESSDATA, which unrar fires per
decompressed chunk during disk extraction, and fold each chunk into
bytes_done + a throttled progress report. (DLL-mode break handling is
never armed, so cancellation stays entry-granular; returning -1 would
be ignored.)
2. A 95k-file game zip was refused as a "compression bomb" because one
small entry exceeded the per-entry uncompressed/compressed ratio cap
(500/1000). Such entries are common and harmless in legitimate archives
(zero-filled placeholders, sparse blobs): actual bytes written are
bounded by the declared size (enforced during extract) and check_space()
verifies the full declared total against real free space before any
writes. Ratio screening now only applies to entries >= 1 GiB
(ratio_min_bytes, default in both profiles); sub-GiB high-ratio entries
are accepted. Error text now includes compressed -> uncompressed sizes.
Tests updated: small high-ratio bomb.zip now extracts OK under both
profiles and is rejected again when ratio_min_bytes is lowered below its
size; RAR multi-volume fixture asserts mid-entry progress events and that
final bytes_done reaches bytes_total. 73 ZIP + 27 RAR checks, 0 failures.
Real-machine diagnosis of the Frostpunk 2 failure: mkdirat() returns -1
with errno 0 ("cannot create directory ... (No error)") on PS5 hardware.
The *at() symbols link against the SDK libc but misbehave at runtime,
while plain path-based calls work (the staging root open() succeeds and
small flat archives extract fine).
Every *at() call in the extract phase now retries as a full-path call
built from the staging root before reporting an I/O error:
- open_parent_dirs: mkdirat -> mkdir, openat -> open
- write_entry: openat -> open, renameat -> rename, unlinkat -> unlink
Error messages now append (errno=%d) so errno=0 can no longer hide
behind a misleading "No error" strerror string. Host suite stays green:
94 checks, 0 failures.
Songlx516