diff --git a/CHANGELOG.md b/CHANGELOG.md index 6bda67e..7805ab6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,19 @@ All notable changes to **PS5 Web File Manager** are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -> Release artifact for v1.9.1: +> Release artifact for v1.9.2: +> `web-file-mgr-v1.9.2.elf` — size 870 488 bytes (~850 KiB) +> sha256 `177e90fecf93a0251e83f67884fba4551051be248330b0d70fda8ea732f88e84` +> ELF class 64, little-endian, e_machine `0x003e` (x86_64-sie-ps5) +> +> Behaviourally identical to the published v1.9.1 binary — the only source +> delta is the version literal itself (`VERSION_TAG` in the Makefile, plus the +> UI footer fallback in `assets/main.js`). The build is reproducible: reverting +> those two literals and rebuilding reproduces the v1.9.1 ELF byte for byte, so +> nothing else differs. See [v1.9.2] below for why the version moved at all. +> +> Release artifact for v1.9.1 (superseded — the tag pointed four commits behind +> the tree that actually produced this binary): > `web-file-mgr-v1.9.1.elf` — size 870 488 bytes (~850 KiB) > sha256 `24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07` > ELF class 64, little-endian, e_machine `0x003e` (x86_64-sie-ps5) diff --git a/HANDOVER.md b/HANDOVER.md index 32e3f74..7c6784a 100644 --- a/HANDOVER.md +++ b/HANDOVER.md @@ -1,10 +1,10 @@ # 交接文档 — ps5-web-file-manager 工作进度 -> 交接时间：2026-09-15 · 分支 `main` · 最新提交 **`0d036a7`** · tag `v1.9.1` 指向 `1fa2f09`（**需重打到 `0d036a7`**） +> 交接时间：2026-09-20 · 分支 `main` · 最新提交 **`807d129`** · tag **`v1.9.2`** 已重打到产出发布二进制的提交（此前 `v1.9.1` 落后 4 个提交，tag 与产物对不上） > > **主线（用户 2026-09-12 指令）**：「先从 zip 分卷开始吧，然后把六种组合打齐，并把密码通道补齐，注意一些报错信息提示的时候尽量详细准确」 > **状态：主线全部闭合。** 六种组合（ZIP/RAR/7z × 单卷/分卷）+ 密码通道（RAR 加密 + 7zAES）+ 报错详细信息，全部落地、测试全绿、PS5 ELF 构建成功。 -> **剩余**：① 推送 `0d036a7`；② tag 重打；③ PS5 真机端到端验证；④ `-mhe=on` 加密头（唯一功能缺口）。 +> **剩余**：① PS5 真机端到端验证（**唯一还没过的关卡**）；② `-mhe=on` 加密头（唯一功能缺口）。 --- @@ -13,29 +13,27 @@ | 维度 | 状态 | |---|---| | 解压引擎 | ZIP / RAR / 7z × 单卷/分卷（6 组合）+ 7zAES / RAR 加密 | -| 主机测试 | **ZIP 108 + RAR 27 + 7z 28 = 163 checks，0 失败**（MinGW gcc 16.2.0） | -| PS5 构建 | ✅ WSL prospero-clang 18.1.8，一键脚本可复现 | -| ELF 产物 | `web-file-mgr-v1.9.1.elf` · 870,488 B · sha256 `24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07` · e_machine=0x003e（2026-09-20 瘦身后；瘦身前 1,017,864 B，见 `docs/SIZE-OPTIMIZATION.md`） | -| GitHub | `main` 已推到 `1fa2f09`；`0d036a7` 待推；`v1.9.1` tag 待重打 | +| 主机测试 | **ZIP 108 + RAR 27 + 7z 28 = 163 checks，0 失败**（MinGW gcc；2026-09-20 在 v1.9.2 树上复跑） | +| PS5 构建 | ✅ WSL prospero-clang 18.1.8，一键脚本可复现；构建已实测**确定性**（同源两次构建 sha256 相同） | +| ELF 产物 | `web-file-mgr-v1.9.2.elf` · 870,488 B · sha256 `177e90fecf93a0251e83f67884fba4551051be248330b0d70fda8ea732f88e84` · e_machine=0x003e（2026-09-20 瘦身后；瘦身前 1,017,864 B，见 `docs/SIZE-OPTIMIZATION.md`） | +| GitHub | `main`（`807d129`）与 tag `v1.9.2` 均已推送；Release `v1.9.2` 资产为该 ELF | | 唯一功能缺口 | 7z `-mhe=on`（加密头） | -### 待用户在自己终端执行 +### 发布命令（v1.9.2） ```bash cd "/c/Users/songl/Desktop/Web File Manager/ps5-web-file-manager" -# 1) 推送本轮的 /api/version 改动 git push origin main +git push origin v1.9.2 -# 2) v1.9.1 重打到最新（1fa2f09 -> 0d036a7，让 tag 内含 /api/version + 新 ELF） -git tag -f -a v1.9.1 -m "v1.9.1 -- 7z + multi-volume + 7zAES + PS5 build + version API" 0d036a7 -git push -f origin v1.9.1 - -# 3) 可选：发 Release（附带 ELF） -gh release create v1.9.1 --title "v1.9.1" --notes-file .git/tag-v191.txt web-file-mgr-v1.9.1.elf +gh release create v1.9.2 web-file-mgr-v1.9.2.elf \ + --repo LisherSong/ps5-web-file-manager --title "v1.9.2" --notes-file ``` -沙箱内 git 出站 HTTPS 被拦（502/403），推送只能用户侧执行。 +沙箱内 git 出站 HTTPS **可用**（早先记的"被拦"是误判：`timeout 25 git ...` +命中的是 `C:\Windows\System32\TIMEOUT.EXE`，报参数错误而非网络错误）。`gh` 同样可用， +所以 commit / tag / push / 发 Release 都可以在会话里直接跑。 --- @@ -69,7 +67,7 @@ bash /home/song/build.sh 现在收敛到 `Makefile` 一处： ```make -VERSION_TAG ?= v1.9.1 # 可用 make VERSION_TAG=v1.9.2 临时覆盖 +VERSION_TAG ?= v1.9.2 # 可用 make VERSION_TAG=v1.9.3 临时覆盖 BIN := web-file-mgr-$(VERSION_TAG).elf ``` @@ -81,12 +79,16 @@ BIN := web-file-mgr-$(VERSION_TAG).elf | `1fa2f09` | 输出文件名派生自 `VERSION_TAG`；`build-elf-wsl.sh` 从 Makefile 反读版本（不硬编）；`build-win.sh` 每次都推 WSL 脚本（原来只在缺失时推，导致改了脚本 WSL 侧仍跑旧版） | | `0d036a7` | **新增 `/api/version`**，前端右下角改从后端取值（见 2.3） | +⚠️ **`assets/main.js` 里还有第二处字面量** `APP_VERSION_FALLBACK`（`/api/version` 取不到时的兜底值）。 +它不在 Makefile 的控制范围内 —— **升版本号时必须一并改**，否则后端请求失败时页脚会显示旧版本。 +v1.9.2 就是这两处一起改的。 + ### 2.3 UI 版本号改由后端提供（`0d036a7`） **问题**：`assets/main.js:38` 的 `const APP_VERSION = "v1.9"` 与 Makefile 无关，必然漂移。 **修法**： -- 新 `src/version.c` — `GET /api/version` → `{"ok":true,"version":"v1.9.1","titleId":"FMGR88888"}`，直接来自 Makefile 已传的 `-DVERSION_TAG` / `-DTITLE_ID` 宏，没有第二处要记得改 +- 新 `src/version.c` — `GET /api/version` → `{"ok":true,"version":"v1.9.2","titleId":"FMGR88888"}`，直接来自 Makefile 已传的 `-DVERSION_TAG` / `-DTITLE_ID` 宏，没有第二处要记得改 - `src/filemgr.c` 路由表加一行（紧邻 `/api/space`）+ `filemgr_internal.h` 声明 + `Makefile` `COMMON_SRCS` - 前端：字面量降级为 `APP_VERSION_FALLBACK`（先渲染，保证页脚不空），`loadVersion()` 后台刷新。**请求失败静默吞掉** —— 版本号显示错属于装饰性问题，不该弹错误 toast @@ -135,15 +137,15 @@ cd "/c/Users/songl/Desktop/Web File Manager/ps5-web-file-manager" ### 4.2 验证清单 ```bash -ls -lh web-file-mgr-v1.9.1.elf -sha256sum web-file-mgr-v1.9.1.elf -od -An -tx2 -j18 -N2 web-file-mgr-v1.9.1.elf # 期望 3e00 -strings -a web-file-mgr-v1.9.1.elf | grep -m1 '^v1\.' +ls -lh web-file-mgr-v1.9.2.elf +sha256sum web-file-mgr-v1.9.2.elf +od -An -tx2 -j18 -N2 web-file-mgr-v1.9.2.elf # 期望 3e00 +strings -a web-file-mgr-v1.9.2.elf | grep -m1 '^v1\.' ``` ⚠️ `od -An -tx2` 打印的是**小端 short 的值**（`003e`），不是字节序（`3e00`）。脚本里比对用 `$((16#$EM))` 转数值（**62 = x86-64 ✅ / 183 = aarch64 ❌**）。 -⚠️ **sha256 只在源码不变时可复现**。核对版本请用 `strings ... | grep ^v1.9`，比 sha256 直观且抗噪。 +✅ **构建已实测可复现**（2026-09-20）：同一源码树两次构建 sha256 完全相同；把两处版本字面量回退成 `v1.9.1` 后重构，产物与已发布的 v1.9.1 ELF **逐字节一致**。所以 sha256 可以作为交付指纹用 —— 但它对任何源码改动都会全变（改一个字符串常量会令链接器重排 `.rodata` 字符串池，牵动 `.text` 里所有 RIP 相对位移，原始 diff 会放大到 5 万字节以上，属正常现象，别误判成"代码改了"）。核对版本仍推荐 `strings ... | grep '^v1\.'`，最直观。 ### 4.3 构建坑（已修，改 Makefile 前必读） @@ -254,7 +256,7 @@ ELF 已构建，但需装 PS5 实测： 2. **加密 7z（7zAES）** 真机解压 3. 160GB / 9.5 万文件大 ZIP 4. 分卷 RAR 进度条实时走动 -5. UI 右下角版本号显示 `v1.9.1` +5. UI 右下角版本号显示 `v1.9.2`（`/api/version` 与兜底字面量应一致） ### 可选项（非阻塞） diff --git a/Makefile b/Makefile index cc50ed4..c4e0b69 100644 --- a/Makefile +++ b/Makefile @@ -17,7 +17,7 @@ endif # notification and `--version` print it) AND into the output filename, so a # stale value silently mislabels everything. Override per-build with: # make VERSION_TAG=v1.9.2 -VERSION_TAG ?= v1.9.1 +VERSION_TAG ?= v1.9.2 TITLE_ID := FMGR88888 PYTHON ?= python3 STRIP ?= $(PS5_PAYLOAD_SDK)/bin/prospero-strip diff --git a/README.md b/README.md index 458d487..b9d38c3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ > Homebrew HTTP file manager for jailbroken PS5 consoles. Browse, edit, upload, download and extract ZIPs through any browser on the same network — single self-contained ELF payload, no external services, no telemetry. -**Version:** v1.9.1 · **Title ID:** `FMGR88888` · **License:** GPLv3+ · **Target:** `x86_64-sie-ps5` +**Version:** v1.9.2 · **Title ID:** `FMGR88888` · **License:** GPLv3+ · **Target:** `x86_64-sie-ps5` --- @@ -16,6 +16,13 @@ A payload ELF that runs an HTTP file manager inside a jailbroken PS5. Open `http The same source tree builds a Linux binary for development and a PS5 payload ELF for deployment — see `make linux` below. +## What's new in v1.9.2 + +Version-string-only re-release. The `v1.9.1` tag sat four commits behind the +tree that produced its binary, so the tag could not rebuild the published +artifact; v1.9.2 is cut from the right commit. It is functionally identical to +the v1.9.1 binary — the only change is the baked-in version string. + ## What's new in v1.9 - **RAR engine replaced with the official rarlab UnRAR 7.20.1** diff --git a/README.zh-CN.md b/README.zh-CN.md index e352db5..3e5427c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,7 +6,7 @@ > 面向已越狱 PS5 主机的自制 HTTP 文件管理器。通过同一局域网内的任意浏览器（包括 PS5 自带浏览器）即可浏览、编辑、上传、下载并解压 ZIP / RAR / 7z 压缩包——单个自包含 ELF 载荷，无外部服务、无遥测上报。 -**版本：** v1.9.1 · **标题 ID：** `FMGR88888` · **许可证：** GPLv3+ · **目标平台：** `x86_64-sie-ps5` +**版本：** v1.9.2 · **标题 ID：** `FMGR88888` · **许可证：** GPLv3+ · **目标平台：** `x86_64-sie-ps5` --- @@ -16,7 +16,9 @@ 同一套源码树可构建出供开发用的 Linux 二进制，以及供部署的 PS5 载荷 ELF——见下方 `make linux`。 -## v1.9.1 新增内容 +## v1.9.2 与 v1.9.1 新增内容 + +> **v1.9.2 与 v1.9.1 的功能完全相同，只换了内嵌版本号。** 原因是原先的 `v1.9.1` tag 指在产出发布二进制的提交**之前 4 个提交**，tag 与产物对不上（clone 该 tag 无法重建出发布的那份 ELF）；v1.9.2 重新从产出该二进制的提交上打，使 tag = 源码 = 二进制。 - **7z 解压引擎**（`src/sevenz_extract.{c,h}`）：自研解码子集 + 拉式 codec 链（`src/sevenz_chain.c`，覆盖 LZMA2 / BCJ2 等），由 `src/extract.c` 按扩展名分派，与 ZIP / RAR 共用同一套三阶段模型与限额档位。`.7z` 文件在文件列表中同样带「解压」按钮。 - **7zAES 内容解密**（AES-256-CBC）：引擎层可解密带密码的 7z 内容；密码输入 UI / API 通道尚未接入，目前加密归档仍被拒绝。 diff --git a/assets/main.js b/assets/main.js index ee9c644..de9da4c 100644 --- a/assets/main.js +++ b/assets/main.js @@ -39,7 +39,7 @@ let L = {}; // /api/version, which reports the build's VERSION_TAG -- see loadVersion(). // Keeping a literal here used to be the only source, and it inevitably // drifted (the footer said "v1.9" throughout the v1.9.1 release). -const APP_VERSION_FALLBACK = "v1.9.1"; +const APP_VERSION_FALLBACK = "v1.9.2"; const LAST_PATH_KEY = "ps5-web-file-mgr:last-path"; const SORT_KEY = "ps5-web-file-mgr:list-sort"; const LOADING_DISPLAY_DELAY = 250; diff --git a/docs/FORUM-POST-v1.9.1.md b/docs/FORUM-POST-v1.9.2.md similarity index 80% rename from docs/FORUM-POST-v1.9.1.md rename to docs/FORUM-POST-v1.9.2.md index 8e7b020..e873195 100644 --- a/docs/FORUM-POST-v1.9.1.md +++ b/docs/FORUM-POST-v1.9.2.md @@ -1,4 +1,4 @@ -# PS5 网页文件管理器 v1.9.1（自编译版）发布 + 与上个版本对比 + 使用简介 +# PS5 网页文件管理器 v1.9.2（自编译版）发布 + 与上游原版对比 + 使用简介 > 这是我自己维护的一个分支版本，基于 owendswang 的 `ps5-web-file-manager` 二次开发。本文讲三件事：**它和上游原版差在哪、我这个版本强在哪、怎么用**。 @@ -8,22 +8,25 @@ 一个在已越狱 PS5 上跑的 HTTP 文件管理器（单文件 ELF 载荷）。局域网内任意浏览器（含 PS5 自带浏览器）打开 `http://:8888/` 就能管理外接 USB 和内置存储：浏览、改权限、上传、下载、原地编辑文本、装/预览 PKG、看图，以及**解压 ZIP / RAR / 7z**。 -- 版本：**v1.9.1** +- 版本：**v1.9.2** - 标题 ID：`FMGR88888` - 许可证：GPLv3+ - 目标平台：`x86_64-sie-ps5`（Zen 2，不是 ARM） +> **关于版本号**：v1.9.2 与 v1.9.1 功能完全相同，只更新了内嵌版本号。原先的 `v1.9.1` tag 打在了产出发布二进制的提交**之前 4 个提交**，导致"tag 对应的源码"重建不出发布的那份 ELF；v1.9.2 重新从正确提交上打，使 **tag = 源码 = 二进制**。 + --- ## 二、和上游原版（owendswang v1.8）对比 先说结论：**不是同一条路线，各有胜负手。** -| 维度 | 上游 owendswang v1.8 | 我的 v1.9.1 | +| 维度 | 上游 owendswang v1.8 | 我的 v1.9.2 | |---|---|---| | 一句话定位 | 把 7-Zip 本体做成**外部 helper 进程**，靠 IPC 调用 | **自研 / vendor 解码库，全部内嵌同一进程** | | 支持的压缩格式 | **30 种**（zip/7z/rar/tar/gz/xz/zst/bz2/cab/arj…） | **3 种**：ZIP / RAR / 7z | | 部署方式 | **两个文件**，helper 必须放 `/data/wfm/` 指定路径 | **单个 ELF，零外部依赖** | +| 载荷体积 | 主程序 + 7-Zip 本体（两份） | **850 KiB 单文件**（含 ZIP/RAR/7z 三个解码引擎） | | 防压缩炸弹（zip bomb） | ❌ 无 | ✅ 压缩比上限 + 1 GiB 下限豁免 | | 磁盘写满保护 | ❌ 无 | ✅ 解压前按实际剩余空间 `statvfs` 预检 | | 路径穿越防护 | ❌ 无（交给 7-Zip） | ✅ 有专项测试 | @@ -52,7 +55,7 @@ --- -## 三、v1.9.1 我做了哪些具体改进 +## 三、v1.9.2 / v1.9.1 我做了哪些具体改进 - **7z 解压引擎**：自研解码子集 + 拉式 codec 链（`sevenz_chain.c`），支持 LZMA2 / LZMA / BCJ2，覆盖 SDK `SzArEx` 装不下的 5-coder 文件夹。 - **分卷（多卷）支持**：三种格式都支持分卷，但命名约定不同（引擎按文件名自动识别，打开首个分卷即可）： @@ -64,6 +67,8 @@ - 纯 LZMA2 多线程解码（8 线程）≈ **1.37×** - 移除 ZIP 逐条目 fsync，减少写盘开销 - **RAR 升级到官方 UnRAR 7.20.1**：RAR5「v6」+ 多卷可用（v1.8 时代在 WinRAR 6/7 文件上报"归档损坏"的问题已消失）。 +- **载荷瘦了 15.8%**（1,034,328 → 870,488 字节）：链接期去掉了 libc++abi 里一段**永远不会执行**的 C++ 名字还原器（只服务于"未捕获异常打印类型名"这条路径），再加上相同机器码折叠。**不损失任何功能，也不影响解压速度**。 +- **上传更顺手**：点上传按钮直接选文件或文件夹（不再弹二级菜单）；支持**把文件/文件夹直接拖到页面上**上传。 - **中英双语界面** + 项目主页右上角可切换语言说明。 ### 已知缺口（诚实列出） @@ -79,7 +84,7 @@ PS5 上先运行一个 ELF 加载器（端口 `9021` 常见），然后在 PC ```sh # 把 PS5_IP 换成你主机实际 IP -nc -q0 9021 < web-file-mgr-v1.9.1.elf +nc -q0 9021 < web-file-mgr-v1.9.2.elf ``` PS5 屏幕会弹通知，显示实际监听端口（默认 `8888`）。首次运行还会在主屏 Media 分类装一个「PS5 Web File Manager」快捷方式。 @@ -95,7 +100,7 @@ PS5 自带浏览器也能开。 ### 3. 日常操作 - **浏览/管理**：列文件、排序、改权限、复制/移动/重命名/删除、原地编辑文本（≤1 MiB 的 txt/json/js/c/h…）。 -- **上传下载**：单文件或文件夹树上传；文件夹/多选以 `.tar` 流式下载。 +- **上传下载**：单文件或文件夹树上传（也可直接拖拽到页面）；文件夹/多选以 `.tar` 流式下载。 - **解压**：在文件列表里点 ZIP / RAR / 7z 的「解压」按钮即可。大档案（>480 GiB）会弹确认提示，确认后用更宽松的限额解压。 - **冲突策略**：同名文件默认拒绝覆盖；需要时可选覆盖或合并。 @@ -103,23 +108,25 @@ PS5 自带浏览器也能开。 ## 五、下载与校验 -> 请在 GitHub Release 页面下载：`https://github.com/lishersong/ps5-web-file-manager/releases/tag/v1.9.1` +> 请在 GitHub Release 页面下载：`https://github.com/lishersong/ps5-web-file-manager/releases/tag/v1.9.2` -发布文件：`web-file-mgr-v1.9.1.elf` +发布文件：`web-file-mgr-v1.9.2.elf` | 项目 | 值 | |---|---| | 大小 | 870,488 字节（约 850 KiB） | -| sha256 | `24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07` | +| sha256 | `177e90fecf93a0251e83f67884fba4551051be248330b0d70fda8ea732f88e84` | | 文件类型 | ELF 64-bit LSB，x86-64（e_machine `0x003e`，即 PS5 目标三元组） | 下载后建议先核对 sha256 再发到主机： ```sh -sha256sum web-file-mgr-v1.9.1.elf -# 应等于 24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07 +sha256sum web-file-mgr-v1.9.2.elf +# 应等于 177e90fecf93a0251e83f67884fba4551051be248330b0d70fda8ea732f88e84 ``` +> 本次构建是**可复现**的：同一份源码树连编两次 sha256 完全相同。所以这个哈希可以放心当交付指纹用。 + --- ## 六、免责声明 diff --git a/docs/SIZE-OPTIMIZATION.md b/docs/SIZE-OPTIMIZATION.md index 002935e..5a70071 100644 --- a/docs/SIZE-OPTIMIZATION.md +++ b/docs/SIZE-OPTIMIZATION.md @@ -140,7 +140,7 @@ COMMON_SRCS := src/main.c src/websrv.c ... src/sevenz_mt.c src/demangle_stub.c | 检查项 | 结果 | |---|---| -| `__cxa_demangle` 本体大小 | **14 B**（我们的桩） | +| `__cxa_demangle` 本体大小 | **11 B**（我们的桩；原 demangler 入口是 1 701 B） | | `itanium_demangle::*` 符号残留 | **0** | | `__cxa_throw` | 存在 | | `__cxa_begin_catch` / `__cxa_end_catch` | 存在 | @@ -202,3 +202,150 @@ LDFLAGS += -Wl,-z,pack-relative-relocs > 本次改动只动链接期（新增一个 TU + 一个 lld 参数），未触碰任何解压逻辑， > 因此 163 checks 的预期是"逐条不变"。 + +**结果（2026-09-20）**：163 checks（ZIP 108 + RAR 27 + 7z 28）**0 失败**， +`aeshe` 仍是已知的 `-mhe=on` 缺口。README / CHANGELOG / HANDOVER / 论坛帖里的 +产物指纹已同步为 870,488 B · sha256 `177e90fe…8e84`。 + +--- + +## 附录 A：v1.9.2 产物一致性验证（2026-09-20） + +v1.9.2 是一次"只改内嵌版本号"的重发（原 `v1.9.1` tag 落后产出发布二进制的提交 +4 个提交）。为确认这次重发**真的**只动了版本号，在 WSL 里做了下面的验证。 + +### A.1 复现性实验（决定性证据） + +当前工作区相对 `HEAD` 只有两处改动：`Makefile` 的 `VERSION_TAG` 与 +`assets/main.js` 的 `APP_VERSION_FALLBACK`。把这两处用 `sed` 回退成 `v1.9.1` +后重新构建： + +| 构建 | sha256 | +|---|---| +| 已发布的 v1.9.1 ELF | `24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07` | +| 回退后重建的产物 | `24392aff6ddcca4dc0ea969cce356bd693ac52efe8a117d61ee1c814aa43cd07` | + +**逐字节相同。** 再恢复 `v1.9.2` 重构，sha256 也精确回到 `177e90fe…8e84`。 + +→ 构建是**确定性**的，因此 v1.9.2 与 v1.9.1 的全部差异就等于那两处版本字面量。 +复现脚本：`.build/_repro.sh`。 + +### A.2 为什么原始字节 diff 有 5.5 万字节 —— 别被吓到 + +`cmp` 两个 ELF 会看到 **55,280 字节不同（6.35%）**，但这是链接器字符串池重排的 +副作用，不是代码变了： + +| section | 差异字节 | 占该 section | +|---|---:|---:| +| `.rodata` | 53,496 | 34.8% | +| `.text` | 1,543 | 0.3% | +| `.rela.dyn` | 241 | 0.9% | + +而**每个 section 的尺寸完全相同**（`.text` 538,336 = 538,336），段数也都 17 个。 + +机制：`.rodata` 里 7 字节的 `"v1.9.1\0"` 被换成 `"v1.9.2\0"` 后落点变了，其后 +所有字符串整体平移 7 字节 → 指向它们的 `lea rdi,[rip+disp]` 位移和 `.rela.dyn` +重定位加数全部跟着变。 + +两条量化证据： + +| 检查 | 结果 | +|---|---| +| 指令**助记符**序列（`objdump -d --no-show-raw-insn` 只取 mnemonic） | 141,780 条 vs 141,780 条，**完全一致** —— 没有任何指令被增删改 | +| `.text` 差异字节的增量分布 | 1,543 个里 **1,506 个恰好是 −7**（正是那个 7 字节平移）；`.rela.dyn` 241/241 个 8 字节字段减 7 | +| 嵌入的 gzip 资产 | 6 个成员，5 个逐字节相同，唯一不同的是 `main.js`，且差异 = `APP_VERSION_FALLBACK` 那一行 | + +脚本：`.build/_diffmap.py`、`.build/_operandcheck.sh`、`.build/_fieldcheck.py`、 +`.build/_verify_v192b.py`、`.build/_seccmp.py`。 + +### A.3 源码层的约束 + +`VERSION_TAG` 在源码里**只出现在字符串上下文**： + +``` +src/version.c:29 json_escape(&b, VERSION_TAG); +src/main.c:127 printf("version: %s\n", VERSION_TAG); +src/main.c:147 notify_user("Web File Manager\nVersion: %s\nPort: %u", VERSION_TAG, port); +``` + +没有任何算术、比较或分支依赖它，因此改版本号在语言层面就不可能改变控制流。 + +--- + +## 附录 B：瘦身逐符号账目 + +在 **同一个 Makefile / 同一个 `VERSION_TAG`（v1.9.2）** 下重建三个变体，差异只落在 +"有没有桩"和"有没有 ICF"这两处，因此是干净的 A/B/C 对照。 + +| 变体 | 内容 | stripped | unstripped | +|---|---|---:|---:| +| `base` | 无桩、无 ICF（瘦身前） | **1,034,328** | 1,222,752 | +| `nicf` | 有桩、无 ICF | **886,872** | 1,010,208 | +| `new` | 有桩 + `--icf=all`（发布态） | **870,488** | 993,824 | + +拆分：桩贡献 **−147,456 B**，ICF 再贡献 **−16,384 B**，合计 **−163,840 B（−15.8%）**。 + +### B.1 section 位移（base → new） + +| section | base | new | 差值 | +|---|---:|---:|---:| +| `.text` | 637,616 | 538,336 | −99,280 | +| `.rela.dyn` | 54,816 | 25,800 | −29,016 | +| `.eh_frame` | 58,904 | 43,972 | −14,932 | +| `.data.rel.ro` | 20,192 | 9,328 | −10,864 | +| `.rodata` | 162,016 | 153,664 | −8,352 | +| `.eh_frame_hdr` | 12,540 | 9,108 | −3,432 | +| `.gcc_except_table` | 8,500 | 7,364 | −1,136 | +| `.dynsym` / `.dynstr` / `.got` | — | — | −24 / −9 / −8 | + +### B.2 符号集合差 + +| 项 | 数量 | +|---|---:| +| base 定义符号 | 2,657 | +| new 定义符号 | 2,029 | +| base → new **消失** | **628** | +| base → new **新增** | **0** | + +628 个消失符号的构成： + +- `itanium_demangle::*` —— **607** +- `GCC_except_table*` —— **21**（上面那批代码自己的异常表标签，不是独立函数） + +### B.3 桩本体与 demangler 符号 + +| 变体 | `__cxa_demangle` 符号大小 | `itanium_demangle` 符号数 | +|---|---:|---:| +| base | 1,701 B（真身） | 607 | +| nicf | **11 B**（我们的桩） | **0** | +| new | **11 B** | **0** | + +异常机制在所有三个变体里都完好：`__cxa_throw` / `__cxa_begin_catch` / +`__cxa_end_catch` / `_Unwind_Resume` / `__gxx_personality_v0` / +`__cxa_allocate_exception` / `__cxa_free_exception` 各 1 个，无变化。 + +自有 `src/` 关键符号（`ctx_fail` / `rarx_fail` / `szx_fail` / `fnv1a` / +`nameset_init` / `remove_tree` / `ensure_parent_dirs` / `zipx_volume_detect` / +`sevenz_extract` / `rar_extract` / `filemgr_api_request` 等）base 与 new 数量一致。 + +### B.4 ICF 折叠了什么 + +**符号数 2,029 → 2,029，一个没少** —— ICF 是"合并"不是"删除"。共 **76 个折叠组**， +全部含具名符号。典型几类： + +- 我们自己的同码副本：`ctx_fail == rarx_fail == szx_fail`、 + `nameset_init == szx_nameset_init`、`fnv1a == rarx_fnv1a == szx_fnv1a`、 + `remove_tree == szx_remove_tree` +- C++ 的 `C1 == C2` / `D1 == D2` 构造析构对（编译器为同一函数生成两个 ABI 入口）： + `_ZN10CmdExtractC1EP11CommandData == ...C2...`、`_ZN4FileD1Ev == _ZN4FileD2Ev` 等 +- 只读常量表：`Sbox == _ZL1S`（AES 表在 `rijndael.cpp` 与 `Aes.c` 各一份）、 + `SHA256_K_ARRAY == _ZL1K`、`PPMD7_kExpEscape == _ZL9ExpEscape` +- RARDLL 模式下被置空的 UI 函数、`mz_stream_read_int64 == read_uint64`、 + libunwind 的 `__unw_* == unw_*`、`__unw_resume == unw_resume` + +**风险提示**：`--icf=all` 是 LLD 的激进模式，**不做地址敏感性检查** +（`--icf=safe` 才会读 `.llvm_addrsig` 跳过被取地址的函数）。逐组核对下来这 76 组 +都是同码副本、没有"比较函数/常量表地址"的用法 —— 但这是人工判断，不是编译器给的 +保证。想绝对保守就把 `--icf=all` 换成 `--icf=safe`，代价是少省几 KB。 + +复现脚本：`.build/_whatremoved_v192.sh`（一次跑完三个变体 + 全部核对）。 diff --git a/tests/bench_driver.py b/tests/bench_driver.py index 994f0a0..2c1b56c 100644 --- a/tests/bench_driver.py +++ b/tests/bench_driver.py @@ -1,14 +1,20 @@ #!/usr/bin/env python3 """Performance baseline for the extraction engines. - python tests/bench_driver.py # 82 MiB fixture, fast - python tests/bench_driver.py --big # 320 MiB fixture, accurate + python tests/bench_driver.py # 82 MiB fixture, fast + python tests/bench_driver.py --big # 320 MiB fixture, accurate + python tests/bench_driver.py --format zip + python tests/bench_driver.py --format rar python tests/bench_driver.py --runs 5 -Times our facades (src/zip_extract.c, src/sevenz_extract.c) against the two -external references that matter: the vendored SDK's own SzArEx path, and the -official 7-Zip binary -- the latter being what upstream v1.8 gets by shelling -out to a helper, so it doubles as the "how fast could we be" ceiling. +Times our facades (src/zip_extract.c, src/rar_extract.c, +src/sevenz_extract.c) against the external references that matter: the +vendored SDK's own SzArEx path, and the official 7-Zip binary -- the latter +being what upstream v1.8 gets by shelling out to a helper, so it doubles as +the "how fast could we be" ceiling. + +The three formats are packed from one shared payload, so the rows are +comparable across formats and not just within one engine. Why Python drives the measurement: in this sandbox a single `date` costs ~350 ms, so the usual `t0=$(date)` / `t1=$(date)` pair adds ~700 ms of pure @@ -22,6 +28,7 @@ spawn tax entirely -- that is the most trustworthy figure for our side. """ import argparse +import glob import os import re import subprocess @@ -35,6 +42,14 @@ HT_BUILD = os.path.join(REPO, ".build", "host-test") SEVENZ_BIN = os.path.join(REPO, ".build", "7zdl", "extra", "x64", "7za.exe") BENCH_BIN = os.path.join(BUILD, "bench_extract.exe") SDK_BIN = os.path.join(SZ_BUILD, "sevenz_e2e.exe") +# RAR is write-only in WinRAR (7-Zip can read the format but not create it), +# so a RAR fixture needs rar.exe. Missing means the rar fixture is skipped, +# never that the run fails. +RAR_BIN = next((p for p in ( + os.environ.get("WFM_RAR"), + r"C:\Program Files\WinRAR\rar.exe", + r"C:\Program Files (x86)\WinRAR\rar.exe", + "/usr/bin/rar", "/usr/local/bin/rar") if p and os.path.exists(p)), None) # Object lists mirror what tests/run-sevenz-tests.sh and tests/run-tests.sh # build; those scripts must have run once before this can link. @@ -60,12 +75,15 @@ def build_bench(): if os.path.exists(BENCH_BIN): return True + # The RAR engine is C++ (vendored UnRAR), so bench_extract.c is compiled + # with gcc and the link goes through g++. + unrar = sorted(glob.glob(os.path.join(HT_BUILD, "unrar7_*.o"))) needed = (objs(SZ_BUILD, ["sevenz_extract", "sevenz_chain", "sevenz_mt", "sevenz_volstream", "zipx_common", "zipx_volume"]) - + objs(HT_BUILD, ["zip_extract", "zipx_volstream"]) + + objs(HT_BUILD, ["zip_extract", "zipx_volstream", "rar_extract"]) + objs(HT_BUILD, ZLIB) + objs(HT_BUILD, MINIZIP) - + objs(SZ_BUILD, VENDOR_7Z)) + + objs(SZ_BUILD, VENDOR_7Z) + unrar) missing = [p for p in needed if not os.path.exists(p)] if missing: print("missing engine objects, run these first:") @@ -74,17 +92,26 @@ def build_bench(): print("first missing: %s" % missing[0]) return False - cmd = ["gcc", "-O2", "-w", - "-I" + os.path.join(REPO, "third_party", "7z"), - "-I" + os.path.join(REPO, "third_party", "minizip-ng", "include"), - "-I" + os.path.join(REPO, "third_party", "zlib", "include"), - "-I" + os.path.join(REPO, "src"), - "-I" + os.path.join(REPO, "tests", "compat"), - "-include", os.path.join(REPO, "tests", "posix_compat.h"), - "-o", BENCH_BIN, os.path.join(REPO, "tests", "bench_extract.c")] - cmd += needed + EXTRA_LIBS + includes = ["-I" + os.path.join(REPO, "third_party", "7z"), + "-I" + os.path.join(REPO, "third_party", "minizip-ng", "include"), + "-I" + os.path.join(REPO, "third_party", "zlib", "include"), + "-I" + os.path.join(REPO, "src"), + "-I" + os.path.join(REPO, "tests", "compat"), + "-include", os.path.join(REPO, "tests", "posix_compat.h")] + obj = os.path.join(BUILD, "bench_extract.o") print("== compiling bench harness ==") - if subprocess.run(cmd, cwd=REPO).returncode != 0: + if subprocess.run(["gcc", "-O2", "-w"] + includes + + ["-c", "-o", obj, + os.path.join(REPO, "tests", "bench_extract.c")], + cwd=REPO).returncode != 0: + return False + + libs = list(EXTRA_LIBS) + if os.name == "nt": + # Windows unrar system.cpp references SetSuspendState (PowrProf). + libs.append("-lpowrprof") + if subprocess.run(["g++", "-O2", "-o", BENCH_BIN, obj] + needed + libs, + cwd=REPO).returncode != 0: return False return True @@ -138,23 +165,48 @@ def make_fixture(big, fmt): out.write(fh.read()) print("== creating fixture (first run only) ==") - if fmt == "7z": - add = ["a", "-t7z", "-m0=lzma2", "-mx=5", "-ms=on"] + if fmt == "rar": + # RAR needs WinRAR's rar.exe; 7-Zip cannot write the format. + rar = RAR_BIN + if not rar: + print("WinRAR (rar.exe) not found; cannot create a RAR fixture") + return None + add = ["a", "-m3", "-idq", "-ep1"] + subprocess.run([rar] + add + [archive, payload], cwd=REPO, + stdout=subprocess.DEVNULL) else: - add = ["a", "-tzip", "-mx=5", "-mm=Deflate"] - subprocess.run([SEVENZ_BIN] + add + [archive, payload], cwd=REPO, - stdout=subprocess.DEVNULL) + if fmt == "7z": + add = ["a", "-t7z", "-m0=lzma2", "-mx=5", "-ms=on"] + else: + add = ["a", "-tzip", "-mx=5", "-mm=Deflate"] + subprocess.run([SEVENZ_BIN] + add + [archive, payload], cwd=REPO, + stdout=subprocess.DEVNULL) return archive def sample(argv, runs, workdir): - """Run argv `runs` times; return (best wall ms, best in-process ms|None).""" + """Run argv `runs` times; return (best wall ms, best in-process ms|None). + + Every run writes to its own path (`workdir0`, `workdir1`, ...) that does + not exist yet. Reusing one output directory is not an option: publishing + into a tree left by the previous run charges the rename step for the + collision, and that alone moved the same archive from 0.77 s to 1.28 s. + The placeholder `{out}` in argv marks where the run directory goes. + + Nothing is deleted between runs either -- clearing these trees is a bulk + delete this host blocks -- so `.build/bench/W*` does accumulate and is + worth clearing by hand now and then. + """ best = None internal = None - for _ in range(runs): - os.makedirs(workdir, exist_ok=True) + for i in range(runs): + # Forward slashes on purpose: the engines derive the destination's + # parent with a '/' scan (they only ever see POSIX paths on the PS5), + # so a Windows-style relative or absolute path is rejected outright. + run_dir = ("%s%d" % (workdir, i)).replace("\\", "/") + cmd = [arg.replace("{out}", run_dir) for arg in argv] started = time.perf_counter() - proc = subprocess.run(argv, stdout=subprocess.PIPE, + proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=REPO, timeout=1800) elapsed = (time.perf_counter() - started) * 1000.0 if proc.returncode != 0: @@ -175,7 +227,7 @@ def main(): ap.add_argument("--big", action="store_true", help="4x fixture (~320 MiB) for accurate ratios") ap.add_argument("--runs", type=int, default=3) - ap.add_argument("--format", choices=["7z", "zip"], default="7z", + ap.add_argument("--format", choices=["7z", "zip", "rar"], default="7z", help="which engine to benchmark (default 7z)") args = ap.parse_args() @@ -199,29 +251,42 @@ def main(): print() rows = [] - wall, inner = sample([BENCH_BIN, archive, ".build/bench/W7"], args.runs, + wall, inner = sample([BENCH_BIN, archive, "{out}"], args.runs, os.path.join(BUILD, "W7")) if wall: rows.append(("ours / " + args.format, wall, inner)) if args.format == "7z": - wall, _ = sample([SDK_BIN, archive, ".build/bench/W8"], args.runs, + wall, _ = sample([SDK_BIN, archive, "{out}"], args.runs, os.path.join(BUILD, "W8")) if wall: rows.append(("sdk SzArEx", wall, None)) variants = (("7za 1 thread", ["-mmt=off"]), ("7za 8 threads", ["-mmt=8"]), ("7za all cores", [])) + elif args.format == "rar": + # 7-Zip reads RAR, so it is a valid cross-check on the same archive. + variants = (("7za 1 thread", ["-mmt=off"]), + ("7za all cores", [])) else: variants = (("7za 1 thread", ["-mmt=off"]),) for label, extra in variants: wall, _ = sample([SEVENZ_BIN, "x", "-y", "-aoa"] + extra - + ["-o.build/bench/W9", archive], args.runs, + + ["-o{out}", archive], args.runs, os.path.join(BUILD, "W9")) if wall: rows.append((label, wall, None)) + if args.format == "rar" and RAR_BIN: + # rar.exe treats the trailing separator as "this is the target + # directory"; without it the path is parsed as a file mask and the + # command reports that there is nothing to extract. + wall, _ = sample([RAR_BIN, "x", "-y", "-o+", archive, "{out}/"], + args.runs, os.path.join(BUILD, "W10")) + if wall: + rows.append(("winrar x", wall, None)) + print(" %-18s %10s %12s" % ("configuration", "external", "internal")) for label, wall, inner in rows: print(" %-18s %9.0f ms %12s" diff --git a/tests/bench_extract.c b/tests/bench_extract.c index 4e70b49..1a5ddde 100644 --- a/tests/bench_extract.c +++ b/tests/bench_extract.c @@ -1,4 +1,4 @@ -/* Wall-clock benchmark for the extraction paths, ZIP and 7z. +/* Wall-clock benchmark for the extraction paths, ZIP, RAR and 7z. * * bench_extract [password] * @@ -10,7 +10,7 @@ * Keeping it in-tree matters because "is our engine fast?" is a question that * will come up again, and the answer should be a command anyone can rerun * rather than a number somebody remembers. The format is picked from the - * suffix so the same binary covers both engines. + * suffix so the same binary covers all three engines. */ #include @@ -20,6 +20,10 @@ #include "sevenz_extract.h" +#ifndef BENCH_NO_RAR +#include "rar_extract.h" +#endif + static int has_suffix(const char *path, const char *suffix) { size_t path_len; @@ -76,6 +80,12 @@ main(int argc, char **argv) { has_suffix(archive, ".z01")) { format = "zip"; } +#endif +#ifndef BENCH_NO_RAR + else if(has_suffix(archive, ".rar") || has_suffix(archive, ".part1.rar") || + has_suffix(archive, ".r00")) { + format = "rar"; + } #endif else { fprintf(stderr, "unsupported benchmark format: %s\n", archive); @@ -84,6 +94,12 @@ main(int argc, char **argv) { memset(&result, 0, sizeof(result)); started = now_seconds(); +#ifndef BENCH_NO_RAR + if(!strcmp(format, "rar")) { + status = rar_extract(archive, out_dir, ZIPX_CONFLICT_OVERWRITE, + zipx_default_limits(), NULL, NULL, NULL, &result); + } else +#endif #ifndef BENCH_SEVENZ_ONLY if(!strcmp(format, "zip")) { status = zipx_extract(archive, out_dir, ZIPX_CONFLICT_OVERWRITE, @@ -105,7 +121,7 @@ main(int argc, char **argv) { printf("unpacked : %.1f MiB\n", mebibytes); printf("wall : %.3f s\n", elapsed); if(elapsed > 0.0) { - printf("through : %.1f MiB/s (single thread)\n", mebibytes / elapsed); + printf("through : %.1f MiB/s\n", mebibytes / elapsed); } if(status != ZIPX_OK) { printf("detail : %s\n", result.detail[0] ? result.detail : "(none)"); diff --git a/tests/bench_formats.py b/tests/bench_formats.py new file mode 100644 index 0000000..7db59fe --- /dev/null +++ b/tests/bench_formats.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Cross-format extraction benchmark: same payload, three containers. + + python tests/bench_formats.py # one 329 MiB file, 3 runs + python tests/bench_formats.py --source .build/bench/payload_bin.bin \\ + --stem bin --runs 5 # 39 MiB of machine code + python tests/bench_formats.py --source .build/bench/manyfiles_src --stem mf + +bench_driver.py answers "how do we compare with 7-Zip for one format"; this +answers "which container should a user expect to unpack fastest", which is a +different question and needs the three archives to hold the same bytes. + +Method notes that took a while to get right, so they are pinned here: + + * Fresh output directory per run. Publishing on top of the previous run's + tree took the same 7z archive from 0.77 s to 1.28 s. + * Forward slashes in every path. The engines derive the destination parent + with a '/' scan, so "C:\\...\\W0" is rejected as an invalid destination. + * Best of N, because a single run varies widely on this host: extraction + creates hundreds of MiB that the on-access scanner inspects and the page + cache has to write back, and neither is under our control. + * Corpus matters as much as container. Deflate decodes faster than LZMA2 on + ordinary data, but on highly repetitive text LZMA2 finds long matches + where deflate only finds 32 KiB ones, and the order flips. Run the same + archive set on more than one source before concluding anything. + +Packed sizes are printed next to the times on purpose: a container that packs +5x smaller also reads 5x less from the card, which is why the ranking on the +PS5 -- where storage is the slow part -- can differ from the ranking here. +""" + +import argparse +import os +import re +import subprocess +import sys +import time + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +BUILD = os.path.join(REPO, ".build", "bench") +BENCH_BIN = os.path.join(BUILD, "bench_extract.exe") +SEVENZ_BIN = os.path.join(REPO, ".build", "7zdl", "extra", "x64", "7za.exe") +RAR_BIN = next((p for p in ( + os.environ.get("WFM_RAR"), + r"C:\Program Files\WinRAR\rar.exe", + r"C:\Program Files (x86)\WinRAR\rar.exe", + "/usr/bin/rar", "/usr/local/bin/rar") if p and os.path.exists(p)), None) +DEFAULT_SOURCE = os.path.join(BUILD, "payload4.bin") + +# Nominal level 5 in both packers, which is the GUI default of each: +# 7-Zip -mx=5, WinRAR -m3 ("Normal"). They are not equivalent amounts of +# work -- LZMA2 at level 5 is a far stronger compressor than deflate at 5 -- +# but they are what a user who never opens the advanced panel ends up with. +FORMATS = ("zip", "7z", "rar") + + +def pack(stem, payload): + """Create stem.{zip,7z,rar} from payload if they are not there yet.""" + made = [] + for fmt in FORMATS: + archive = os.path.join(BUILD, "%s.%s" % (stem, fmt)) + if os.path.exists(archive): + continue + if fmt == "rar": + if not RAR_BIN: + print("skip %s: WinRAR (rar.exe) not found" % fmt) + continue + cmd = [RAR_BIN, "a", "-m3", "-idq"] + if os.path.isfile(payload): + cmd.append("-ep1") + cmd += [archive, payload] + else: + cmd = [SEVENZ_BIN, "a", "-t" + fmt] + ( + ["-m0=lzma2", "-mx=5", "-ms=on"] if fmt == "7z" + else ["-mx=5", "-mm=Deflate"]) + [archive, payload] + print("== packing %s ==" % os.path.basename(archive)) + if subprocess.run(cmd, cwd=REPO, stdout=subprocess.DEVNULL).returncode: + print("packing failed") + return None + made.append(archive) + return [os.path.join(BUILD, "%s.%s" % (stem, f)) for f in FORMATS] + + +def time_archive(archive, runs): + """Best in-process wall time over `runs` fresh-directory extractions.""" + tag = os.path.splitext(os.path.basename(archive))[0] + times = [] + unpacked = entries = 0 + for i in range(runs): + # A path that does not exist yet: the engines publish with a rename, + # and a rename into a tree that already has the file costs extra. + out = os.path.join(BUILD, "F-%s-%d" % (tag, i)).replace("\\", "/") + started = time.perf_counter() + proc = subprocess.run([BENCH_BIN, archive, out], cwd=REPO, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, timeout=1800) + external = time.perf_counter() - started + if proc.returncode != 0: + sys.stdout.write(proc.stdout.decode("utf-8", "replace")[:400]) + return None + text = proc.stdout.decode("utf-8", "replace") + match = re.search(r"wall\s*:\s*([0-9.]+)", text) + if not match: + return None + times.append(float(match.group(1))) + unpacked = float(re.search(r"unpacked\s*:\s*([0-9.]+)", text).group(1)) + entries = int(re.search(r"entries\s*:\s*(\d+)", text).group(1)) + return {"times": sorted(times), "unpacked": unpacked, "entries": entries} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--runs", type=int, default=3) + ap.add_argument("--source", default=DEFAULT_SOURCE, + help="file or directory to pack (default: %s)" % DEFAULT_SOURCE) + ap.add_argument("--stem", help="archive base name (default: source basename)") + args = ap.parse_args() + + if not os.path.exists(BENCH_BIN): + print("missing %s -- run: python tests/bench_driver.py" % BENCH_BIN) + return 1 + + payload = args.source + if not os.path.isabs(payload): + payload = os.path.join(REPO, payload) + if not os.path.exists(payload): + print("missing payload: %s" % payload) + return 1 + stem = args.stem or os.path.splitext(os.path.basename(payload))[0] + + archives = pack(stem, payload) + if not archives: + return 1 + + rows = [] + for archive in archives: + if not os.path.exists(archive): + continue + result = time_archive(archive, args.runs) + if result is None: + print("failed: %s" % archive) + continue + rows.append((os.path.splitext(archive)[1][1:], archive, result)) + + if not rows: + return 1 + + print() + print("best of %d runs, fresh output directory each time" % args.runs) + print("payload: %d entries, %.1f MiB unpacked" + % (rows[0][2]["entries"], rows[0][2]["unpacked"])) + print() + print(" %-5s %10s %9s %12s %12s" % + ("fmt", "packed", "best", "median", "throughput")) + best = min(r[2]["times"][0] for r in rows) + for fmt, archive, result in sorted(rows, key=lambda r: r[2]["times"][0]): + packed = os.path.getsize(archive) / 1048576.0 + times = result["times"] + median = times[len(times) // 2] + rate = result["unpacked"] / times[0] + print(" %-5s %8.1f M %7.0f ms %9.0f ms %8.0f MiB/s" + % (fmt, packed, times[0] * 1000.0, median * 1000.0, rate)) + print() + for fmt, archive, result in sorted(rows, key=lambda r: r[2]["times"][0]): + print(" %-5s vs fastest: %.2fx" % (fmt, result["times"][0] / best)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/bench_progress.c b/tests/bench_progress.c new file mode 100644 index 0000000..2f9f40d --- /dev/null +++ b/tests/bench_progress.c @@ -0,0 +1,161 @@ +/* Prints every extraction progress report, with a monotonic timestamp. + * + * bench_progress [--mode print|empty|none] + * + * bench_extract answers "how long"; this answers "did the UI move while it + * took that long". A report that sits at entries=0/N for minutes reads to the + * user as a hang even though bytes are still flowing, so being able to see the + * phase/entries/bytes sequence is what turns "the progress bar freezes" into a + * named phase. Format is picked from the suffix, same as bench_extract. + * + * --mode exists to answer a second question: "does reporting itself cost + * time?". The engine's report() helper runs on the hot path -- unrar calls it + * once per decompressed chunk -- and it reads the clock before it decides + * whether to throttle, so the cost is paid even when no report goes out. + * print a callback that formats and prints every report (diagnostic) + * empty a callback that returns immediately (engine cost, no consumer) + * none no callback at all (the engine skips report() entirely) + * Comparing wall time across the three separates "our bookkeeping" from + * "the file I/O we cannot avoid". + */ + +#include +#include +#include +#include + +#include "zip_extract.h" +#include "rar_extract.h" +#include "sevenz_extract.h" + +static double g_t0; + +static int +has_suffix(const char *path, const char *suffix) { + size_t path_len, suffix_len, i; + + if(!path || !suffix) return 0; + path_len = strlen(path); + suffix_len = strlen(suffix); + if(path_len < suffix_len) return 0; + for(i = 0; i < suffix_len; i++) { + char a = path[path_len - suffix_len + i]; + char b = suffix[i]; + + if(a >= 'A' && a <= 'Z') a = (char)(a - 'A' + 'a'); + if(b >= 'A' && b <= 'Z') b = (char)(b - 'A' + 'a'); + if(a != b) return 0; + } + return 1; +} + +static double +now_seconds(void) { + struct timespec ts; + + if(timespec_get(&ts, TIME_UTC) != TIME_UTC) return 0.0; + return (double)ts.tv_sec + (double)ts.tv_nsec / 1000000000.0; +} + +static const char * +phase_name(int phase) { + switch(phase) { + case ZIPX_PHASE_SCAN: return "scan"; + case ZIPX_PHASE_EXTRACT: return "extract"; + case ZIPX_PHASE_PUBLISH: return "publish"; + case ZIPX_PHASE_CLEANUP: return "cleanup"; + default: return "?"; + } +} + +/* Costs exactly what the engine's report() costs, without a consumer. */ +static void +on_progress_empty(void *userdata, const zipx_progress_t *p) { + (void)userdata; + (void)p; +} + +static void +on_progress(void *userdata, const zipx_progress_t *p) { + (void)userdata; + printf("%8.3f %-7s entries=%llu/%llu bytes=%llu/%llu %s\n", + now_seconds() - g_t0, phase_name(p->phase), + (unsigned long long)p->entries_done, + (unsigned long long)p->entries_total, + (unsigned long long)p->bytes_done, + (unsigned long long)p->bytes_total, + p->current ? p->current : ""); + fflush(stdout); +} + +int +main(int argc, char **argv) { + zipx_result_t result; + zipx_status_t status; + const char *archive; + const char *out_dir; + const char *format; + const char *mode = "print"; + zipx_progress_fn on_report = on_progress; + double elapsed; + int i; + + if(argc < 3) { + fprintf(stderr, "usage: %s [--mode print|empty|none]\n", + argv[0]); + return 2; + } + archive = argv[1]; + out_dir = argv[2]; + + for(i = 3; i < argc; i++) { + if(!strcmp(argv[i], "--mode") && i + 1 < argc) { + mode = argv[++i]; + } else { + fprintf(stderr, "unknown argument: %s\n", argv[i]); + return 2; + } + } + if(!strcmp(mode, "empty")) { + on_report = on_progress_empty; + } else if(!strcmp(mode, "none")) { + on_report = NULL; + } else if(strcmp(mode, "print")) { + fprintf(stderr, "unknown --mode: %s\n", mode); + return 2; + } + + if(has_suffix(archive, ".rar") || has_suffix(archive, ".part1.rar") || + has_suffix(archive, ".r00")) { + format = "rar"; + } else if(has_suffix(archive, ".zip") || has_suffix(archive, ".z01") || + has_suffix(archive, ".zip.001")) { + format = "zip"; + } else { + format = "7z"; + } + + memset(&result, 0, sizeof(result)); + g_t0 = now_seconds(); + + if(!strcmp(format, "rar")) { + status = rar_extract(archive, out_dir, ZIPX_CONFLICT_OVERWRITE, + zipx_default_limits(), NULL, on_report, NULL, + &result); + } else if(!strcmp(format, "zip")) { + status = zipx_extract(archive, out_dir, ZIPX_CONFLICT_OVERWRITE, + zipx_default_limits(), NULL, on_report, NULL, + &result); + } else { + status = sevenz_extract(archive, out_dir, ZIPX_CONFLICT_OVERWRITE, + zipx_default_limits(), NULL, on_report, NULL, + NULL, &result); + } + + elapsed = now_seconds() - g_t0; + printf("---- done: format=%s mode=%s status=%s entries=%llu bytes=%llu wall=%.3fs\n", + format, mode, zipx_status_string(status), + (unsigned long long)result.entries_total, + (unsigned long long)result.bytes_total, elapsed); + return status == ZIPX_OK ? 0 : 1; +}