mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 04:04:09 +02:00
Hands: upload from the hand recorder's window, export checks, a rehearsal
hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+) checks an export before upload and when it's received: SHA256SUMS, an allow-list of files, the manifest's schema and keys, the consent version, a uuid4 contributor, no identifying fields in calibration.json or device.json, every sets.bin.zst decompressed to its end as a stream with each FHSET01 header checked against the manifest, jsonl lines, the total size. It decompresses with compression.zstd, zstandard or the zstd program. validate.py DIR [--json]. hands/rec/hub.py uploads an export with huggingface_hub, as a pull request to contributions/<contributor>/<session>: validate first, refuse a repeat of the same export, check the login (whoami) and access (auth_check), upload_folder(create_pr=True) with the manifest summary as the description, then record the PR under "uploads" in session.json. Errors are explained (terms not accepted, not found, 401/403, network). --dry-run makes no network calls. FT_HANDREC_DATASET overrides HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real upload needs FT_HANDREC_ALLOW_UPLOAD=1. The Upload page shows the login with "Check again" and how to run hf auth login in a terminal (the token never enters the window), then Upload with a phase, progress and Cancel (hub.py as a child process), the PR link, and a warning for an export uploaded before. The manual command stays as the fallback. ft-handrec --hub-dry-run. session.py also saves device.json: cv.cad_from_cal and head from /persist/device_config.json, the labeller's shape, nothing identifying; export copies it. Session ids with a -N suffix are accepted everywhere. hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays 30 s of a capture into a ring, session.py records a short test script with ft-handpanel --no-vr and a tracker, then export, validate and a dry-run upload (--repo ID uploads for real). It runs in one frame-job scope, deletes its data and stops its processes, also on Ctrl+C. hands/rec/tests/test_validate.py covers good and broken exports and hub.py without the network. The dev container gains python3-huggingface-hub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
1 parent
27b731878c
commit
1fb6e7a38e
11 files changed
+2322
-55
No files matched your search
+44
-5
@@ -37,9 +37,10 @@ Test hooks:
|
||||
```
|
||||
~/.local/share/frametop/hands/contrib/
|
||||
profile.json consent and contributor id (below)
|
||||
sessions/<YYYYMMDD-HHMMSS>/
|
||||
sessions/<YYYYMMDD-HHMMSS>/ (a second session started in the same second gets -2, and so on)
|
||||
session.json the session: checklist answers, lighting, versions, takes
|
||||
calibration.json /persist/xrservice.json with identifying fields removed (below)
|
||||
device.json the rig's pose in the CAD frame from /persist/device_config.json (below)
|
||||
takes/<NN>-<section>/
|
||||
sets.bin ft-hands' recording (FHSET01, hands/track/record.h), 10 sets/s
|
||||
prompts.jsonl what the person was asked to do, when (below)
|
||||
@@ -75,6 +76,17 @@ No name, email, or account. The contributor id is random, so several sessions fr
|
||||
|
||||
This is a copy of `/persist/xrservice.json` (`/run/host/persist/` in the container). Keep the cameras' intrinsics and extrinsics, and drop anything that identifies the unit: keys containing `serial`, `sn`, `uuid`, `mac` or `id`, or values that look like serial numbers. List what was removed in `session.json` (`"calibration_removed": [...]`), so a reviewer can check it.
|
||||
|
||||
### device.json
|
||||
|
||||
The head frame needs the rig's pose in the CAD frame, from `/persist/device_config.json`. Only two of its keys are kept, `cv.cad_from_cal` (Cam0 in the CAD frame) and `head` (the head in CAD), in the shape the labeller in frame-hands `train/label` reads, as its `cut.py` writes it:
|
||||
|
||||
```json
|
||||
{"cv": {"cad_from_cal": {"method": "FrontAndUpperCamPositions", "plus_x": [x, y, z], "plus_z": [x, y, z], "position": [x, y, z]}},
|
||||
"head": {"plus_x": [x, y, z], "plus_z": [x, y, z], "position": [x, y, z]}}
|
||||
```
|
||||
|
||||
The rest of that file identifies the unit (serial number, display EDID) and is never copied. The two kept keys go through `strip_calibration` as well, and anything it removes is listed in `calibration_removed` as `device.json:<path>`. Sessions recorded before device.json existed have none: they still validate, with a warning, and the labeller falls back to another unit's pose.
|
||||
|
||||
### prompts.jsonl
|
||||
|
||||
One JSON object per line:
|
||||
@@ -185,13 +197,40 @@ Before section 8: "Put on both controllers and tighten the straps". Before secti
|
||||
|
||||
- **Review.** Each session lists its takes: title, duration, status, a thumbnail (the first set's `slam_left`). A viewer shows one frame set (all cameras side by side, 8-bit grey) at a time with a slider. It can mark a range and delete it. Deleted ranges go into `take.json`, and export leaves them out; the files keep everything until export. A whole take or session can be deleted (files removed, after a confirmation).
|
||||
- **Export.** It writes `exports/<session>/`:
|
||||
- `manifest.json`: profile fields except `optional.notes` unless kept, session.json, takes, schema, tool version, the consent version.
|
||||
- `calibration.json`.
|
||||
- `manifest.json`: profile fields except `optional.notes` unless kept, session.json (without its `uploads` records), takes, schema, tool version, the consent version.
|
||||
- `calibration.json` and `device.json`, when the session has them.
|
||||
- Per take: `prompts.jsonl`, `poses.jsonl`, `take.json`, and `sets.bin.zst` (sets in deleted ranges removed, then zstd -10 with 2 threads).
|
||||
- `SHA256SUMS`.
|
||||
|
||||
Compression runs at nice 19. Before starting, the window warns if the headset is worn, judged the way `frame-job` does: `vrcompositor` runs and a `/sys/class/backlight/*/brightness` reads over 0 (SteamVR turns the panel off 5 s after the headset comes off). CPU work while in VR causes stutter. The proximity sensor is no use here: it read 9-43 with the headset sitting unworn.
|
||||
- **Upload.** The Upload page shows `UPLOAD.md` with the export's path and size filled in, plus the copyable command.
|
||||
- **Upload.** The Upload page uploads an export from the window. It also keeps the manual way as a fallback: `UPLOAD.md` with the export's path and size filled in, plus the copyable command (`validate.py`, then `hf upload ... --create-pr`).
|
||||
- **`hands/rec/validate.py`** (standard library) checks an export. The window runs it before an upload, and the maintainer runs it on each submission (`validate.py DIR [--json]`, exit status 1 on errors). It checks:
|
||||
- `SHA256SUMS`: every file listed and matching.
|
||||
- An allow-list: `manifest.json`, `calibration.json`, `device.json` and `SHA256SUMS` at the top, and `prompts.jsonl`, `poses.jsonl`, `take.json` and `sets.bin.zst` in `takes/<NN>-<section>/`. Anything else is an error, and so is a symlink.
|
||||
- The manifest's schema, keys and types, and that it matches the files.
|
||||
- The consent version is present and the contributor confirmed being an adult. The contributor id is a uuid4.
|
||||
- `calibration.json` has nothing that `session.py`'s `strip_calibration` would still remove.
|
||||
- `device.json` holds only `cv.cad_from_cal` and `head`, each `plus_x`, `plus_z` and `position` as 3 numbers (plus `cad_from_cal`'s `method`). Without it: a warning.
|
||||
- Each `sets.bin.zst` decompresses to its end, so a truncated one fails, and every set's FHSET01 header is sane: camera names, sizes, record length. Set counts and raw bytes match the manifest. Pixels aren't decoded.
|
||||
- Every jsonl line parses.
|
||||
- The total size: a warning over 15 GB, an error over 40 GB.
|
||||
|
||||
Warnings cover notes kept in the export, a home folder path in the manifest, and missing `poses.jsonl` files.
|
||||
|
||||
It runs on Linux and on Windows (the maintainer's PC) with Python 3.12 or later. It decompresses with Python 3.14's `compression.zstd`, else the `zstandard` package, else the `zstd` program, and handles several zstd frames in a row.
|
||||
- **`hands/rec/hub.py`** does the upload. It runs as a child process of the window (Cancel ends it), or from the command line (`hub.py [--base DIR] whoami | upload SESSION [--dry-run] [--again] [--json]`). It uses `huggingface_hub` (`python3-huggingface-hub` in the dev container) with the token `hf auth login` saved, and never handles a token itself. An upload goes through these steps:
|
||||
1. Validate, and stop on errors.
|
||||
2. Stop if the same export was uploaded before (same `SHA256SUMS`), unless asked again.
|
||||
3. Stop while the texts are drafts, unless `FT_HANDREC_ALLOW_UPLOAD=1`.
|
||||
4. `whoami`: a read-only token is refused.
|
||||
5. `auth_check` on the dataset: a gated dataset whose terms aren't accepted gives "accept the dataset's terms first", with the link.
|
||||
6. `upload_folder(repo_id=HF_DATASET, repo_type="dataset", folder_path=EXPORT, path_in_repo="contributions/<contributor>/<session>", create_pr=True, commit_message=..., commit_description=...)`. The description summarizes the manifest: takes, minutes, sets, lighting, objects, controllers, the consent and tool versions, the size, and validate's warnings.
|
||||
7. Add `{"repo", "pr_url", "uploaded", "export_sha"}` to `uploads` in `session.json`. `export_sha` is the SHA256 of `SHA256SUMS`. The file keeps its modification time, so the export doesn't count as out of date.
|
||||
|
||||
Errors get a plain explanation: not logged in, a token Hugging Face rejects (401), a token that can't open a pull request (403), terms not accepted, dataset not found, network errors. `--dry-run` does everything except the network calls and the record, and lists what it would upload.
|
||||
- **The page** shows the login (`whoami`, with "Check again"). If nobody is logged in, it explains how to run `distrobox enter dev -- hf auth login` in Konsole with a write token: the token goes only into that terminal. The page then has Upload and Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), and the pull request's link when it's done. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded.
|
||||
- **While the texts are drafts**, Upload stays off unless `FT_HANDREC_ALLOW_UPLOAD=1`, so the maintainer can rehearse against a private test repo. `FT_HANDREC_DATASET` overrides `HF_DATASET`. `ft-handrec --hub-dry-run` makes Upload a dry run: no network, so it isn't held back by the drafts.
|
||||
- **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start` records a two-section test script, about 11 s and about 200 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).
|
||||
|
||||
## Licensing and consent (texts in `CONSENT.md`)
|
||||
|
||||
@@ -200,4 +239,4 @@ Before section 8: "Put on both controllers and tighten the straps". Before secti
|
||||
- Contributors confirm they're 18 or older.
|
||||
- The text explains what's recorded and that nothing uploads automatically, how review works, how to withdraw (by contributor id), and that a withdrawal is purged from the repo's history.
|
||||
- The texts need a legal review before the dataset launches. Until then they carry a "draft" banner, and the Upload page says contributions aren't open yet.
|
||||
- The dataset repo: `HF_DATASET` in `ft_handrec.py`, a placeholder (`DeeJanuz/frametop-hands`) until the maintainer decides.
|
||||
- The dataset repo: `HF_DATASET` in `hub.py`, `DeeJanuz/frametop-hands` (private until launch).
|
||||
+18
-14
@@ -8,7 +8,7 @@ Your export is ready:
|
||||
- Size: @EXPORT_SIZE@
|
||||
- Contributor id: `@CONTRIBUTOR@`
|
||||
|
||||
Uploads go to the Hugging Face dataset [@DATASET@](https://huggingface.co/datasets/@DATASET@). You do it yourself, from your own account; the hand recorder never uploads anything. The upload opens a pull request, so nothing is published until the maintainer has looked at it.
|
||||
Uploads go to the Hugging Face dataset [@DATASET@](https://huggingface.co/datasets/@DATASET@), from your own Hugging Face account. Nothing is uploaded until you press Upload (or run the command below). The upload opens a pull request, so nothing is published until the maintainer has looked at it.
|
||||
|
||||
## 1. Make a Hugging Face account
|
||||
|
||||
@@ -16,38 +16,42 @@ Sign up at <https://huggingface.co/join>, if you don't have an account. Your use
|
||||
|
||||
## 2. Accept the dataset's terms
|
||||
|
||||
Open <https://huggingface.co/datasets/@DATASET@>, read the terms and accept them. They're the same as the consent you agreed to in the hand recorder.
|
||||
Open <https://huggingface.co/datasets/@DATASET@>, read the terms and accept them. They're the same as the consent you agreed to in the hand recorder. Until you've accepted them, the upload stops with "accept the dataset's terms first".
|
||||
|
||||
## 3. Create a write token
|
||||
|
||||
Go to <https://huggingface.co/settings/tokens>, press "Create new token", choose "Write" and give it a name like "frametop-hands". Copy the token.
|
||||
|
||||
The token lets anyone who has it change things in your account. Paste it only into your own terminal in the next step: never into a chat, a website, an issue or this window.
|
||||
The token lets anyone who has it change things in your account. Paste it only into your own terminal in the next step: never into a chat, a website, an issue or this window. The hand recorder never asks for it.
|
||||
|
||||
## 4. Install the Hugging Face tools
|
||||
## 4. Log in, in a terminal
|
||||
|
||||
Open a terminal (Konsole) and enter the dev container, then install `huggingface_hub` and log in:
|
||||
Open a terminal (Konsole) and run:
|
||||
|
||||
```
|
||||
distrobox enter dev
|
||||
pip install --user huggingface_hub
|
||||
huggingface-cli login
|
||||
@LOGIN@
|
||||
```
|
||||
|
||||
`huggingface-cli login` asks for the token: paste it there. If it asks whether to add the token as a git credential, answer no.
|
||||
It asks for the token: paste it there (it doesn't show as you paste) and press Enter. The token is saved in your home folder, in `~/.cache/huggingface/token`, where the hand recorder's upload finds it. Then press "Check again" on this page: it should say you're logged in.
|
||||
|
||||
If the page says `huggingface_hub` isn't installed, update the dev container first: run `setup/dev-container.sh` from the Frametop folder.
|
||||
|
||||
## 5. Upload
|
||||
|
||||
In the same terminal, run:
|
||||
Press **Upload** on this page. It first checks the export (that every file is complete and matches its checksum, and that nothing identifying is left in), then uploads it to `contributions/@CONTRIBUTOR@/@SESSION@` in the dataset and opens a pull request. Large uploads take a while, and Cancel stops it. When it's done, the page shows your pull request's link.
|
||||
|
||||
If it fails partway, press Upload again: files already sent usually aren't sent twice. If a pull request was opened anyway, that's fine: the maintainer closes the incomplete one.
|
||||
|
||||
Keep the headset on its charger or plugged in while it uploads.
|
||||
|
||||
### Or upload from a terminal
|
||||
|
||||
If the Upload button doesn't work for you, run this in the dev container (`distrobox enter dev`) after step 4. It checks the export the same way, then uploads it:
|
||||
|
||||
```
|
||||
@COMMAND@
|
||||
```
|
||||
|
||||
It uploads the export folder to `contributions/@CONTRIBUTOR@/@SESSION@` in the dataset and opens a pull request. Large uploads take a while. If it fails partway, run the same command again: parts already sent usually aren't sent twice. That can open a second pull request, which is fine: the maintainer closes the incomplete one.
|
||||
|
||||
Keep the headset on its charger or plugged in while it uploads.
|
||||
|
||||
## 6. The maintainer reviews it
|
||||
|
||||
The maintainer checks your pull request (that the files are complete, and that nobody else and nothing private is in view) before merging it into the dataset. You can follow it, and answer questions, on the pull request's page.
|
||||
|
||||
+241
-20
@@ -12,16 +12,23 @@ A Kirigami (QML) app with a Python backend. It runs in the dev container:
|
||||
takes and sessions can be deleted (takes.py).
|
||||
- Export: compress what's kept into exports/<session>/ at nice 19 (takes.py), with a warning
|
||||
when the headset is worn.
|
||||
- Upload: UPLOAD.md with the export filled in and the upload command to copy.
|
||||
- Upload: the Hugging Face login status, and Upload: hub.py checks the export (validate.py)
|
||||
and uploads it as a pull request, in a child process (Cancel ends it). UPLOAD.md explains
|
||||
the steps, with the export filled in, and the command to copy for a terminal upload.
|
||||
Everything lives under ~/.local/share/frametop/hands/contrib (--base). Nothing is uploaded
|
||||
from here. Launch with hands/rec/ft-handrec (host wrapper).
|
||||
unless the person presses Upload; while CONSENT.md or UPLOAD.md is a draft, Upload stays off
|
||||
unless FT_HANDREC_ALLOW_UPLOAD=1 (the maintainer's rehearsal against a test repo, picked with
|
||||
FT_HANDREC_DATASET). --hub-dry-run does everything but the network calls.
|
||||
Launch with hands/rec/ft-handrec (host wrapper).
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import uuid
|
||||
@@ -34,12 +41,17 @@ from PySide6.QtQuickControls2 import QQuickStyle
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
import hub # noqa: E402
|
||||
import takes # noqa: E402 (next to this file)
|
||||
|
||||
# The dataset contributions go to: a placeholder until the maintainer decides (DESIGN.md).
|
||||
HF_DATASET = "DeeJanuz/frametop-hands"
|
||||
CONSENT_PATH = os.path.join(HERE, "CONSENT.md")
|
||||
UPLOAD_PATH = os.path.join(HERE, "UPLOAD.md")
|
||||
# The dataset contributions go to (hub.py; FT_HANDREC_DATASET overrides it, for tests).
|
||||
HF_DATASET = hub.HF_DATASET
|
||||
CONSENT_PATH = hub.CONSENT_PATH
|
||||
UPLOAD_PATH = hub.UPLOAD_PATH
|
||||
HUB_PATH = os.path.join(HERE, "hub.py")
|
||||
VALIDATE_PATH = os.path.join(HERE, "validate.py")
|
||||
# What `hf auth login` needs: run in a terminal, where the token is typed (never in this window).
|
||||
LOGIN_COMMAND = "distrobox enter dev -- hf auth login"
|
||||
SCRIPT_PATH = os.path.join(HERE, "script.json")
|
||||
# The headset counts as worn while vrcompositor runs and a display panel is lit: SteamVR turns
|
||||
# the panels off 5 s after the headset comes off (frame-job's check; the proximity sensor's
|
||||
@@ -74,10 +86,6 @@ def consent_version():
|
||||
return m.group(1) if m else "unknown"
|
||||
|
||||
|
||||
def is_draft(path):
|
||||
return "DRAFT" in read_text(path).split("\n", 1)[0]
|
||||
|
||||
|
||||
def process_running(name):
|
||||
"""Like pgrep -x NAME, from /proc."""
|
||||
for pid in os.listdir("/proc"):
|
||||
@@ -111,11 +119,13 @@ def gigabytes(n):
|
||||
|
||||
|
||||
def session_label(sid):
|
||||
"""20261002-101500 -> 2026-10-02 10:15."""
|
||||
"""20261002-101500 -> 2026-10-02 10:15; 20261002-101500-2 -> 2026-10-02 10:15 (2)."""
|
||||
stamp, _, n = sid[:15], sid[15:16], sid[16:]
|
||||
try:
|
||||
return datetime.datetime.strptime(sid, "%Y%m%d-%H%M%S").strftime("%Y-%m-%d %H:%M")
|
||||
label = datetime.datetime.strptime(stamp, "%Y%m%d-%H%M%S").strftime("%Y-%m-%d %H:%M")
|
||||
except ValueError:
|
||||
return sid
|
||||
return f"{label} ({n})" if n else label
|
||||
|
||||
|
||||
def grey_image(cam):
|
||||
@@ -191,11 +201,23 @@ class Backend(QObject):
|
||||
_lightingArrived = Signal(str)
|
||||
_exportProgress = Signal(float, str)
|
||||
_exportFinished = Signal(str, str) # path, error ("" when it worked; "cancelled")
|
||||
loginChanged = Signal()
|
||||
uploadChanged = Signal()
|
||||
_loginArrived = Signal(dict)
|
||||
_uploadLine = Signal(dict)
|
||||
_uploadFinished = Signal(dict)
|
||||
|
||||
def __init__(self, store, session_options=None):
|
||||
def __init__(self, store, session_options=None, hub_dry_run=False):
|
||||
super().__init__()
|
||||
self.store = store
|
||||
self._session_options = session_options or {} # test hooks for Session: dry_run, speed
|
||||
self._hub_dry_run = hub_dry_run
|
||||
self._login = {"state": "dry" if hub_dry_run else "unknown"}
|
||||
self._login_busy = False
|
||||
self._upload_proc = None
|
||||
self._upload_thread = None
|
||||
self._upload_cancelled = False
|
||||
self._upload = {}
|
||||
os.makedirs(store.base, mode=0o700, exist_ok=True)
|
||||
self._session_mod = None
|
||||
self._session_error = ""
|
||||
@@ -212,6 +234,9 @@ class Backend(QObject):
|
||||
self._lightingArrived.connect(self._on_lighting)
|
||||
self._exportProgress.connect(self._on_export_progress)
|
||||
self._exportFinished.connect(self._on_export_finished)
|
||||
self._loginArrived.connect(self._on_login)
|
||||
self._uploadLine.connect(self._on_upload_line)
|
||||
self._uploadFinished.connect(self._on_upload_finished)
|
||||
self.disk_timer = QTimer(interval=30000, timeout=self.diskChanged.emit)
|
||||
self.disk_timer.start()
|
||||
|
||||
@@ -246,7 +271,7 @@ class Backend(QObject):
|
||||
|
||||
@Property(bool, constant=True)
|
||||
def textsDraft(self):
|
||||
return is_draft(CONSENT_PATH) or is_draft(UPLOAD_PATH)
|
||||
return hub.texts_draft()
|
||||
|
||||
@Property(bool, notify=profileChanged)
|
||||
def needsConsent(self):
|
||||
@@ -439,6 +464,9 @@ class Backend(QObject):
|
||||
if self._export_cancel:
|
||||
self._export_cancel.set() # and wait, so export can remove its half-written copy
|
||||
self._export_thread.join(15)
|
||||
if self._upload_proc:
|
||||
self.cancelUpload()
|
||||
self._upload_thread.join(10)
|
||||
|
||||
# --- review
|
||||
@Property("QVariantList", notify=sessionsChanged)
|
||||
@@ -633,8 +661,197 @@ class Backend(QObject):
|
||||
# --- upload
|
||||
@Property(str, constant=True)
|
||||
def dataset(self):
|
||||
return HF_DATASET
|
||||
try:
|
||||
return hub.dataset_id()
|
||||
except hub.HubError:
|
||||
return os.environ.get(hub.DATASET_ENV, HF_DATASET)
|
||||
|
||||
@Property(str, constant=True)
|
||||
def datasetUrl(self):
|
||||
return hub.dataset_url(self.dataset)
|
||||
|
||||
@Property(bool, constant=True)
|
||||
def hubDryRun(self):
|
||||
return self._hub_dry_run
|
||||
|
||||
@Property(bool, constant=True)
|
||||
def uploadAllowed(self):
|
||||
"""Real uploads may start: the texts aren't drafts, or FT_HANDREC_ALLOW_UPLOAD=1."""
|
||||
return hub.upload_allowed()
|
||||
|
||||
@Property(bool, constant=True)
|
||||
def allowUploadSet(self):
|
||||
return os.environ.get(hub.ALLOW_ENV) == "1"
|
||||
|
||||
@Property(str, constant=True)
|
||||
def loginCommand(self):
|
||||
return LOGIN_COMMAND
|
||||
|
||||
def _hub_argv(self, *args):
|
||||
return [sys.executable, HUB_PATH, "--base", self.store.base, *args]
|
||||
|
||||
# The login: hub.py whoami in a child process (it asks huggingface.co)
|
||||
@Property("QVariantMap", notify=loginChanged)
|
||||
def login(self):
|
||||
"""{"state": unknown|checking|ok|none|read|error|missing|dry, "name", "role", "text", "link"}."""
|
||||
return self._login
|
||||
|
||||
@Slot()
|
||||
def checkLogin(self):
|
||||
if self._hub_dry_run or self._login_busy:
|
||||
return
|
||||
self._login_busy = True
|
||||
self._login = {"state": "checking", "text": "Checking your Hugging Face login"}
|
||||
self.loginChanged.emit()
|
||||
argv = self._hub_argv("whoami", "--json")
|
||||
|
||||
def run():
|
||||
try:
|
||||
r = subprocess.run(argv, capture_output=True, text=True, timeout=60)
|
||||
result = self._last_json(r.stdout) or {"error": {"kind": "hub", "text": (r.stderr.strip().splitlines()
|
||||
or ["hub.py failed"])[-1]}}
|
||||
except (OSError, subprocess.TimeoutExpired) as e:
|
||||
result = {"error": {"kind": "network", "text": f"The login check didn't finish: {e}"}}
|
||||
self._loginArrived.emit(result)
|
||||
self._thread(run)
|
||||
|
||||
@staticmethod
|
||||
def _last_json(text):
|
||||
for line in reversed(text.strip().splitlines()):
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(obj, dict):
|
||||
return obj
|
||||
return None
|
||||
|
||||
def _on_login(self, result):
|
||||
self._login_busy = False
|
||||
if "done" in result:
|
||||
who = result["done"]
|
||||
role = who.get("role", "")
|
||||
if role == "read":
|
||||
self._login = {"state": "read", "name": who.get("name", ""), "role": role, "link": hub.TOKENS_URL,
|
||||
"text": f"Logged in as {who.get('name', '')}, but with a read-only token: uploads "
|
||||
"need a token with the Write role. Log in again with one."}
|
||||
else:
|
||||
self._login = {"state": "ok", "name": who.get("name", ""), "role": role,
|
||||
"text": f"Logged in as {who.get('name', '')}"}
|
||||
else:
|
||||
e = result.get("error") or {}
|
||||
state = {"login": "none", "missing": "missing"}.get(e.get("kind"), "error")
|
||||
self._login = {"state": state, "text": e.get("text", "The login check failed"), "link": e.get("link", "")}
|
||||
self.loginChanged.emit()
|
||||
|
||||
# The upload: hub.py upload --json in a child process; its lines say how it goes
|
||||
@Property(bool, notify=uploadChanged)
|
||||
def uploading(self):
|
||||
return self._upload_proc is not None
|
||||
|
||||
@Property("QVariantMap", notify=uploadChanged)
|
||||
def upload(self):
|
||||
"""{"session", "phase", "text", "fraction" (-1: unknown), "log", "result": {...}, "error": {...}}."""
|
||||
return self._upload
|
||||
|
||||
@Slot(str, result="QVariantMap")
|
||||
def uploadInfo(self, session):
|
||||
"""{"previous": the last upload of this same export or {}, "uploads": how many in all}."""
|
||||
try:
|
||||
export = self.store.export_dir(session)
|
||||
before = hub.previous_upload(self.store, session, hub.export_sha(export))
|
||||
count = len(hub.uploads(self.store, session))
|
||||
except (OSError, ValueError):
|
||||
return {"previous": {}, "uploads": 0}
|
||||
return {"previous": before or {}, "uploads": count}
|
||||
|
||||
@Slot(str, bool)
|
||||
def startUpload(self, session, again):
|
||||
if self.uploading or self.exporting or self._active_guard(session):
|
||||
return
|
||||
if not self._hub_dry_run and not hub.upload_allowed():
|
||||
self.message.emit("Contributions aren't open yet", True)
|
||||
return
|
||||
argv = self._hub_argv("upload", session, "--json")
|
||||
if self._hub_dry_run:
|
||||
argv.append("--dry-run")
|
||||
if again:
|
||||
argv.append("--again")
|
||||
try:
|
||||
proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, bufsize=1)
|
||||
except OSError as e:
|
||||
self.message.emit(f"Couldn't start the upload: {e}", True)
|
||||
return
|
||||
self._upload_proc = proc
|
||||
self._upload_cancelled = False
|
||||
self._upload = {"session": session, "phase": "check", "text": "Starting", "fraction": 0.0, "log": "",
|
||||
"result": {}, "error": {}}
|
||||
self.uploadChanged.emit()
|
||||
|
||||
def run():
|
||||
last = {}
|
||||
for line in proc.stdout:
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(obj, dict):
|
||||
if "done" in obj or "error" in obj:
|
||||
last = obj
|
||||
else:
|
||||
self._uploadLine.emit(obj)
|
||||
err = proc.stderr.read()
|
||||
proc.wait()
|
||||
proc.stdout.close()
|
||||
proc.stderr.close()
|
||||
if not last:
|
||||
tail = (err.strip().splitlines() or [f"exit status {proc.returncode}"])[-1]
|
||||
last = {"error": {"kind": "cancelled" if self._upload_cancelled else "hub",
|
||||
"text": "Cancelled" if self._upload_cancelled else f"The upload stopped: {tail}"}}
|
||||
self._uploadFinished.emit(last)
|
||||
self._upload_thread = self._thread(run)
|
||||
|
||||
@Slot()
|
||||
def cancelUpload(self):
|
||||
if self._upload_proc and self._upload_proc.poll() is None:
|
||||
self._upload_cancelled = True
|
||||
self._upload_proc.terminate()
|
||||
self._upload = dict(self._upload, text="Cancelling")
|
||||
self.uploadChanged.emit()
|
||||
|
||||
def _on_upload_line(self, obj):
|
||||
up = dict(self._upload)
|
||||
if "log" in obj:
|
||||
up["log"] = (up.get("log", "") + obj["log"] + "\n")[-20000:]
|
||||
if "phase" in obj:
|
||||
f = obj.get("fraction")
|
||||
up.update(phase=obj["phase"], text=obj.get("text", ""), fraction=-1.0 if f is None else float(f))
|
||||
self._upload = up
|
||||
self.uploadChanged.emit()
|
||||
|
||||
def _on_upload_finished(self, last):
|
||||
self._upload_proc = None
|
||||
up = dict(self._upload)
|
||||
if "done" in last:
|
||||
result = last["done"]
|
||||
up.update(result=result, error={}, phase="done", fraction=1.0,
|
||||
text="Dry run: nothing was uploaded" if result.get("dry_run") else "Uploaded")
|
||||
if not result.get("dry_run"):
|
||||
self.message.emit("Uploaded: the pull request is open", False)
|
||||
else:
|
||||
e = last.get("error") or {}
|
||||
cancelled = e.get("kind") == "cancelled"
|
||||
up.update(error=e, result={}, phase="failed", fraction=0.0,
|
||||
text=("Cancelled. If it had got as far as opening the pull request, check the dataset's "
|
||||
"Community tab: an incomplete one can be closed there.") if cancelled else e.get("text", ""))
|
||||
if not cancelled:
|
||||
self.message.emit("Upload failed", True)
|
||||
self._upload = up
|
||||
self.uploadChanged.emit()
|
||||
self.sessionsChanged.emit()
|
||||
|
||||
# The manual way, for a terminal (UPLOAD.md)
|
||||
@Slot(str, result=str)
|
||||
def uploadCommand(self, session):
|
||||
try:
|
||||
@@ -642,9 +859,11 @@ class Backend(QObject):
|
||||
except ValueError:
|
||||
return ""
|
||||
contributor = self.contributor or "CONTRIBUTOR"
|
||||
return " ".join(["huggingface-cli", "upload", HF_DATASET, shlex.quote(path),
|
||||
f"contributions/{contributor}/{session}", "--repo-type", "dataset", "--create-pr",
|
||||
check = " ".join(["python3", shlex.quote(VALIDATE_PATH), shlex.quote(path)])
|
||||
upload = " ".join(["hf", "upload", self.dataset, shlex.quote(path), f"contributions/{contributor}/{session}",
|
||||
"--repo-type", "dataset", "--create-pr",
|
||||
"--commit-message", shlex.quote(f"Hands: session {session} from {contributor}")])
|
||||
return check + " && " + upload
|
||||
|
||||
@Slot(str, result=str)
|
||||
def uploadText(self, session):
|
||||
@@ -654,8 +873,8 @@ class Backend(QObject):
|
||||
except ValueError:
|
||||
return ""
|
||||
values = {"EXPORT_PATH": path, "EXPORT_SIZE": takes.human_bytes(takes.tree_bytes(path)),
|
||||
"CONTRIBUTOR": self.contributor or "CONTRIBUTOR", "SESSION": session, "DATASET": HF_DATASET,
|
||||
"COMMAND": self.uploadCommand(session)}
|
||||
"CONTRIBUTOR": self.contributor or "CONTRIBUTOR", "SESSION": session, "DATASET": self.dataset,
|
||||
"COMMAND": self.uploadCommand(session), "LOGIN": LOGIN_COMMAND}
|
||||
text = read_text(UPLOAD_PATH) or "UPLOAD.md is missing."
|
||||
return re.sub(r"@([A-Z_]+)@", lambda m: values.get(m.group(1), m.group(0)), text)
|
||||
|
||||
@@ -678,6 +897,8 @@ def main():
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="test: sessions start no processes and print the panel's commands")
|
||||
ap.add_argument("--speed", type=float, default=1.0, help="test, with --dry-run: run sessions this much faster")
|
||||
ap.add_argument("--hub-dry-run", action="store_true",
|
||||
help="test: Upload checks the export and says what it would send, with no network calls")
|
||||
a, qt_args = ap.parse_known_args()
|
||||
app = QGuiApplication([sys.argv[0]] + qt_args)
|
||||
app.setApplicationName("ft-handrec")
|
||||
@@ -689,7 +910,7 @@ def main():
|
||||
store = takes.Store(a.base)
|
||||
engine = QQmlApplicationEngine()
|
||||
engine.addImageProvider("frames", FrameProvider(store))
|
||||
backend = Backend(store, {"dry_run": True, "speed": a.speed} if a.dry_run else {})
|
||||
backend = Backend(store, {"dry_run": True, "speed": a.speed} if a.dry_run else {}, hub_dry_run=a.hub_dry_run)
|
||||
app.aboutToQuit.connect(backend.shutdown)
|
||||
engine.rootContext().setContextProperty("backend", backend)
|
||||
engine.rootContext().setContextProperty("startPage", a.page)
|
||||
|
||||
Executable
+377
@@ -0,0 +1,377 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Upload a hand recorder export to the hand dataset on Hugging Face (DESIGN.md "Upload").
|
||||
|
||||
The window runs this as a child process (so Cancel can end it, and huggingface_hub stays out
|
||||
of the window's process); it also runs from the command line. It uses huggingface_hub (in the
|
||||
dev container: python3-huggingface-hub, from setup/dev-container.sh) with the token that
|
||||
`hf auth login` saved. It never asks for or handles a token itself.
|
||||
|
||||
An upload:
|
||||
1. checks the export with validate.py, and stops on errors;
|
||||
2. stops if this export (same SHA256SUMS) was uploaded before, unless --again;
|
||||
3. stops while CONSENT.md or UPLOAD.md is a draft, unless FT_HANDREC_ALLOW_UPLOAD=1;
|
||||
4. checks the login (whoami: a read-only token can't open a pull request) and access to the
|
||||
dataset (auth_check: a gated dataset's terms must be accepted first);
|
||||
5. upload_folder(..., create_pr=True) to contributions/<contributor>/<session>;
|
||||
6. records {"repo", "pr_url", "uploaded", "export_sha"} under "uploads" in session.json.
|
||||
--dry-run does all of it except the network calls (4 and 5) and recording (6), and says what
|
||||
it would upload.
|
||||
|
||||
The dataset is HF_DATASET; FT_HANDREC_DATASET overrides it (a test repo, for rehearsals).
|
||||
|
||||
usage: hub.py [--base DIR] whoami [--json]
|
||||
hub.py [--base DIR] upload SESSION [--dry-run] [--again] [--json]
|
||||
With --json, each line of output is one JSON object: {"phase", "text", "fraction"} as it goes,
|
||||
then {"done": {...}} or {"error": {"kind", "text", "link", "errors"}}. Exit status 0: done.
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
import takes # noqa: E402 (next to this file)
|
||||
|
||||
# The dataset contributions go to (DESIGN.md "Licensing and consent").
|
||||
HF_DATASET = "DeeJanuz/frametop-hands"
|
||||
DATASET_ENV = "FT_HANDREC_DATASET"
|
||||
ALLOW_ENV = "FT_HANDREC_ALLOW_UPLOAD"
|
||||
CONSENT_PATH = os.path.join(HERE, "CONSENT.md")
|
||||
UPLOAD_PATH = os.path.join(HERE, "UPLOAD.md")
|
||||
REPO_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$")
|
||||
TOKENS_URL = "https://huggingface.co/settings/tokens"
|
||||
# Quiet huggingface_hub: no progress bars on stderr, no telemetry or update hints.
|
||||
HF_ENV = {"HF_HUB_DISABLE_PROGRESS_BARS": "1", "HF_HUB_DISABLE_TELEMETRY": "1", "HF_HUB_DISABLE_UPDATE_CHECK": "1"}
|
||||
|
||||
|
||||
class HubError(Exception):
|
||||
"""What went wrong, for people: kind (below), text, a link to open, validation errors.
|
||||
Kinds: missing (no huggingface_hub), login, read-token, terms, not-found, permission,
|
||||
network, hub, invalid, duplicate, closed, no-export."""
|
||||
|
||||
def __init__(self, kind, text, link="", errors=None, extra=None):
|
||||
super().__init__(text)
|
||||
self.kind, self.text, self.link = kind, text, link
|
||||
self.errors = errors or []
|
||||
self.extra = extra or {}
|
||||
|
||||
def as_dict(self):
|
||||
return dict(self.extra, kind=self.kind, text=self.text, link=self.link, errors=self.errors)
|
||||
|
||||
|
||||
def dataset_id():
|
||||
"""HF_DATASET, or FT_HANDREC_DATASET when set."""
|
||||
repo = os.environ.get(DATASET_ENV, "").strip() or HF_DATASET
|
||||
if not REPO_RE.match(repo):
|
||||
raise HubError("not-found", f"{DATASET_ENV}={repo!r} isn't a dataset id like owner/name")
|
||||
return repo
|
||||
|
||||
|
||||
def dataset_url(repo=None):
|
||||
return "https://huggingface.co/datasets/" + (repo or dataset_id())
|
||||
|
||||
|
||||
def is_draft(path):
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return "DRAFT" in f.readline()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def texts_draft():
|
||||
return is_draft(CONSENT_PATH) or is_draft(UPLOAD_PATH)
|
||||
|
||||
|
||||
def upload_allowed():
|
||||
"""Real uploads: once the texts aren't drafts, or for the maintainer's rehearsal against a
|
||||
test repo (FT_HANDREC_ALLOW_UPLOAD=1)."""
|
||||
return not texts_draft() or os.environ.get(ALLOW_ENV) == "1"
|
||||
|
||||
|
||||
def now_iso():
|
||||
return datetime.datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- records in session.json
|
||||
|
||||
def export_sha(export_dir):
|
||||
"""The export's identity: the SHA256 of its SHA256SUMS ("" if there's none)."""
|
||||
try:
|
||||
with open(os.path.join(export_dir, "SHA256SUMS"), "rb") as f:
|
||||
return hashlib.sha256(f.read()).hexdigest()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def uploads(store, session):
|
||||
meta = takes.read_json(os.path.join(store.session_dir(session), "session.json"))
|
||||
return [u for u in meta.get("uploads") or [] if isinstance(u, dict)]
|
||||
|
||||
|
||||
def previous_upload(store, session, sha):
|
||||
"""The latest upload of this same export, or None."""
|
||||
same = [u for u in uploads(store, session) if sha and u.get("export_sha") == sha]
|
||||
return same[-1] if same else None
|
||||
|
||||
|
||||
def record_upload(store, session, record):
|
||||
"""Add record to session.json's "uploads". The file keeps its time: an upload doesn't
|
||||
change the recordings, so the export mustn't count as out of date because of it
|
||||
(takes.Store.sessions compares times)."""
|
||||
path = os.path.join(store.session_dir(session), "session.json")
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
st = None
|
||||
meta = takes.read_json(path)
|
||||
meta.setdefault("uploads", []).append(record)
|
||||
takes.write_json(path, meta)
|
||||
if st:
|
||||
os.utime(path, ns=(st.st_atime_ns, st.st_mtime_ns))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the pull request's text
|
||||
|
||||
def describe(summary, report, sha, nbytes):
|
||||
"""(commit message, commit description) from validate's summary of the manifest."""
|
||||
s = summary
|
||||
objects = ", ".join(s.get("objects") or []) or "none"
|
||||
if s.get("own_objects"):
|
||||
objects += f" (+{s['own_objects']} of their own)"
|
||||
message = f"Hands: session {s.get('session')} from {s.get('contributor')}"
|
||||
lines = ["Contribution to the Frametop hand dataset, uploaded from the hand recorder.", "",
|
||||
f"- Session: {s.get('session')}",
|
||||
f"- Contributor: {s.get('contributor')}",
|
||||
f"- Takes: {s.get('takes')} ({s.get('minutes')} minutes, {s.get('sets')} frame sets)",
|
||||
f"- Lighting: {s.get('lighting') or 'not given'}",
|
||||
f"- Objects: {objects}",
|
||||
f"- Controllers: {s.get('controllers') or 'not given'}",
|
||||
f"- Consent version: {s.get('consent_version')}",
|
||||
f"- Tool: {s.get('tool')}",
|
||||
f"- Size: {takes.human_bytes(nbytes)}",
|
||||
f"- SHA256 of SHA256SUMS: {sha}", "",
|
||||
"Checked with hands/rec/validate.py: no errors" + (f", {len(report.warnings)} warnings:"
|
||||
if report.warnings else ".")]
|
||||
lines += [f"- {w}" for w in report.warnings]
|
||||
return message, "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- talking to the Hub
|
||||
|
||||
def _hf():
|
||||
os.environ.update({k: v for k, v in HF_ENV.items() if k not in os.environ})
|
||||
try:
|
||||
import huggingface_hub
|
||||
except ImportError:
|
||||
raise HubError("missing", "huggingface_hub isn't installed in the dev container: run setup/dev-container.sh "
|
||||
"(or: pip install --user huggingface_hub)") from None
|
||||
return huggingface_hub
|
||||
|
||||
|
||||
def explain(e, repo):
|
||||
"""A huggingface_hub (or network) exception as a HubError."""
|
||||
if isinstance(e, HubError):
|
||||
return e
|
||||
try:
|
||||
from huggingface_hub import errors as hf
|
||||
except ImportError:
|
||||
hf = None
|
||||
url = dataset_url(repo)
|
||||
if hf is not None:
|
||||
if isinstance(e, hf.LocalTokenNotFoundError):
|
||||
return HubError("login", "You're not logged in to Hugging Face. Log in with hf auth login (see below).")
|
||||
if isinstance(e, hf.GatedRepoError):
|
||||
return HubError("terms", f"Accept the dataset's terms first: open {url}, read them and accept them, "
|
||||
"then try again.", url)
|
||||
if isinstance(e, hf.RepositoryNotFoundError):
|
||||
return HubError("not-found", f"The dataset {repo} wasn't found, or it's private and your account has "
|
||||
"no access to it.", url)
|
||||
if isinstance(e, hf.HfHubHTTPError):
|
||||
status = getattr(getattr(e, "response", None), "status_code", None)
|
||||
first = str(e).strip().splitlines()[0] if str(e).strip() else type(e).__name__
|
||||
if status == 401:
|
||||
return HubError("login", "Hugging Face didn't accept your token (it may have been deleted or have "
|
||||
"expired). Log in again with hf auth login.", TOKENS_URL)
|
||||
if status == 403:
|
||||
return HubError("permission", "Your token isn't allowed to open a pull request. Create a token with "
|
||||
"the Write role and log in again with it: hf auth login --force.",
|
||||
TOKENS_URL)
|
||||
return HubError("hub", f"Hugging Face refused the upload ({status or 'no status'}): {first}")
|
||||
try:
|
||||
import httpx
|
||||
net = (httpx.TransportError, OSError)
|
||||
except ImportError:
|
||||
net = (OSError,)
|
||||
if isinstance(e, net):
|
||||
return HubError("network", f"Couldn't reach huggingface.co ({type(e).__name__}: {e}). Check the network "
|
||||
"and try again: files already sent usually aren't sent twice.")
|
||||
return HubError("hub", f"{type(e).__name__}: {e}")
|
||||
|
||||
|
||||
def whoami():
|
||||
"""{"name", "role"} for the saved login; HubError if there's none or it doesn't work.
|
||||
role: "write", "read", "fineGrained" or ""."""
|
||||
hf = _hf()
|
||||
try:
|
||||
info = hf.HfApi().whoami()
|
||||
except Exception as e:
|
||||
raise explain(e, dataset_id()) from None
|
||||
token = ((info.get("auth") or {}).get("accessToken") or {})
|
||||
return {"name": info.get("name", ""), "role": token.get("role", "")}
|
||||
|
||||
|
||||
def upload(store, session, dry_run=False, again=False, progress=None, log=None):
|
||||
"""Upload exports/<session> (the steps in this file's docstring). progress(phase, text,
|
||||
fraction or None); log(text) for the dry run's account. Returns the result; raises HubError."""
|
||||
tell = progress or (lambda phase, text, fraction=None: None)
|
||||
say = log or (lambda text: None)
|
||||
import validate
|
||||
repo = dataset_id()
|
||||
try:
|
||||
export = store.export_dir(session)
|
||||
except ValueError as e:
|
||||
raise HubError("no-export", str(e)) from None
|
||||
if not os.path.isfile(os.path.join(export, "manifest.json")):
|
||||
raise HubError("no-export", f"No export of session {session}: export it first")
|
||||
|
||||
tell("check", "Checking the export", 0.0)
|
||||
report = validate.validate(export, progress=lambda f, text: tell("check", text, f))
|
||||
if not report.ok:
|
||||
raise HubError("invalid", f"The export has {len(report.errors)} problems, so it can't be uploaded. Export "
|
||||
"the session again; if that doesn't help, report it.", errors=report.errors)
|
||||
summary = report.summary
|
||||
contributor = summary.get("contributor", "")
|
||||
sha = export_sha(export)
|
||||
before = previous_upload(store, session, sha)
|
||||
if before and not again:
|
||||
raise HubError("duplicate", f"This export was uploaded already, on {before.get('uploaded', '?')}: "
|
||||
f"{before.get('pr_url', '')}", before.get("pr_url", ""), extra={"previous": before})
|
||||
if not dry_run and not upload_allowed():
|
||||
raise HubError("closed", "Contributions aren't open yet: the texts are drafts waiting for a legal review. "
|
||||
f"(For a rehearsal against a test repo: {ALLOW_ENV}=1.)")
|
||||
path_in_repo = f"contributions/{contributor}/{session}"
|
||||
message, description = describe(summary, report, sha, report.bytes)
|
||||
files = []
|
||||
for root, _, names in os.walk(export):
|
||||
files += [os.path.relpath(os.path.join(root, n), export) for n in names]
|
||||
plan = {"repo": repo, "repo_type": "dataset", "folder_path": export, "path_in_repo": path_in_repo,
|
||||
"create_pr": True, "commit_message": message, "commit_description": description,
|
||||
"files": len(files), "bytes": report.bytes, "warnings": report.warnings}
|
||||
|
||||
if dry_run:
|
||||
say(f"dry run: would check the login (whoami) and access to {repo} (auth_check)")
|
||||
say(f"dry run: would upload_folder {len(files)} files, {takes.human_bytes(report.bytes)}, "
|
||||
f"from {export} to datasets/{repo}/{path_in_repo}, as a pull request")
|
||||
for rel in sorted(files):
|
||||
say(f" {rel} {takes.human_bytes(os.path.getsize(os.path.join(export, rel)))}")
|
||||
say(f"dry run: commit message: {message}")
|
||||
say("dry run: commit description:\n" + description.rstrip())
|
||||
record = {"repo": repo, "pr_url": "", "uploaded": now_iso(), "export_sha": sha}
|
||||
say(f"dry run: would record in session.json's uploads: {json.dumps(record)}")
|
||||
tell("done", "Dry run: nothing was uploaded", 1.0)
|
||||
return dict(plan, dry_run=True, pr_url="", record=record)
|
||||
|
||||
hf = _hf()
|
||||
api = hf.HfApi()
|
||||
tell("login", "Checking your Hugging Face login", None)
|
||||
try:
|
||||
who = api.whoami()
|
||||
except Exception as e:
|
||||
raise explain(e, repo) from None
|
||||
role = ((who.get("auth") or {}).get("accessToken") or {}).get("role", "")
|
||||
if role == "read":
|
||||
raise HubError("read-token", "Your saved token can only read, so it can't open a pull request. Create one "
|
||||
"with the Write role and log in again with it: hf auth login --force.",
|
||||
TOKENS_URL)
|
||||
tell("access", f"Checking access to {repo}", None)
|
||||
try:
|
||||
api.auth_check(repo, repo_type="dataset")
|
||||
except Exception as e:
|
||||
raise explain(e, repo) from None
|
||||
tell("upload", f"Uploading {len(files)} files ({takes.human_bytes(report.bytes)}) and opening a pull request",
|
||||
None)
|
||||
try:
|
||||
info = api.upload_folder(repo_id=repo, repo_type="dataset", folder_path=export, path_in_repo=path_in_repo,
|
||||
create_pr=True, commit_message=message, commit_description=description)
|
||||
except Exception as e:
|
||||
raise explain(e, repo) from None
|
||||
pr_url = getattr(info, "pr_url", None) or ""
|
||||
record = {"repo": repo, "pr_url": pr_url, "uploaded": now_iso(), "export_sha": sha}
|
||||
record_upload(store, session, record)
|
||||
tell("done", "Uploaded", 1.0)
|
||||
return dict(plan, dry_run=False, pr_url=pr_url, user=who.get("name", ""), record=record)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the command line
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Upload a hand recorder export to the hand dataset on Hugging Face.")
|
||||
ap.add_argument("--base", default=takes.DEFAULT_BASE)
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
w = sub.add_parser("whoami", help="show the saved Hugging Face login")
|
||||
w.add_argument("--json", action="store_true")
|
||||
u = sub.add_parser("upload", help="upload exports/SESSION as a pull request")
|
||||
u.add_argument("session")
|
||||
u.add_argument("--dry-run", action="store_true", help="no network: say what would be uploaded")
|
||||
u.add_argument("--again", action="store_true", help="upload even if this export was uploaded before")
|
||||
u.add_argument("--json", action="store_true", help="JSON lines, for the window")
|
||||
a = ap.parse_args()
|
||||
|
||||
def emit(obj):
|
||||
print(json.dumps(obj), flush=True)
|
||||
|
||||
try:
|
||||
if a.cmd == "whoami":
|
||||
who = whoami()
|
||||
if a.json:
|
||||
emit({"done": who})
|
||||
else:
|
||||
print(f"logged in as {who['name']} (token role: {who['role'] or 'unknown'})")
|
||||
return 0
|
||||
if a.json:
|
||||
def progress(phase, text, fraction=None):
|
||||
emit({"phase": phase, "text": text, "fraction": fraction})
|
||||
|
||||
def log(text):
|
||||
emit({"log": text})
|
||||
else:
|
||||
last = {}
|
||||
|
||||
def progress(phase, text, fraction=None):
|
||||
line = text if fraction is None else f"{fraction * 100:5.1f}% {text}"
|
||||
if (phase, text) != last.get("key") or fraction in (0.0, 1.0):
|
||||
print(line, file=sys.stderr, flush=True)
|
||||
last["key"] = (phase, text)
|
||||
|
||||
def log(text):
|
||||
print(text, flush=True)
|
||||
if hasattr(os, "setpriority"):
|
||||
try:
|
||||
os.setpriority(os.PRIO_PROCESS, 0, 19) # validation reads and hashes everything: stay out of VR's way
|
||||
except OSError:
|
||||
pass
|
||||
result = upload(takes.Store(a.base), a.session, dry_run=a.dry_run, again=a.again, progress=progress, log=log)
|
||||
if a.json:
|
||||
emit({"done": result})
|
||||
elif result["dry_run"]:
|
||||
print(f"dry run done: {result['files']} files would go to datasets/{result['repo']}/{result['path_in_repo']}")
|
||||
else:
|
||||
print(f"uploaded: {result['pr_url']}")
|
||||
return 0
|
||||
except HubError as e:
|
||||
if a.json:
|
||||
emit({"error": e.as_dict()})
|
||||
else:
|
||||
print(f"error ({e.kind}): {e.text}", file=sys.stderr)
|
||||
for line in e.errors:
|
||||
print(f" {line}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+261
-10
@@ -117,6 +117,10 @@ Kirigami.ApplicationWindow {
|
||||
close.accepted = false
|
||||
confirm.ask("Stop the session?", "A session is recording. Closing stops it; what's recorded so far is kept.",
|
||||
() => { root.quitting = true; Qt.quit() })
|
||||
} else if (backend.uploading && !quitting) {
|
||||
close.accepted = false
|
||||
confirm.ask("Cancel the upload?", "An upload is running. Closing cancels it; you can start it again later.",
|
||||
() => { root.quitting = true; Qt.quit() })
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1028,7 +1032,7 @@ Kirigami.ApplicationWindow {
|
||||
RowLayout {
|
||||
visible: exportView.chosen !== null && exportView.chosen.exported && !backend.exporting
|
||||
Controls.Button {
|
||||
text: "Upload instructions"
|
||||
text: "Upload"
|
||||
icon.name: "go-next"
|
||||
onClicked: root.show(uploadPage)
|
||||
}
|
||||
@@ -1050,7 +1054,7 @@ Kirigami.ApplicationWindow {
|
||||
opacity: 0.7
|
||||
text: "Export leaves out the ranges you deleted, compresses the rest and adds a manifest and "
|
||||
+ "checksums, in " + backend.exportsDir + ". It runs at the lowest priority "
|
||||
+ "and takes a few minutes per round. Nothing is uploaded: the Upload page explains how."
|
||||
+ "and takes a few minutes per round. Nothing is uploaded until you press Upload on the Upload page."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1063,8 +1067,68 @@ Kirigami.ApplicationWindow {
|
||||
title: "Upload"
|
||||
readonly property var exported: backend.sessions.filter(s => s.exported)
|
||||
readonly property string session: exportBox.currentIndex >= 0 ? exportBox.currentValue || "" : ""
|
||||
readonly property var chosen: exportBox.currentIndex >= 0 && exportBox.currentIndex < exported.length
|
||||
? exported[exportBox.currentIndex] : null
|
||||
readonly property var up: backend.upload
|
||||
readonly property var upError: up.error || ({})
|
||||
readonly property var upResult: up.result || ({})
|
||||
// The upload shown is this export's (the window keeps the last one's result).
|
||||
readonly property bool mine: up.session === session
|
||||
property var info: ({ previous: {}, uploads: 0 })
|
||||
readonly property bool loggedIn: backend.login.state === "ok"
|
||||
// Why Upload is off, or "" when it can go.
|
||||
readonly property string blocked: {
|
||||
if (session === "") return "Choose an export"
|
||||
if (chosen && chosen.export_stale) return "Export the session again first"
|
||||
if (backend.exporting) return "Wait for the export to finish"
|
||||
if (backend.uploading) return ""
|
||||
if (backend.hubDryRun) return ""
|
||||
if (!backend.uploadAllowed) return "Contributions aren't open yet"
|
||||
if (!loggedIn) return "Log in to Hugging Face first (below)"
|
||||
return ""
|
||||
}
|
||||
|
||||
header: DraftBanner { }
|
||||
function refreshInfo() { info = backend.uploadInfo(session) }
|
||||
onSessionChanged: refreshInfo()
|
||||
Connections {
|
||||
target: backend
|
||||
function onSessionsChanged() { uploadView.refreshInfo() }
|
||||
}
|
||||
Component.onCompleted: {
|
||||
refreshInfo()
|
||||
if (backend.login.state === "unknown") backend.checkLogin()
|
||||
}
|
||||
|
||||
function startUpload() {
|
||||
if (info.previous && info.previous.export_sha)
|
||||
confirm.ask("Upload this export again?",
|
||||
"It was uploaded on " + info.previous.uploaded + " (" + (info.previous.pr_url || "no link")
|
||||
+ "). Uploading it again opens a second pull request.",
|
||||
() => backend.startUpload(uploadView.session, true))
|
||||
else
|
||||
backend.startUpload(session, false)
|
||||
}
|
||||
|
||||
header: ColumnLayout {
|
||||
spacing: 0
|
||||
DraftBanner { Layout.fillWidth: true }
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: backend.hubDryRun
|
||||
position: Kirigami.InlineMessage.Position.Header
|
||||
type: Kirigami.MessageType.Information
|
||||
text: "Dry run (--hub-dry-run): Upload checks the export and lists what it would send. "
|
||||
+ "Nothing goes over the network."
|
||||
}
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: backend.textsDraft && backend.allowUploadSet && !backend.hubDryRun
|
||||
position: Kirigami.InlineMessage.Position.Header
|
||||
type: Kirigami.MessageType.Warning
|
||||
text: "Rehearsal: FT_HANDREC_ALLOW_UPLOAD=1 allows uploads to " + backend.dataset
|
||||
+ " while the texts are drafts."
|
||||
}
|
||||
}
|
||||
|
||||
ColumnLayout {
|
||||
spacing: Kirigami.Units.largeSpacing
|
||||
@@ -1074,30 +1138,217 @@ Kirigami.ApplicationWindow {
|
||||
visible: uploadView.exported.length === 0
|
||||
icon.name: "document-export"
|
||||
text: "Nothing exported yet"
|
||||
explanation: "Review a session, then export it. Its upload instructions appear here."
|
||||
explanation: "Review a session, then export it. It can be uploaded from here."
|
||||
helpfulAction: Kirigami.Action {
|
||||
text: "Export"; icon.name: "document-export"
|
||||
onTriggered: root.show(exportPage)
|
||||
}
|
||||
}
|
||||
|
||||
RowLayout {
|
||||
Kirigami.FormLayout {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.exported.length > 0
|
||||
Controls.Label { text: "Export:" }
|
||||
Controls.ComboBox {
|
||||
id: exportBox
|
||||
Kirigami.FormData.label: "Export:"
|
||||
model: uploadView.exported
|
||||
textRole: "label"
|
||||
valueRole: "id"
|
||||
enabled: !backend.uploading
|
||||
Component.onCompleted: currentIndex = Math.max(0, indexOfValue(root.chosenSession))
|
||||
onActivated: root.chosenSession = currentValue
|
||||
}
|
||||
Controls.Label {
|
||||
Kirigami.FormData.label: "Size:"
|
||||
text: uploadView.chosen ? uploadView.chosen.exportText : ""
|
||||
}
|
||||
Controls.Label {
|
||||
Kirigami.FormData.label: "Goes to:"
|
||||
textFormat: Text.StyledText
|
||||
text: "<a href=\"" + backend.datasetUrl + "\">" + backend.dataset + "</a>, as a pull request"
|
||||
onLinkActivated: link => Qt.openUrlExternally(link)
|
||||
}
|
||||
RowLayout {
|
||||
Kirigami.FormData.label: "Hugging Face:"
|
||||
Controls.Label {
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 22
|
||||
wrapMode: Text.Wrap
|
||||
text: backend.login.state === "dry" ? "not checked in a dry run"
|
||||
: backend.login.state === "unknown" ? "not checked yet"
|
||||
: backend.login.state === "none" ? "not logged in"
|
||||
: backend.login.text || ""
|
||||
}
|
||||
Controls.Button {
|
||||
visible: backend.login.state !== "dry"
|
||||
text: "Check again"
|
||||
icon.name: "view-refresh"
|
||||
enabled: backend.login.state !== "checking"
|
||||
onClicked: backend.checkLogin()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.chosen !== null && uploadView.chosen.export_stale
|
||||
type: Kirigami.MessageType.Warning
|
||||
text: "This session changed since it was exported. Export it again so the upload has your latest deletions."
|
||||
}
|
||||
|
||||
// Not logged in: how to log in, in a terminal. The token never goes into this window.
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.session !== "" && ["none", "read", "error", "missing"].indexOf(backend.login.state) >= 0
|
||||
type: backend.login.state === "error" ? Kirigami.MessageType.Error : Kirigami.MessageType.Warning
|
||||
text: backend.login.state === "none"
|
||||
? "You're not logged in to Hugging Face. Open a terminal (Konsole) and run the command below. "
|
||||
+ "Paste a token with the Write role when it asks (steps 1 to 3 below say how to make one). "
|
||||
+ "The token goes only into that terminal, never into this window. Then press Check again."
|
||||
: backend.login.text || ""
|
||||
actions: backend.login.link ? [linkAction] : []
|
||||
Kirigami.Action {
|
||||
id: linkAction
|
||||
text: "Open"
|
||||
icon.name: "internet-services"
|
||||
onTriggered: Qt.openUrlExternally(backend.login.link)
|
||||
}
|
||||
}
|
||||
RowLayout {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.session !== "" && ["none", "read"].indexOf(backend.login.state) >= 0
|
||||
Controls.TextField {
|
||||
id: loginCommand
|
||||
Layout.fillWidth: true
|
||||
readOnly: true
|
||||
font.family: "monospace"
|
||||
text: backend.loginCommand + (backend.login.state === "read" ? " --force" : "")
|
||||
}
|
||||
Controls.Button {
|
||||
text: "Copy command"
|
||||
icon.name: "edit-copy"
|
||||
onClicked: backend.copy(loginCommand.text)
|
||||
}
|
||||
}
|
||||
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.session !== "" && uploadView.info.previous.export_sha !== undefined
|
||||
type: Kirigami.MessageType.Information
|
||||
text: "This export was uploaded on " + (uploadView.info.previous.uploaded || "") + ": "
|
||||
+ (uploadView.info.previous.pr_url || "") + ". There's no need to upload it again."
|
||||
}
|
||||
|
||||
RowLayout {
|
||||
visible: uploadView.session !== ""
|
||||
Controls.Button {
|
||||
text: backend.hubDryRun ? "Upload (dry run)" : "Upload"
|
||||
icon.name: "cloud-upload"
|
||||
enabled: uploadView.blocked === "" && !backend.uploading
|
||||
onClicked: uploadView.startUpload()
|
||||
}
|
||||
Controls.Button {
|
||||
visible: backend.uploading
|
||||
text: "Cancel"
|
||||
icon.name: "dialog-cancel"
|
||||
onClicked: backend.cancelUpload()
|
||||
}
|
||||
Controls.Label {
|
||||
visible: !backend.uploading && uploadView.blocked !== ""
|
||||
opacity: 0.7
|
||||
text: uploadView.blocked
|
||||
}
|
||||
}
|
||||
|
||||
// Progress: a share while the export is checked, a sweeping bar while it's sent
|
||||
// (huggingface_hub doesn't report progress). Drawn here, as on the Export page.
|
||||
Rectangle {
|
||||
id: track
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 20
|
||||
implicitHeight: Kirigami.Units.smallSpacing * 2
|
||||
visible: backend.uploading
|
||||
radius: height / 2
|
||||
clip: true
|
||||
color: Qt.rgba(Kirigami.Theme.textColor.r, Kirigami.Theme.textColor.g, Kirigami.Theme.textColor.b, 0.15)
|
||||
readonly property bool unknown: uploadView.up.fraction === undefined || uploadView.up.fraction < 0
|
||||
Rectangle {
|
||||
id: fill
|
||||
height: parent.height
|
||||
radius: parent.radius
|
||||
color: Kirigami.Theme.highlightColor
|
||||
width: track.unknown ? parent.width / 4 : parent.width * Math.max(0, Math.min(1, uploadView.up.fraction))
|
||||
x: 0
|
||||
SequentialAnimation on x {
|
||||
running: track.visible && track.unknown
|
||||
loops: Animation.Infinite
|
||||
onRunningChanged: if (!running) fill.x = 0
|
||||
NumberAnimation { from: -fill.width; to: track.width; duration: 1600 }
|
||||
}
|
||||
}
|
||||
}
|
||||
Controls.Label {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.mine && (uploadView.up.text || "") !== ""
|
||||
&& (backend.uploading || uploadView.up.phase === "failed" && uploadView.upError.kind === "cancelled")
|
||||
wrapMode: Text.Wrap
|
||||
text: uploadView.up.text || ""
|
||||
}
|
||||
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: !backend.uploading && uploadView.mine && uploadView.up.phase === "done"
|
||||
type: Kirigami.MessageType.Positive
|
||||
text: uploadView.upResult.dry_run
|
||||
? "Dry run: the export passed its checks. It would go to " + uploadView.upResult.repo + "/"
|
||||
+ uploadView.upResult.path_in_repo + " (" + uploadView.upResult.files + " files). Nothing was sent."
|
||||
: "Uploaded. Your pull request: " + (uploadView.upResult.pr_url || "")
|
||||
+ ". The maintainer reviews it before it joins the dataset."
|
||||
actions: uploadView.upResult.pr_url ? [openPr, copyPr] : []
|
||||
Kirigami.Action {
|
||||
id: openPr
|
||||
text: "Open"
|
||||
icon.name: "internet-services"
|
||||
onTriggered: Qt.openUrlExternally(uploadView.upResult.pr_url)
|
||||
}
|
||||
Kirigami.Action {
|
||||
id: copyPr
|
||||
text: "Copy link"
|
||||
icon.name: "edit-copy"
|
||||
onTriggered: backend.copy(uploadView.upResult.pr_url)
|
||||
}
|
||||
}
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.exported.length > 0 && exportBox.currentIndex >= 0
|
||||
&& uploadView.exported[exportBox.currentIndex].export_stale
|
||||
type: Kirigami.MessageType.Warning
|
||||
text: "This session changed since it was exported. Export it again so the upload has your latest deletions."
|
||||
visible: !backend.uploading && uploadView.mine && uploadView.up.phase === "failed"
|
||||
&& uploadView.upError.kind !== "cancelled"
|
||||
type: Kirigami.MessageType.Error
|
||||
text: (uploadView.upError.text || "")
|
||||
+ ((uploadView.upError.errors || []).length ? "\n\n• " + uploadView.upError.errors.join("\n• ") : "")
|
||||
actions: uploadView.upError.link ? [openErrorLink] : []
|
||||
Kirigami.Action {
|
||||
id: openErrorLink
|
||||
text: "Open"
|
||||
icon.name: "internet-services"
|
||||
onTriggered: Qt.openUrlExternally(uploadView.upError.link)
|
||||
}
|
||||
}
|
||||
Controls.TextArea {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.mine && (uploadView.up.log || "") !== ""
|
||||
readOnly: true
|
||||
selectByMouse: true
|
||||
wrapMode: Text.WrapAnywhere
|
||||
font.family: "monospace"
|
||||
text: uploadView.up.log || ""
|
||||
}
|
||||
|
||||
Kirigami.Separator {
|
||||
Layout.fillWidth: true
|
||||
visible: uploadView.session !== ""
|
||||
}
|
||||
Kirigami.Heading {
|
||||
level: 3
|
||||
visible: uploadView.session !== ""
|
||||
text: "Or upload from a terminal"
|
||||
}
|
||||
RowLayout {
|
||||
Layout.fillWidth: true
|
||||
|
||||
Executable
+207
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env bash
|
||||
# The hand recorder end to end, without the headset (DESIGN.md "Upload"): a short stretch of
|
||||
# an ft-hands recording plays into a camera ring (ft-ringplay), a session records it with a
|
||||
# short test script (session.py, the panel as ft-handpanel --no-vr, a tracking ft-hands),
|
||||
# takes.py exports it, validate.py checks the export, and hub.py uploads it: a dry run by
|
||||
# default, or for real to a (test) dataset with --repo.
|
||||
#
|
||||
# hands/rec/rehearse.sh [--repo ID] [--capture DIR] [--from S] [--prompt-seconds N] [--keep]
|
||||
#
|
||||
# --repo ID upload for real to this dataset, as a pull request. While the texts are
|
||||
# drafts that also needs FT_HANDREC_ALLOW_UPLOAD=1 in the environment.
|
||||
# --capture DIR the recording to play (default the frame-hands capture
|
||||
# rec-20260930-103803-lit); --from S: start this far into it (60)
|
||||
# --prompt-seconds N each test prompt's length (3): about 11 s are recorded in all, a few
|
||||
# hundred MB before compression
|
||||
# --keep keep the temporary folders (camera images of a room: delete them after)
|
||||
#
|
||||
# It runs in the dev container (where ft-hands, ft-handpanel, zstd and huggingface_hub are),
|
||||
# and in one frame-job scope when frame-job is installed, so it stays capped while the headset
|
||||
# is worn. Everything goes in temporary folders that are deleted at the end, and every process
|
||||
# it started is stopped, also on Ctrl+C. A tracking ft-hands that's already running is used as
|
||||
# it is; otherwise one is started for the rehearsal (it publishes the usual hands file).
|
||||
set -euo pipefail
|
||||
here=$(cd "$(dirname "$(readlink -f "$0")")" && pwd)
|
||||
root=$(cd "$here/../.." && pwd)
|
||||
uid=$(id -u)
|
||||
|
||||
if [ ! -e /run/.containerenv ]; then
|
||||
"$root/scripts/container-up.sh"
|
||||
# Ctrl+C and kill don't reach through distrobox: the rehearsal inside writes its PID to this
|
||||
# file, and a signal here is passed on to it, so it still cleans up.
|
||||
pidfile=$(mktemp "/run/user/$uid/ft-handrec-rehearse-pid.XXXXXX")
|
||||
# The real user bus, so frame-job's systemd-run reaches the host's user manager.
|
||||
"$HOME/.local/bin/distrobox" enter dev -- env DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$uid/bus" \
|
||||
XDG_RUNTIME_DIR="/run/user/$uid" FT_REHEARSE_PIDFILE="$pidfile" bash "$here/rehearse.sh" "$@" </dev/null &
|
||||
child=$! stopping=0 status=0
|
||||
pass_on() { [ -s "$pidfile" ] && kill -TERM "$(cat "$pidfile")" 2>/dev/null; return 0; }
|
||||
trap 'stopping=1; pass_on' INT TERM HUP
|
||||
while kill -0 "$child" 2>/dev/null; do
|
||||
wait "$child" && status=0 || status=$? # (set -e would end this on a signal)
|
||||
[ "$stopping" = 0 ] || pass_on
|
||||
done
|
||||
rm -f "$pidfile"
|
||||
exit "$status"
|
||||
fi
|
||||
if [ -z "${FT_REHEARSE_SCOPE:-}" ] && command -v frame-job >/dev/null; then
|
||||
export FT_REHEARSE_SCOPE=1
|
||||
cd /tmp # no .frame-job up from here: it runs locally, capped
|
||||
exec frame-job --local -- bash "$here/rehearse.sh" "$@"
|
||||
fi
|
||||
[ -z "${FT_REHEARSE_PIDFILE:-}" ] || echo $$ >"$FT_REHEARSE_PIDFILE"
|
||||
|
||||
repo="" capture="$HOME/Desktop/Projects/frame-hands/captures/rec-20260930-103803-lit" from=60 prompt_s=3 keep=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--repo) repo=$2; shift 2 ;;
|
||||
--capture) capture=$2; shift 2 ;;
|
||||
--from) from=$2; shift 2 ;;
|
||||
--prompt-seconds) prompt_s=$2; shift 2 ;;
|
||||
--keep) keep=1; shift ;;
|
||||
-h|--help) sed -n '2,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "rehearse: unknown option $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
ringplay=$root/hands/build/ft-ringplay
|
||||
hands=$root/hands/build/ft-hands
|
||||
panel=$here/build/ft-handpanel
|
||||
for b in "$ringplay" "$hands"; do
|
||||
[ -x "$b" ] || { echo "rehearse: $b isn't built: hands/build.sh --tools" >&2; exit 1; }
|
||||
done
|
||||
[ -x "$panel" ] || { echo "rehearse: $panel isn't built: hands/rec/build.sh" >&2; exit 1; }
|
||||
[ -f "$capture/sets.bin" ] || { echo "rehearse: no recording in $capture" >&2; exit 1; }
|
||||
if pgrep -x ft-handpanel >/dev/null; then
|
||||
echo "rehearse: an ft-handpanel is running (a real session?): try again when it's done" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "$repo" ]; then
|
||||
if python3 -c "import sys; sys.path.insert(0, '$here'); import hub; sys.exit(0 if hub.upload_allowed() else 1)"; then :; else
|
||||
echo "rehearse: the texts are drafts: a real upload needs FT_HANDREC_ALLOW_UPLOAD=1 as well as --repo" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 -c "import huggingface_hub" 2>/dev/null || {
|
||||
echo "rehearse: huggingface_hub isn't installed: setup/dev-container.sh" >&2; exit 1; }
|
||||
fi
|
||||
|
||||
# The data: the ring in the runtime folder (memory), the rest in /tmp. Both go at the end.
|
||||
work=$(mktemp -d "${TMPDIR:-/tmp}/ft-handrec-rehearse.XXXXXX")
|
||||
ringdir=$(mktemp -d "/run/user/$uid/ft-handrec-rehearse.XXXXXX")
|
||||
base=$work/base ring=$ringdir/cam-ring logs=$work/logs
|
||||
mkdir -p "$base" "$logs"
|
||||
pids=()
|
||||
cleanup() {
|
||||
local code=$?
|
||||
# Finish even if more signals come: frame-job stops its scope with SIGTERM to every process
|
||||
# in it when it's interrupted.
|
||||
trap '' INT TERM HUP
|
||||
trap - EXIT
|
||||
for p in "${pids[@]}"; do kill "$p" 2>/dev/null || true; done
|
||||
for p in "${pids[@]}"; do wait "$p" 2>/dev/null || true; done
|
||||
if [ "$keep" = 1 ]; then
|
||||
echo "kept: $work and $ringdir (camera images of a room: delete them when done)"
|
||||
else
|
||||
rm -rf "$work" "$ringdir"
|
||||
fi
|
||||
exit "$code"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 130' INT TERM
|
||||
t0=$(date +%s)
|
||||
step() { echo; echo "== $*"; }
|
||||
# Long steps run in the background and are waited for: a signal then ends the wait at once,
|
||||
# and cleanup stops them (bash runs traps only after a foreground command ends).
|
||||
run() { "$@" & local p=$!; pids+=("$p"); wait "$p"; }
|
||||
indent() { sed 's/^/ /'; }
|
||||
|
||||
step "1. Playing ${capture##*/} from ${from} s into $ring"
|
||||
"$ringplay" "$capture" --ring "$ring" --from "$from" --to "$((from + 30))" --loop >"$logs/ringplay.log" 2>&1 &
|
||||
pids+=($!)
|
||||
for _ in $(seq 100); do
|
||||
python3 -c "import sys; sys.path.insert(0, '$here'); import session
|
||||
r = session.Ring('$ring'); sys.exit(0 if r.alive() else 1)" 2>/dev/null && break
|
||||
sleep 0.2
|
||||
done
|
||||
python3 -c "import sys; sys.path.insert(0, '$here'); import session
|
||||
r = session.Ring('$ring'); print(' cameras:', ', '.join('%s %dx%d' % (c['name'], c['width'], c['height']) for c in r.cams))
|
||||
sys.exit(0 if r.alive() else 1)" || { echo "rehearse: ft-ringplay doesn't publish: $(tail -3 "$logs/ringplay.log")" >&2; exit 1; }
|
||||
|
||||
step "2. Session (test script, panel --no-vr)"
|
||||
tracker=existing
|
||||
if ! python3 -c "import sys; sys.path.insert(0, '$here'); import session; sys.exit(0 if session.tracker_running() else 1)"; then
|
||||
"$hands" --ring "$ring" --no-gestures --status 0 >"$logs/tracker.log" 2>&1 &
|
||||
pids+=($!)
|
||||
tracker=started
|
||||
fi
|
||||
echo " tracker: $tracker"
|
||||
"$panel" --no-vr >"$logs/panel.log" 2>&1 &
|
||||
pids+=($!)
|
||||
python3 - "$here" "$base" "$prompt_s" "$work/script.json" <<'EOF'
|
||||
import json, os, re, sys, uuid
|
||||
here, base, secs, out = sys.argv[1], sys.argv[2], float(sys.argv[3]), sys.argv[4]
|
||||
consent = re.search(r"^Version:\s*(\S+)", open(os.path.join(here, "CONSENT.md")).read(), re.M).group(1)
|
||||
json.dump({"schema": 1, "contributor": str(uuid.uuid4()),
|
||||
"consent": {"version": consent, "accepted": "2026-10-02T00:00:00+00:00", "adult": True},
|
||||
"optional": {"handedness": "", "notes": ""}}, open(os.path.join(base, "profile.json"), "w"))
|
||||
json.dump({"version": 1, "intro_s": 1, "between_s": 1,
|
||||
"welcome": {"title": "Rehearsal", "seconds": 1, "text": "A rehearsal of the hand recorder."},
|
||||
"done": {"title": "Done", "seconds": 1, "text": "Rehearsal done."},
|
||||
"stopped": {"title": "Stopped", "seconds": 1, "text": "Rehearsal stopped."},
|
||||
"sections": [
|
||||
{"id": "hand-size", "title": "Hand size", "intro": "Rehearsal: hands.",
|
||||
"prompts": [{"text": "Both hands flat, palms toward you.", "seconds": secs, "hands": "both",
|
||||
"pose": "flat", "distance": "mid"},
|
||||
{"text": "Fingers spread wide.", "seconds": secs, "hands": "both", "pose": "spread"}]},
|
||||
{"id": "no-hands", "title": "No hands", "intro": "",
|
||||
"prompts": [{"text": "Hands out of view.", "seconds": secs, "hands": "none"}]}]},
|
||||
open(out, "w"))
|
||||
EOF
|
||||
for _ in $(seq 50); do
|
||||
python3 -c "import sys; sys.path.insert(0, '$here'); import session
|
||||
sys.exit(0 if session.Panel().cmd('ping', reply=True) else 1)" 2>/dev/null && break
|
||||
sleep 0.2
|
||||
done
|
||||
run python3 "$here/session.py" --ring "$ring" --no-start --base "$base" --script "$work/script.json" --lighting room \
|
||||
</dev/null > >(indent)
|
||||
sid=$(ls "$base/sessions" | tail -1)
|
||||
[ -n "$sid" ] || { echo "rehearse: the session left nothing" >&2; exit 1; }
|
||||
python3 "$here/takes.py" --base "$base" takes "$sid" | sed 's/^/ /'
|
||||
echo " panel pictures: $(grep -c '^--- picture' "$logs/panel.log" || true)"
|
||||
|
||||
step "3. Export"
|
||||
run python3 "$here/takes.py" --base "$base" export "$sid" >"$logs/export.log"
|
||||
tr '\r' '\n' <"$logs/export.log" | tail -1 | indent
|
||||
export_dir=$base/exports/$sid
|
||||
|
||||
step "4. Validate"
|
||||
valid=1
|
||||
run python3 "$here/validate.py" "$export_dir" --json >"$work/validate.json" || valid=0
|
||||
python3 -c "import json, sys; v = json.load(open(sys.argv[1]))
|
||||
for k in ('errors', 'warnings'):
|
||||
for line in v[k]: print(' %s: %s' % (k[:-1], line))
|
||||
print(' OK' if v['ok'] else ' %d errors' % len(v['errors']))" "$work/validate.json"
|
||||
[ "$valid" = 1 ] || { echo "rehearse: the export didn't validate" >&2; exit 1; }
|
||||
|
||||
if [ -n "$repo" ]; then
|
||||
step "5. Upload to $repo (for real)"
|
||||
run env FT_HANDREC_DATASET="$repo" python3 "$here/hub.py" --base "$base" upload "$sid" >"$logs/hub.log"
|
||||
indent <"$logs/hub.log"
|
||||
else
|
||||
step "5. Upload (dry run: --repo ID uploads for real)"
|
||||
run python3 "$here/hub.py" --base "$base" upload "$sid" --dry-run >"$logs/hub.log"
|
||||
indent <"$logs/hub.log"
|
||||
fi
|
||||
|
||||
step "Summary"
|
||||
python3 - "$work/validate.json" "$logs/hub.log" "$tracker" "$(( $(date +%s) - t0 ))" <<'EOF'
|
||||
import json, sys
|
||||
v = json.load(open(sys.argv[1]))
|
||||
s = v["summary"]
|
||||
hub = open(sys.argv[2]).read().strip().splitlines()
|
||||
print(f" session {s['session']}: {s['takes']} takes, {s['sets']} sets, {s['minutes']} min recorded")
|
||||
print(f" export: {v['bytes'] / 1e6:.1f} MB, {len(v['errors'])} errors, {len(v['warnings'])} warnings")
|
||||
for w in v["warnings"]:
|
||||
print(f" warning: {w}")
|
||||
print(f" hub: {hub[-1] if hub else '-'}")
|
||||
print(f" tracker: {sys.argv[3]}; took {sys.argv[4]} s")
|
||||
EOF
|
||||
+21
-1
@@ -828,7 +828,7 @@ class Session:
|
||||
ring.close()
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
removed = self._write_calibration()
|
||||
removed = self._write_calibration() + self._write_device()
|
||||
self._session_json = {
|
||||
"schema": 1, "tool": "ft-handrec " + _git_describe(), "started": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"contributor": self.profile.get("contributor", ""),
|
||||
@@ -863,6 +863,26 @@ class Session:
|
||||
write_json(os.path.join(self.session_dir, "calibration.json"), clean)
|
||||
return removed
|
||||
|
||||
def _write_device(self):
|
||||
"""device.json: the rig's pose in the CAD frame from /persist/device_config.json, only
|
||||
cv.cad_from_cal (Cam0 in CAD) and head (the head in CAD), in the shape the labeller reads
|
||||
(frame-hands train/label, as its cut.py writes it). The rest of that file names the unit
|
||||
(serial number, EDID). Returns what was removed, as "device.json:<path>"."""
|
||||
src = host_path("/persist/device_config.json")
|
||||
if not src:
|
||||
self._log("no /persist/device_config.json: no device.json")
|
||||
return []
|
||||
try:
|
||||
with open(src) as f:
|
||||
dev = json.load(f)
|
||||
picked = {"cv": {"cad_from_cal": dev["cv"]["cad_from_cal"]}, "head": dev["head"]}
|
||||
except (OSError, ValueError, KeyError, TypeError) as e:
|
||||
self._log("device_config.json unreadable (%s): no device.json" % e)
|
||||
return []
|
||||
clean, removed = strip_calibration(picked)
|
||||
write_json(os.path.join(self.session_dir, "device.json"), clean)
|
||||
return ["device.json:" + r for r in removed]
|
||||
|
||||
def _save_session(self):
|
||||
if self._session_json is not None:
|
||||
write_json(os.path.join(self.session_dir, "session.json"), self._session_json)
|
||||
|
||||
+7
-3
@@ -33,7 +33,7 @@ HDR = struct.Struct("<8sII") # fh_set_hdr_t: magic, ncams, bytes (the whol
|
||||
CAM = struct.Struct("<16sIIQQ") # fh_set_cam_t: name, width, height, capture_ns, dqbuf_ns
|
||||
MAX_CAMS = 16 # SetReader's limit
|
||||
PART_RE = re.compile(r"^sets(?:-(\d+))?\.bin$")
|
||||
SESSION_RE = re.compile(r"^\d{8}-\d{6}$")
|
||||
SESSION_RE = re.compile(r"^\d{8}-\d{6}(?:-\d+)?$") # session.py adds -2, -3 to a second one in a second
|
||||
EXPORT_SCHEMA = 1
|
||||
# zstd as DESIGN.md has it: level 10, two threads, at the lowest CPU priority (CPU work while
|
||||
# someone is in VR makes the headset stutter).
|
||||
@@ -415,6 +415,7 @@ class Store:
|
||||
def _export(self, session, src, work, zstd, report, cancelled, keep_notes):
|
||||
profile = self.profile()
|
||||
meta = read_json(os.path.join(src, "session.json"))
|
||||
meta.pop("uploads", None) # earlier uploads' records (hub.py) aren't part of the contribution
|
||||
takes = self.takes(session)
|
||||
total = sum(t["bytes"] for t in takes) or 1
|
||||
done = 0
|
||||
@@ -447,8 +448,9 @@ class Store:
|
||||
take_entries.append(entry)
|
||||
if cancelled():
|
||||
raise Cancelled()
|
||||
if os.path.isfile(os.path.join(src, "calibration.json")):
|
||||
shutil.copyfile(os.path.join(src, "calibration.json"), os.path.join(work, "calibration.json"))
|
||||
for name in ("calibration.json", "device.json"):
|
||||
if os.path.isfile(os.path.join(src, name)):
|
||||
shutil.copyfile(os.path.join(src, name), os.path.join(work, name))
|
||||
shown_profile = json.loads(json.dumps(profile))
|
||||
if not keep_notes and isinstance(shown_profile.get("optional"), dict):
|
||||
shown_profile["optional"].pop("notes", None)
|
||||
@@ -491,6 +493,8 @@ class Store:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
raise
|
||||
finally:
|
||||
proc.stderr.close()
|
||||
return raw
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,409 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for validate.py and hub.py (no network): a good export, and broken ones.
|
||||
|
||||
python3 hands/rec/tests/test_validate.py (in the dev container, or anywhere with zstd)
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest import mock
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, os.path.dirname(HERE))
|
||||
import hub # noqa: E402
|
||||
import session # noqa: E402
|
||||
import takes # noqa: E402
|
||||
import validate # noqa: E402
|
||||
|
||||
SESSION = "20261002-101500"
|
||||
DEVICE = {"cv": {"cad_from_cal": {"method": "FrontAndUpperCamPositions", "plus_x": [0.63, -0.77, 0.02],
|
||||
"plus_z": [-0.45, -0.35, 0.82], "position": [-0.05, -0.02, 0.04]}},
|
||||
"head": {"plus_x": [-1, 0, 0], "plus_z": [0, 0, -1], "position": [0, 0, 0]}}
|
||||
CAMS = [("slam_left", 32, 24), ("slam_right", 32, 24)]
|
||||
|
||||
|
||||
def fhset(t_ns, seed):
|
||||
"""One FHSET01 record: header, camera headers, pixels."""
|
||||
pixels = b"".join(bytes((seed + i) % 256 for i in range(w * h)) for _, w, h in CAMS)
|
||||
size = validate.HDR.size + validate.CAM.size * len(CAMS) + len(pixels)
|
||||
out = validate.HDR.pack(b"FHSET01\0", len(CAMS), size)
|
||||
for name, w, h in CAMS:
|
||||
out += validate.CAM.pack(name.encode(), w, h, t_ns - 1000, t_ns)
|
||||
return out + pixels
|
||||
|
||||
|
||||
def write_sums(export):
|
||||
sums = []
|
||||
for root, _, files in os.walk(export):
|
||||
for name in files:
|
||||
rel = os.path.relpath(os.path.join(root, name), export)
|
||||
if rel != "SHA256SUMS":
|
||||
with open(os.path.join(root, name), "rb") as f:
|
||||
sums.append((rel, hashlib.sha256(f.read()).hexdigest()))
|
||||
with open(os.path.join(export, "SHA256SUMS"), "w") as f:
|
||||
for rel, digest in sorted(sums):
|
||||
f.write(f"{digest} {rel}\n")
|
||||
|
||||
|
||||
def make_session(base, contributor=None, sid=None, device=True):
|
||||
"""A small recorded session in base, as session.py leaves it."""
|
||||
contributor = contributor or str(uuid.uuid4())
|
||||
os.makedirs(base, exist_ok=True)
|
||||
takes.write_json(os.path.join(base, "profile.json"), {
|
||||
"schema": 1, "contributor": contributor,
|
||||
"consent": {"version": "2026-10-02", "accepted": "2026-10-02T10:00:00+00:00", "adult": True},
|
||||
"optional": {"handedness": "right", "notes": ""}})
|
||||
sdir = os.path.join(base, "sessions", sid or SESSION)
|
||||
os.makedirs(os.path.join(sdir, "takes"))
|
||||
takes.write_json(os.path.join(sdir, "session.json"), {
|
||||
"schema": 1, "tool": "ft-handrec test", "started": "2026-10-02T10:15:00+0000", "contributor": contributor,
|
||||
"lighting": {"chosen": "room", "ring": {}},
|
||||
"checklist": {"objects": ["pencil"], "own_objects": ["stapler"], "controllers": "none", "sleeves": "",
|
||||
"rings": False, "watch": False, "notes": ""},
|
||||
"device": {"steamos": "3.8", "steamvr": "r25358740+28b72a4f-1", "cameras": [{"name": n, "width": w, "height": h}
|
||||
for n, w, h in CAMS]},
|
||||
"calibration_removed": ["serial"], "takes": ["01-hand-size", "02-no-hands"], "status": "done"})
|
||||
takes.write_json(os.path.join(sdir, "calibration.json"),
|
||||
{"cameras": [{"name": "slam_left", "intrinsics": [1.0, 2.0, 3.0]}]})
|
||||
if device:
|
||||
takes.write_json(os.path.join(sdir, "device.json"), DEVICE)
|
||||
for k, tid in enumerate(("01-hand-size", "02-no-hands")):
|
||||
tdir = os.path.join(sdir, "takes", tid)
|
||||
os.makedirs(tdir)
|
||||
t0 = 10 ** 12 * (k + 1)
|
||||
with open(os.path.join(tdir, "sets.bin"), "wb") as f:
|
||||
for i in range(30):
|
||||
f.write(fhset(t0 + i * 100_000_000, i))
|
||||
with open(os.path.join(tdir, "prompts.jsonl"), "w") as f:
|
||||
f.write(json.dumps({"t": t0, "event": "take", "section": tid[3:], "take": tid}) + "\n")
|
||||
f.write(json.dumps({"t": t0 + 5, "event": "end", "status": "complete"}) + "\n")
|
||||
with open(os.path.join(tdir, "poses.jsonl"), "w") as f:
|
||||
for i in range(10):
|
||||
f.write(json.dumps({"t": t0 + i, "hmd": {"m": [0.0] * 12, "r": 200, "ok": True},
|
||||
"left": None, "right": None}) + "\n")
|
||||
takes.write_json(os.path.join(tdir, "take.json"), {"section": tid[3:], "title": tid, "started_ns": t0,
|
||||
"ended_ns": t0 + 3 * 10 ** 9, "status": "complete",
|
||||
"deleted": [], "notes": ""})
|
||||
return contributor
|
||||
|
||||
|
||||
class ValidateTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not takes.find_zstd():
|
||||
raise unittest.SkipTest("no zstd")
|
||||
cls.tmp = tempfile.mkdtemp(prefix="handrec-validate-test-")
|
||||
cls.base = os.path.join(cls.tmp, "base")
|
||||
cls.contributor = make_session(cls.base)
|
||||
cls.store = takes.Store(cls.base)
|
||||
# One deleted range, so the export leaves sets out.
|
||||
index = takes.take_index(cls.store.take_dir(SESSION, "01-hand-size"))
|
||||
cls.store.delete_range(SESSION, "01-hand-size", index.time_ns(3), index.time_ns(5))
|
||||
cls.export = cls.store.export(SESSION, low_priority=False)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
shutil.rmtree(cls.tmp, ignore_errors=True)
|
||||
|
||||
def copy(self):
|
||||
d = tempfile.mkdtemp(dir=self.tmp)
|
||||
dest = os.path.join(d, SESSION)
|
||||
shutil.copytree(self.export, dest)
|
||||
return dest
|
||||
|
||||
def assertError(self, report, text):
|
||||
self.assertTrue(any(text in e for e in report.errors), f"no error with {text!r}: {report.errors}")
|
||||
|
||||
def test_good(self):
|
||||
r = validate.validate(self.export)
|
||||
self.assertEqual(r.errors, [])
|
||||
self.assertEqual(r.warnings, [])
|
||||
self.assertTrue(os.path.isfile(os.path.join(self.export, "device.json")))
|
||||
self.assertEqual(r.summary["takes"], 2)
|
||||
self.assertEqual(r.summary["sets"], 57)
|
||||
self.assertEqual(r.summary["consent_version"], "2026-10-02")
|
||||
self.assertEqual(r.summary["contributor"], self.contributor)
|
||||
|
||||
def test_extra_file(self):
|
||||
d = self.copy()
|
||||
with open(os.path.join(d, "notes.txt"), "w") as f:
|
||||
f.write("hello\n")
|
||||
os.makedirs(os.path.join(d, "takes", "01-hand-size", "extra"))
|
||||
r = validate.validate(d)
|
||||
self.assertError(r, "notes.txt: not an allowed file")
|
||||
self.assertError(r, "not in SHA256SUMS: notes.txt")
|
||||
self.assertError(r, "extra/: not an allowed folder")
|
||||
|
||||
def test_symlink(self):
|
||||
d = self.copy()
|
||||
os.symlink("/etc/hostname", os.path.join(d, "takes", "01-hand-size", "take.json.link"))
|
||||
self.assertError(validate.validate(d), "a symlink")
|
||||
|
||||
def test_bad_checksum(self):
|
||||
d = self.copy()
|
||||
path = os.path.join(d, "takes", "02-no-hands", "poses.jsonl")
|
||||
with open(path, "r+b") as f:
|
||||
f.seek(5)
|
||||
f.write(b"9") # "t": 2... becomes "t": 9...: still valid JSON
|
||||
r = validate.validate(d)
|
||||
self.assertError(r, "checksum mismatch: takes/02-no-hands/poses.jsonl")
|
||||
self.assertEqual(len(r.errors), 1, r.errors)
|
||||
|
||||
def test_calibration_serial(self):
|
||||
d = self.copy()
|
||||
takes.write_json(os.path.join(d, "calibration.json"),
|
||||
{"cameras": [{"name": "slam_left", "serial": "X", "label": "unit 4H2K19A7731"}]})
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertError(r, "calibration.json: identifying fields left in")
|
||||
self.assertIn("cameras[0].serial", r.errors[0])
|
||||
self.assertIn("cameras[0].label", r.errors[0])
|
||||
self.assertEqual(len(r.errors), 1, r.errors)
|
||||
|
||||
def test_truncated_zst(self):
|
||||
d = self.copy()
|
||||
path = os.path.join(d, "takes", "01-hand-size", "sets.bin.zst")
|
||||
os.truncate(path, os.path.getsize(path) // 2)
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertTrue(any("sets.bin.zst" in e and ("cut short" in e or "set " in e or "decompress" in e)
|
||||
for e in r.errors), r.errors)
|
||||
|
||||
def test_truncated_zst_command(self):
|
||||
"""The same with the zstd program (Python before 3.14)."""
|
||||
d = self.copy()
|
||||
path = os.path.join(d, "takes", "01-hand-size", "sets.bin.zst")
|
||||
os.truncate(path, os.path.getsize(path) // 2)
|
||||
with mock.patch.object(validate, "_zstd", None):
|
||||
r = validate.validate(d)
|
||||
self.assertTrue(any("sets.bin.zst" in e and "checksum" not in e for e in r.errors), r.errors)
|
||||
self.assertEqual(validate.validate(self.export).errors, [])
|
||||
|
||||
def test_garbage_records(self):
|
||||
d = self.copy()
|
||||
rec = bytearray(fhset(10 ** 9, 0))
|
||||
rec[validate.HDR.size:validate.HDR.size + 16] = b"slam left!\0\0\0\0\0\0"
|
||||
zst = os.path.join(d, "takes", "02-no-hands", "sets.bin.zst")
|
||||
with open(zst + ".raw", "wb") as f:
|
||||
f.write(bytes(rec))
|
||||
os.system(f"zstd -q -f {zst}.raw -o {zst} && rm {zst}.raw")
|
||||
write_sums(d)
|
||||
self.assertError(validate.validate(d), "a camera name that isn't one")
|
||||
|
||||
def test_manifest(self):
|
||||
d = self.copy()
|
||||
path = os.path.join(d, "manifest.json")
|
||||
with open(path) as f:
|
||||
m = json.load(f)
|
||||
m["contributor"] = m["profile"]["contributor"] = str(uuid.uuid1())
|
||||
m["consent_version"] = ""
|
||||
m["profile"]["consent"]["version"] = ""
|
||||
m["takes"][0]["sets"] += 1
|
||||
m["session"]["dry_run"] = True
|
||||
m["session"]["device"]["serial_number"] = "x"
|
||||
del m["exported"]
|
||||
takes.write_json(path, m)
|
||||
with open(os.path.join(d, "takes", "02-no-hands", "prompts.jsonl"), "a") as f:
|
||||
f.write("{not json\n")
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertError(r, "isn't a random uuid4")
|
||||
self.assertError(r, "the consent version is empty")
|
||||
self.assertError(r, "27 sets, the manifest says 28")
|
||||
self.assertError(r, "dry-run session")
|
||||
self.assertError(r, "identifying fields: session.device.serial_number")
|
||||
self.assertError(r, "exported is missing")
|
||||
self.assertError(r, "prompts.jsonl line 3: not valid JSON")
|
||||
|
||||
def test_device(self):
|
||||
d = self.copy()
|
||||
bad = json.loads(json.dumps(DEVICE))
|
||||
bad["device_serial_number"] = "X"
|
||||
bad["head"]["position"] = [0, 0]
|
||||
takes.write_json(os.path.join(d, "device.json"), bad)
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertError(r, "expected cv and head only")
|
||||
self.assertError(r, "head.position isn't 3 numbers")
|
||||
self.assertError(r, "device.json: identifying fields: device_serial_number")
|
||||
|
||||
def test_no_device(self):
|
||||
"""Sessions from before device.json still pass, with a warning."""
|
||||
d = self.copy()
|
||||
os.remove(os.path.join(d, "device.json"))
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertEqual(r.errors, [])
|
||||
self.assertTrue(any("no device.json" in w for w in r.warnings), r.warnings)
|
||||
|
||||
def test_zstandard(self):
|
||||
"""The zstandard package's path (the maintainer's Windows PC), good and truncated."""
|
||||
if validate._zstandard is None:
|
||||
self.skipTest("no zstandard")
|
||||
with mock.patch.object(validate, "_zstd", None):
|
||||
self.assertEqual(validate.validate(self.export).errors, [])
|
||||
d = self.copy()
|
||||
path = os.path.join(d, "takes", "01-hand-size", "sets.bin.zst")
|
||||
os.truncate(path, os.path.getsize(path) // 2)
|
||||
write_sums(d)
|
||||
self.assertError(validate.validate(d), "sets.bin.zst: the zstd stream is cut short")
|
||||
|
||||
def test_frames(self):
|
||||
"""Several zstd frames one after another (zstd -T2 can write them) read as one stream."""
|
||||
if validate._zstd is None and validate._zstandard is None:
|
||||
self.skipTest("no compression.zstd or zstandard")
|
||||
d = self.copy()
|
||||
zst = os.path.join(d, "takes", "02-no-hands", "sets.bin.zst")
|
||||
with open(zst, "rb") as f:
|
||||
one = f.read()
|
||||
with open(zst, "ab") as f:
|
||||
f.write(one)
|
||||
with open(os.path.join(d, "manifest.json")) as f:
|
||||
m = json.load(f)
|
||||
m["takes"][1]["sets"] *= 2
|
||||
m["takes"][1]["raw_bytes"] *= 2
|
||||
takes.write_json(os.path.join(d, "manifest.json"), m)
|
||||
write_sums(d)
|
||||
r = validate.validate(d)
|
||||
self.assertEqual([e for e in r.errors if "go back" not in e], [])
|
||||
|
||||
def test_cli(self):
|
||||
self.assertEqual(os.system(f"{sys.executable} {validate.__file__} {self.export} --json >/dev/null"), 0)
|
||||
d = self.copy()
|
||||
os.remove(os.path.join(d, "manifest.json"))
|
||||
self.assertNotEqual(os.system(f"{sys.executable} {validate.__file__} {d} >/dev/null"), 0)
|
||||
|
||||
|
||||
class SessionFilesTest(unittest.TestCase):
|
||||
"""session.py's device.json, and session ids with a suffix."""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.mkdtemp(prefix="handrec-session-test-")
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||
|
||||
def test_write_device(self):
|
||||
full = dict(json.loads(json.dumps(DEVICE)), device_serial_number="ABC12345678", display_edid="00ff",
|
||||
model_number="Frame")
|
||||
src = os.path.join(self.tmp, "device_config.json")
|
||||
takes.write_json(src, full)
|
||||
s = session.Session.__new__(session.Session)
|
||||
s.session_dir, s._log = self.tmp, lambda text: None
|
||||
with mock.patch.object(session, "host_path", lambda p: src if p.endswith("device_config.json") else None):
|
||||
self.assertEqual(s._write_device(), [])
|
||||
with open(os.path.join(self.tmp, "device.json")) as f:
|
||||
self.assertEqual(json.load(f), DEVICE)
|
||||
with mock.patch.object(session, "host_path", lambda p: None):
|
||||
self.assertEqual(s._write_device(), [])
|
||||
|
||||
def test_suffix_id(self):
|
||||
if not takes.find_zstd():
|
||||
self.skipTest("no zstd")
|
||||
sid = SESSION + "-2"
|
||||
make_session(self.tmp, sid=sid)
|
||||
store = takes.Store(self.tmp)
|
||||
self.assertEqual([x["id"] for x in store.sessions()], [sid])
|
||||
path = store.export(sid, low_priority=False)
|
||||
self.assertEqual(validate.validate(path).errors, [])
|
||||
|
||||
|
||||
class HubTest(unittest.TestCase):
|
||||
"""hub.py without the network: the dry run, the draft gate, upload records."""
|
||||
|
||||
def setUp(self):
|
||||
if not takes.find_zstd():
|
||||
self.skipTest("no zstd")
|
||||
self.tmp = tempfile.mkdtemp(prefix="handrec-hub-test-")
|
||||
self.contributor = make_session(self.tmp)
|
||||
self.store = takes.Store(self.tmp)
|
||||
self.store.export(SESSION, low_priority=False)
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||
|
||||
def test_dry_run(self):
|
||||
logged = []
|
||||
with mock.patch.dict(os.environ, {hub.DATASET_ENV: "someone/test-hands"}):
|
||||
result = hub.upload(self.store, SESSION, dry_run=True, log=logged.append)
|
||||
self.assertTrue(result["dry_run"])
|
||||
self.assertEqual(result["repo"], "someone/test-hands")
|
||||
self.assertEqual(result["path_in_repo"], f"contributions/{self.contributor}/{SESSION}")
|
||||
self.assertIn("- Takes: 2", result["commit_description"])
|
||||
self.assertIn("- Consent version: 2026-10-02", result["commit_description"])
|
||||
self.assertIn("- Objects: pencil (+1 of their own)", result["commit_description"])
|
||||
self.assertTrue(any("manifest.json" in line for line in logged))
|
||||
self.assertEqual(hub.uploads(self.store, SESSION), []) # a dry run records nothing
|
||||
|
||||
def test_dataset_env(self):
|
||||
self.assertEqual(hub.dataset_id(), hub.HF_DATASET)
|
||||
with mock.patch.dict(os.environ, {hub.DATASET_ENV: "not a repo"}):
|
||||
self.assertRaises(hub.HubError, hub.dataset_id)
|
||||
|
||||
def test_draft_gate(self):
|
||||
with mock.patch.object(hub, "texts_draft", return_value=True), \
|
||||
mock.patch.dict(os.environ, {hub.ALLOW_ENV: ""}):
|
||||
with self.assertRaises(hub.HubError) as cm:
|
||||
hub.upload(self.store, SESSION)
|
||||
self.assertEqual(cm.exception.kind, "closed")
|
||||
self.assertFalse(hub.upload_allowed())
|
||||
with mock.patch.object(hub, "texts_draft", return_value=True), \
|
||||
mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1"}):
|
||||
self.assertTrue(hub.upload_allowed())
|
||||
|
||||
def test_invalid_blocks(self):
|
||||
with open(os.path.join(self.store.export_dir(SESSION), "extra.bin"), "w") as f:
|
||||
f.write("x")
|
||||
with self.assertRaises(hub.HubError) as cm:
|
||||
hub.upload(self.store, SESSION, dry_run=True)
|
||||
self.assertEqual(cm.exception.kind, "invalid")
|
||||
self.assertTrue(cm.exception.errors)
|
||||
|
||||
def test_record_and_duplicate(self):
|
||||
export = self.store.export_dir(SESSION)
|
||||
sha = hub.export_sha(export)
|
||||
self.assertEqual(len(sha), 64)
|
||||
hub.record_upload(self.store, SESSION, {"repo": "a/b", "pr_url": "https://huggingface.co/datasets/a/b/discussions/1",
|
||||
"uploaded": hub.now_iso(), "export_sha": sha})
|
||||
# The upload record doesn't make the export look out of date...
|
||||
self.assertFalse(self.store.sessions()[0]["export_stale"])
|
||||
# ...and the next upload of the same export stops, unless asked again.
|
||||
with self.assertRaises(hub.HubError) as cm:
|
||||
hub.upload(self.store, SESSION, dry_run=True)
|
||||
self.assertEqual(cm.exception.kind, "duplicate")
|
||||
self.assertTrue(hub.upload(self.store, SESSION, dry_run=True, again=True)["dry_run"])
|
||||
# A new export leaves the records out of its manifest.
|
||||
self.store.export(SESSION, low_priority=False)
|
||||
with open(os.path.join(export, "manifest.json")) as f:
|
||||
self.assertNotIn("uploads", json.load(f)["session"])
|
||||
|
||||
def test_explain(self):
|
||||
try:
|
||||
import httpx
|
||||
from huggingface_hub import errors as hf
|
||||
except ImportError:
|
||||
self.skipTest("no huggingface_hub")
|
||||
|
||||
def response(code):
|
||||
return httpx.Response(code, request=httpx.Request("POST", "https://huggingface.co/api/x"))
|
||||
cases = [(hf.GatedRepoError("gated", response=response(403)), "terms"),
|
||||
(hf.RepositoryNotFoundError("nope", response=response(404)), "not-found"),
|
||||
(hf.HfHubHTTPError("no", response=response(403)), "permission"),
|
||||
(hf.HfHubHTTPError("bad token", response=response(401)), "login"),
|
||||
(hf.LocalTokenNotFoundError("none"), "login"),
|
||||
(httpx.ConnectError("down"), "network"),
|
||||
(ConnectionResetError("reset"), "network")]
|
||||
for exc, kind in cases:
|
||||
e = hub.explain(exc, "a/b")
|
||||
self.assertEqual(e.kind, kind, exc)
|
||||
self.assertIn("https://huggingface.co/datasets/a/b", hub.explain(cases[0][0], "a/b").link)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Executable
+734
@@ -0,0 +1,734 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check a hand recorder export before it's uploaded, or when it's received (DESIGN.md "Upload").
|
||||
|
||||
The window runs it before an upload and refuses on errors; the maintainer runs it on each
|
||||
submission. Standard library only. Given an export folder (takes.py's exports/<session>/), it
|
||||
checks:
|
||||
- SHA256SUMS: every file listed and matching, nothing missing from it;
|
||||
- an allow-list of files: anything else (or a symlink) is an error;
|
||||
- manifest.json: the schema, required keys and types, and that it agrees with the files;
|
||||
- the consent version is there, the contributor id is a random uuid4, the contributor is an adult;
|
||||
- calibration.json has nothing session.py's strip_calibration would still remove;
|
||||
- device.json (the rig's pose in the CAD frame) holds only cv.cad_from_cal and head, as
|
||||
plus_x, plus_z and position; sessions recorded before it existed get a warning;
|
||||
- each sets.bin.zst decompresses to the end (so a truncated one fails) and parses as FHSET01:
|
||||
every set's header is checked (camera names, sizes, record length) and counted against the
|
||||
manifest; the pixels themselves aren't looked at;
|
||||
- every .jsonl line parses, and take.json does;
|
||||
- the total size.
|
||||
It runs on Linux and Windows, with Python 3.12 or later. It reports errors (don't upload or accept this) and warnings (look at it).
|
||||
|
||||
usage: validate.py DIR [--json] exit status 0: no errors, 1: errors, 2: not an export
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
from session import _key_identifies, strip_calibration # noqa: E402 (standard library only, next to this file)
|
||||
|
||||
# Decompression, the first that's there: Python 3.14's compression.zstd, the zstandard package
|
||||
# (pip install zstandard: the maintainer's Windows PC), the zstd program.
|
||||
try:
|
||||
from compression import zstd as _zstd
|
||||
except ImportError:
|
||||
_zstd = None
|
||||
try:
|
||||
import zstandard as _zstandard
|
||||
except ImportError:
|
||||
_zstandard = None
|
||||
|
||||
EXPORT_SCHEMA = 1
|
||||
TOP_FILES = {"manifest.json", "calibration.json", "device.json", "SHA256SUMS"}
|
||||
TAKE_FILES = {"prompts.jsonl", "poses.jsonl", "take.json", "sets.bin.zst"}
|
||||
TAKE_RE = re.compile(r"^\d{2}-[a-z0-9][a-z0-9-]*$")
|
||||
SESSION_RE = re.compile(r"^\d{8}-\d{6}(?:-\d+)?$")
|
||||
SUM_RE = re.compile(r"^([0-9a-f]{64}) (\S.*)$")
|
||||
CAM_NAME_RE = re.compile(r"^[a-z][a-z0-9_]{0,15}$")
|
||||
TAKE_STATUSES = ("complete", "stopped", "skipped")
|
||||
HDR = struct.Struct("<8sII") # fh_set_hdr_t: magic, ncams, bytes (hands/track/record.h)
|
||||
CAM = struct.Struct("<16sIIQQ") # fh_set_cam_t: name, width, height, capture_ns, dqbuf_ns
|
||||
MAX_CAMS = 16
|
||||
MIN_SIDE, MAX_SIDE = 16, 4096
|
||||
# One round is about 10 GB before compression (ft_handrec.ROUND_BYTES), a few GB after it.
|
||||
WARN_BYTES = 15 * 1000 ** 3
|
||||
MAX_BYTES = 40 * 1000 ** 3
|
||||
CHUNK = 1 << 20
|
||||
OUT_CHUNK = 8 << 20
|
||||
# zstandard's decompressobj has no output limit: it gets the input this much at a time, so even a
|
||||
# stream made to inflate hugely gives out a few hundred MB at most per call.
|
||||
ZSTANDARD_PIECE = 8 << 10
|
||||
MAX_LISTED = 10 # removed calibration paths and the like: list this many, then "and N more"
|
||||
|
||||
|
||||
class Cancelled(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Report:
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
self.errors = []
|
||||
self.warnings = []
|
||||
self.summary = {}
|
||||
self.bytes = 0
|
||||
|
||||
def error(self, text):
|
||||
self.errors.append(text)
|
||||
|
||||
def warn(self, text):
|
||||
self.warnings.append(text)
|
||||
|
||||
@property
|
||||
def ok(self):
|
||||
return not self.errors
|
||||
|
||||
def as_dict(self):
|
||||
return {"path": self.path, "ok": self.ok, "errors": self.errors, "warnings": self.warnings,
|
||||
"summary": self.summary, "bytes": self.bytes}
|
||||
|
||||
|
||||
def listed(items):
|
||||
items = list(items)
|
||||
more = len(items) - MAX_LISTED
|
||||
return ", ".join(items[:MAX_LISTED]) + (f" and {more} more" if more > 0 else "")
|
||||
|
||||
|
||||
def is_uuid4(s):
|
||||
try:
|
||||
u = uuid.UUID(s)
|
||||
except (TypeError, ValueError, AttributeError):
|
||||
return False
|
||||
return u.version == 4 and str(u) == s
|
||||
|
||||
|
||||
def sha256_file(path, step=None, cancelled=lambda: False):
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
while chunk := f.read(CHUNK):
|
||||
if cancelled():
|
||||
raise Cancelled()
|
||||
h.update(chunk)
|
||||
if step:
|
||||
step(len(chunk))
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the files
|
||||
|
||||
def walk_files(root, report):
|
||||
"""The export's files as relative paths, and allow-list errors on the way."""
|
||||
files = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
rel_dir = os.path.relpath(dirpath, root).replace(os.sep, "/")
|
||||
rel_dir = "" if rel_dir == "." else rel_dir
|
||||
depth = 0 if not rel_dir else rel_dir.count("/") + 1
|
||||
for d in list(dirnames):
|
||||
rel = f"{rel_dir}/{d}" if rel_dir else d
|
||||
ok = (depth == 0 and d == "takes") or (depth == 1 and rel_dir == "takes" and TAKE_RE.match(d))
|
||||
if os.path.islink(os.path.join(dirpath, d)):
|
||||
report.error(f"{rel}: a symlink, not allowed in an export")
|
||||
dirnames.remove(d)
|
||||
elif not ok:
|
||||
report.error(f"{rel}/: not an allowed folder")
|
||||
dirnames.remove(d)
|
||||
for name in filenames:
|
||||
rel = f"{rel_dir}/{name}" if rel_dir else name
|
||||
full = os.path.join(dirpath, name)
|
||||
if os.path.islink(full) or not os.path.isfile(full):
|
||||
report.error(f"{rel}: a symlink or special file, not allowed in an export")
|
||||
continue
|
||||
if not ((depth == 0 and name in TOP_FILES) or (depth == 2 and name in TAKE_FILES)):
|
||||
report.error(f"{rel}: not an allowed file")
|
||||
files.append(rel)
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def check_sums(root, files, report, step, cancelled):
|
||||
path = os.path.join(root, "SHA256SUMS")
|
||||
if not os.path.isfile(path):
|
||||
report.error("SHA256SUMS is missing")
|
||||
return
|
||||
sums = {}
|
||||
with open(path, encoding="utf-8", errors="replace") as f:
|
||||
for n, line in enumerate(f, 1):
|
||||
line = line.rstrip("\n")
|
||||
m = SUM_RE.match(line)
|
||||
if not m:
|
||||
report.error(f"SHA256SUMS line {n}: not \"<sha256> <path>\"")
|
||||
continue
|
||||
digest, rel = m.groups()
|
||||
if rel.startswith("/") or ".." in rel.split("/") or rel == "SHA256SUMS":
|
||||
report.error(f"SHA256SUMS line {n}: a path that doesn't belong: {rel}")
|
||||
continue
|
||||
if rel in sums:
|
||||
report.error(f"SHA256SUMS lists {rel} twice")
|
||||
sums[rel] = digest
|
||||
present = set(files) - {"SHA256SUMS"}
|
||||
missing = sorted(set(sums) - present)
|
||||
if missing:
|
||||
report.error(f"in SHA256SUMS but missing: {listed(missing)}")
|
||||
unlisted = sorted(present - set(sums))
|
||||
if unlisted:
|
||||
report.error(f"not in SHA256SUMS: {listed(unlisted)}")
|
||||
bad = []
|
||||
for rel in sorted(set(sums) & present):
|
||||
if sha256_file(os.path.join(root, rel), step, cancelled) != sums[rel]:
|
||||
bad.append(rel)
|
||||
if bad:
|
||||
report.error(f"checksum mismatch: {listed(bad)}")
|
||||
|
||||
|
||||
def read_json_file(path, rel, report):
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except OSError as e:
|
||||
report.error(f"{rel}: can't read it: {e.strerror or e}")
|
||||
except ValueError as e:
|
||||
report.error(f"{rel}: not valid JSON: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def check_jsonl(path, rel, report, cancelled):
|
||||
"""Every line a JSON object with a time "t"."""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for n, line in enumerate(f, 1):
|
||||
if n % 50000 == 0 and cancelled():
|
||||
raise Cancelled()
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except ValueError as e:
|
||||
report.error(f"{rel} line {n}: not valid JSON: {e}")
|
||||
return
|
||||
if not isinstance(obj, dict) or not isinstance(obj.get("t"), int):
|
||||
report.error(f"{rel} line {n}: not an object with a time \"t\"")
|
||||
return
|
||||
except UnicodeDecodeError:
|
||||
report.error(f"{rel}: not UTF-8 text")
|
||||
except OSError as e:
|
||||
report.error(f"{rel}: can't read it: {e.strerror or e}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- FHSET01 inside the zstd stream
|
||||
|
||||
class SetParser:
|
||||
"""Feed it the decompressed stream in pieces: it checks each set's header and skips the
|
||||
pixels. problem holds the first thing wrong (parsing stops there)."""
|
||||
|
||||
def __init__(self):
|
||||
self.buf = bytearray()
|
||||
self.skip = 0 # pixel bytes of the current set still to pass over
|
||||
self.sets = 0
|
||||
self.raw = 0
|
||||
self.cams = {} # name -> (width, height), every camera seen
|
||||
self.first = None # the first set's cameras, [{name, width, height}]
|
||||
self.last_t = 0
|
||||
self.backwards = 0 # sets whose earliest dqbuf_ns went back in time
|
||||
self.problem = ""
|
||||
|
||||
def feed(self, data):
|
||||
if self.problem:
|
||||
return
|
||||
self.raw += len(data)
|
||||
view = memoryview(data)
|
||||
while len(view) and not self.problem:
|
||||
if self.skip:
|
||||
n = min(self.skip, len(view))
|
||||
self.skip -= n
|
||||
view = view[n:]
|
||||
continue
|
||||
self.buf += view
|
||||
view = view[len(view):]
|
||||
self._parse()
|
||||
|
||||
def _parse(self):
|
||||
while not self.problem and not self.skip:
|
||||
if len(self.buf) < HDR.size:
|
||||
return
|
||||
magic, ncams, nbytes = HDR.unpack_from(self.buf, 0)
|
||||
where = f"set {self.sets + 1}"
|
||||
if magic != b"FHSET01\0":
|
||||
self.problem = f"{where}: not an FHSET01 record"
|
||||
return
|
||||
if not 0 < ncams <= MAX_CAMS:
|
||||
self.problem = f"{where}: {ncams} cameras"
|
||||
return
|
||||
head = HDR.size + CAM.size * ncams
|
||||
if len(self.buf) < head:
|
||||
return
|
||||
pixels, cams, times = 0, [], []
|
||||
for k in range(ncams):
|
||||
raw_name, w, h, _, dqbuf_ns = CAM.unpack_from(self.buf, HDR.size + k * CAM.size)
|
||||
name_bytes, _, pad = raw_name.partition(b"\0")
|
||||
name = name_bytes.decode("ascii", errors="replace")
|
||||
if pad.strip(b"\0") or not CAM_NAME_RE.match(name):
|
||||
self.problem = f"{where}: a camera name that isn't one: {raw_name!r}"
|
||||
return
|
||||
if not (MIN_SIDE <= w <= MAX_SIDE and MIN_SIDE <= h <= MAX_SIDE):
|
||||
self.problem = f"{where}: camera {name} is {w}x{h}"
|
||||
return
|
||||
if self.cams.setdefault(name, (w, h)) != (w, h):
|
||||
self.problem = f"{where}: camera {name} changed size to {w}x{h}"
|
||||
return
|
||||
pixels += w * h
|
||||
cams.append({"name": name, "width": w, "height": h})
|
||||
times.append(dqbuf_ns)
|
||||
if nbytes != head + pixels:
|
||||
self.problem = f"{where}: {nbytes} bytes, but its cameras need {head + pixels}"
|
||||
return
|
||||
if self.first is None:
|
||||
self.first = cams
|
||||
t = min(times)
|
||||
if t < self.last_t:
|
||||
self.backwards += 1
|
||||
self.last_t = t
|
||||
self.sets += 1
|
||||
rest = len(self.buf) - head
|
||||
if rest >= pixels:
|
||||
del self.buf[:head + pixels]
|
||||
else:
|
||||
self.skip = pixels - rest
|
||||
self.buf.clear()
|
||||
|
||||
def finish(self):
|
||||
"""After the stream's end: a set cut short is a problem."""
|
||||
if not self.problem and (self.skip or self.buf):
|
||||
self.problem = f"the stream ends inside set {self.sets + 1}"
|
||||
|
||||
|
||||
def find_zstd():
|
||||
found = shutil.which("zstd")
|
||||
if found:
|
||||
return found
|
||||
return next((p for p in ("/usr/bin/zstd", "/run/host/usr/bin/zstd") if os.access(p, os.X_OK)), None)
|
||||
|
||||
|
||||
class Frames:
|
||||
"""A zstd stream, decompressed piece by piece with compression.zstd or zstandard, frame after
|
||||
frame (zstd -T2 may write several). feed(data) yields the output; complete is false while a
|
||||
frame is unfinished, which at the end of the input means the stream was cut short."""
|
||||
|
||||
def __init__(self):
|
||||
self.dec = self._new()
|
||||
self.fed = False # the current decompressor has had input
|
||||
|
||||
@staticmethod
|
||||
def _new():
|
||||
return _zstd.ZstdDecompressor() if _zstd is not None else _zstandard.ZstdDecompressor().decompressobj()
|
||||
|
||||
@property
|
||||
def complete(self):
|
||||
return self.dec.eof or not self.fed
|
||||
|
||||
def feed(self, data):
|
||||
while data:
|
||||
if self.dec.eof: # another frame follows the one that ended
|
||||
self.dec, self.fed = self._new(), False
|
||||
if _zstd is not None:
|
||||
out = self.dec.decompress(data, max_length=OUT_CHUNK)
|
||||
data = b""
|
||||
self.fed = True
|
||||
yield out
|
||||
# More output may wait without more input.
|
||||
while not self.dec.eof and not self.dec.needs_input:
|
||||
yield self.dec.decompress(b"", max_length=OUT_CHUNK)
|
||||
else:
|
||||
piece, data = data[:ZSTANDARD_PIECE], data[ZSTANDARD_PIECE:]
|
||||
self.fed = True
|
||||
out = self.dec.decompress(piece)
|
||||
if out:
|
||||
yield out
|
||||
if self.dec.eof:
|
||||
data = self.dec.unused_data + data
|
||||
|
||||
|
||||
def check_sets(path, rel, entry, report, step, cancelled):
|
||||
"""Decompress rel as a stream and parse it. entry: its take in the manifest, or None."""
|
||||
parser = SetParser()
|
||||
try:
|
||||
if _zstd is not None or _zstandard is not None:
|
||||
frames = Frames()
|
||||
with open(path, "rb") as f:
|
||||
while not parser.problem and (data := f.read(CHUNK)):
|
||||
if cancelled():
|
||||
raise Cancelled()
|
||||
step(len(data))
|
||||
for out in frames.feed(data):
|
||||
parser.feed(out)
|
||||
if parser.problem:
|
||||
break
|
||||
if not parser.problem and not frames.complete:
|
||||
report.error(f"{rel}: the zstd stream is cut short")
|
||||
return
|
||||
else:
|
||||
zstd = find_zstd()
|
||||
if not zstd:
|
||||
report.warn(f"{rel}: not checked: no zstd (Python 3.14's compression.zstd, the zstandard "
|
||||
"package or the zstd program)")
|
||||
return
|
||||
argv = [zstd, "-dcq", path]
|
||||
if os.name == "posix" and shutil.which("nice"):
|
||||
argv = ["nice", "-n", "19"] + argv
|
||||
proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
while not parser.problem and (data := proc.stdout.read(CHUNK)):
|
||||
if cancelled():
|
||||
raise Cancelled()
|
||||
parser.feed(data)
|
||||
except BaseException:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
proc.stdout.close()
|
||||
raise
|
||||
if parser.problem:
|
||||
proc.kill()
|
||||
code = proc.wait()
|
||||
proc.stdout.close()
|
||||
step(os.path.getsize(path))
|
||||
if not parser.problem and code != 0:
|
||||
report.error(f"{rel}: doesn't decompress (zstd exit {code})")
|
||||
return
|
||||
except Cancelled:
|
||||
raise
|
||||
except Exception as e: # OSError, compression.zstd.ZstdError
|
||||
report.error(f"{rel}: doesn't decompress: {e}")
|
||||
return
|
||||
parser.finish()
|
||||
if parser.problem:
|
||||
report.error(f"{rel}: not a valid FHSET01 recording: {parser.problem}")
|
||||
return
|
||||
if parser.sets == 0:
|
||||
report.error(f"{rel}: holds no frame sets")
|
||||
if parser.backwards:
|
||||
report.warn(f"{rel}: {parser.backwards} sets go back in time")
|
||||
if entry is not None:
|
||||
if entry.get("sets") != parser.sets:
|
||||
report.error(f"{rel}: {parser.sets} sets, the manifest says {entry.get('sets')}")
|
||||
if "raw_bytes" in entry and entry.get("raw_bytes") != parser.raw:
|
||||
report.error(f"{rel}: {parser.raw} bytes uncompressed, the manifest says {entry.get('raw_bytes')}")
|
||||
if isinstance(entry.get("cameras"), list) and parser.first is not None and entry["cameras"] != parser.first:
|
||||
report.error(f"{rel}: its cameras don't match the manifest's")
|
||||
return parser
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the manifest
|
||||
|
||||
def need(obj, key, types, where, report, required=True):
|
||||
"""obj[key] if it's of the given type(s); an error otherwise."""
|
||||
if not isinstance(obj, dict) or key not in obj:
|
||||
if required:
|
||||
report.error(f"manifest: {where}{key} is missing")
|
||||
return None
|
||||
v = obj[key]
|
||||
allowed = types if isinstance(types, tuple) else (types,)
|
||||
if not isinstance(v, allowed) or (isinstance(v, bool) and bool not in allowed):
|
||||
report.error(f"manifest: {where}{key} has the wrong type ({type(v).__name__})")
|
||||
return None
|
||||
return v
|
||||
|
||||
|
||||
def check_pose(pose, where, report, extra=()):
|
||||
"""A pose as device_config.json has it: plus_x, plus_z, position, each 3 finite numbers."""
|
||||
if not isinstance(pose, dict):
|
||||
report.error(f"device.json: {where} isn't an object")
|
||||
return
|
||||
for key in ("plus_x", "plus_z", "position"):
|
||||
v = pose.get(key)
|
||||
if not (isinstance(v, list) and len(v) == 3 and all(isinstance(x, (int, float)) and not isinstance(x, bool)
|
||||
and math.isfinite(x) for x in v)):
|
||||
report.error(f"device.json: {where}.{key} isn't 3 numbers")
|
||||
unknown = set(pose) - {"plus_x", "plus_z", "position"} - set(extra)
|
||||
if unknown:
|
||||
report.error(f"device.json: {where} has unexpected keys: {listed(sorted(unknown))}")
|
||||
|
||||
|
||||
def check_device(dev, report):
|
||||
"""device.json: only cv.cad_from_cal and head (session.py's _write_device)."""
|
||||
if not isinstance(dev, dict):
|
||||
report.error("device.json: not a JSON object")
|
||||
return
|
||||
if set(dev) != {"cv", "head"}:
|
||||
report.error(f"device.json: keys {listed(sorted(dev))}, expected cv and head only")
|
||||
cv = dev.get("cv")
|
||||
if not isinstance(cv, dict) or set(cv) != {"cad_from_cal"}:
|
||||
report.error("device.json: cv should hold cad_from_cal only")
|
||||
else:
|
||||
check_pose(cv["cad_from_cal"], "cv.cad_from_cal", report, extra=("method",))
|
||||
if not isinstance(cv["cad_from_cal"].get("method", ""), str):
|
||||
report.error("device.json: cv.cad_from_cal.method isn't text")
|
||||
if "head" in dev:
|
||||
check_pose(dev["head"], "head", report)
|
||||
_, removed = strip_calibration(dev)
|
||||
if removed:
|
||||
report.error(f"device.json: identifying fields: {listed(removed)}")
|
||||
|
||||
|
||||
def identifying_keys(o, path):
|
||||
"""Paths of keys under o that name a serial, uuid, mac or id (session.py's rule)."""
|
||||
out = []
|
||||
if isinstance(o, dict):
|
||||
for k, v in o.items():
|
||||
p = f"{path}.{k}"
|
||||
out += [p] if _key_identifies(k) else identifying_keys(v, p)
|
||||
elif isinstance(o, list):
|
||||
for i, v in enumerate(o):
|
||||
out += identifying_keys(v, f"{path}[{i}]")
|
||||
return out
|
||||
|
||||
|
||||
def check_manifest(m, report, root):
|
||||
"""The manifest's keys and types. Returns {take id: entry} for the file checks."""
|
||||
if not isinstance(m, dict):
|
||||
report.error("manifest: not a JSON object")
|
||||
return {}
|
||||
if m.get("schema") != EXPORT_SCHEMA:
|
||||
report.error(f"manifest: schema {m.get('schema')!r}, expected {EXPORT_SCHEMA}")
|
||||
tool = need(m, "tool", str, "", report)
|
||||
if tool is not None and not tool.startswith("ft-handrec"):
|
||||
report.warn(f"manifest: tool is {tool!r}")
|
||||
need(m, "exported", str, "", report)
|
||||
sid = need(m, "session_id", str, "", report)
|
||||
if sid is not None and not SESSION_RE.match(sid):
|
||||
report.error(f"manifest: session_id {sid!r} isn't a session id")
|
||||
if sid and SESSION_RE.match(os.path.basename(root)) and os.path.basename(root) != sid:
|
||||
report.warn(f"the folder is named {os.path.basename(root)}, the manifest's session is {sid}")
|
||||
contributor = need(m, "contributor", str, "", report)
|
||||
if contributor is not None and not is_uuid4(contributor):
|
||||
report.error(f"manifest: the contributor id {contributor!r} isn't a random uuid4")
|
||||
parent = os.path.basename(os.path.dirname(root))
|
||||
if contributor and is_uuid4(parent) and parent != contributor:
|
||||
report.error(f"the export sits under contributor {parent}, but its manifest says {contributor}")
|
||||
consent = need(m, "consent_version", str, "", report)
|
||||
if consent is not None and not consent.strip():
|
||||
report.error("manifest: the consent version is empty")
|
||||
|
||||
profile = need(m, "profile", dict, "", report)
|
||||
if profile is not None:
|
||||
if profile.get("contributor") != contributor:
|
||||
report.error("manifest: profile.contributor differs from contributor")
|
||||
pc = need(profile, "consent", dict, "profile.", report)
|
||||
if pc is not None:
|
||||
if pc.get("version") != consent:
|
||||
report.error("manifest: profile.consent.version differs from consent_version")
|
||||
if pc.get("adult") is not True:
|
||||
report.error("manifest: the contributor didn't confirm being 18 or older")
|
||||
need(pc, "accepted", str, "profile.consent.", report)
|
||||
optional = profile.get("optional")
|
||||
if isinstance(optional, dict) and str(optional.get("notes") or "").strip():
|
||||
report.warn("the profile's notes are included: check they hold nothing private")
|
||||
extra = set(profile) - {"schema", "contributor", "consent", "optional"}
|
||||
if extra:
|
||||
report.error(f"manifest: the profile has unexpected keys: {listed(sorted(extra))}")
|
||||
|
||||
session = need(m, "session", dict, "", report)
|
||||
if session is not None:
|
||||
if session.get("contributor") not in (contributor, ""):
|
||||
report.error("manifest: session.contributor differs from contributor")
|
||||
for key, t in (("started", str), ("lighting", dict), ("checklist", dict), ("device", dict),
|
||||
("takes", list)):
|
||||
need(session, key, t, "session.", report)
|
||||
if session.get("dry_run"):
|
||||
report.error("this is a dry-run session: it has no real recordings")
|
||||
if "uploads" in session:
|
||||
report.warn("manifest: session.uploads is included (earlier uploads' links)")
|
||||
if not isinstance(session.get("calibration_removed", []), list):
|
||||
report.error("manifest: session.calibration_removed isn't a list")
|
||||
checklist = session.get("checklist")
|
||||
if isinstance(checklist, dict) and str(checklist.get("notes") or "").strip():
|
||||
report.warn("the checklist's notes are included: check they hold nothing private")
|
||||
# Keys only: version strings like SteamVR's "r25358740+28b72a4f-1" look like serials.
|
||||
keys = identifying_keys(session.get("device"), "session.device")
|
||||
if keys:
|
||||
report.error(f"manifest: identifying fields: {listed(keys)}")
|
||||
if "/home/" in json.dumps(m):
|
||||
report.warn("manifest: it mentions a home folder path (a username may show)")
|
||||
|
||||
entries = {}
|
||||
takes = need(m, "takes", list, "", report)
|
||||
for i, t in enumerate(takes or []):
|
||||
where = f"takes[{i}]."
|
||||
if not isinstance(t, dict):
|
||||
report.error(f"manifest: takes[{i}] isn't an object")
|
||||
continue
|
||||
tid = need(t, "id", str, where, report)
|
||||
if tid is None:
|
||||
continue
|
||||
if not TAKE_RE.match(tid):
|
||||
report.error(f"manifest: take id {tid!r} isn't one")
|
||||
continue
|
||||
if tid in entries:
|
||||
report.error(f"manifest: take {tid} is listed twice")
|
||||
entries[tid] = t
|
||||
for key, typ in (("section", str), ("title", str), ("status", str), ("sets", int),
|
||||
("sets_deleted", int), ("cameras", list), ("duration_s", (int, float))):
|
||||
need(t, key, typ, where, report)
|
||||
if t.get("status") not in TAKE_STATUSES:
|
||||
report.warn(f"manifest: take {tid} has status {t.get('status')!r}")
|
||||
f = t.get("file")
|
||||
if f is None:
|
||||
if t.get("sets"):
|
||||
report.error(f"manifest: take {tid} has {t.get('sets')} sets but no file")
|
||||
elif f != f"takes/{tid}/sets.bin.zst":
|
||||
report.error(f"manifest: take {tid}'s file is {f!r}")
|
||||
elif not isinstance(t.get("raw_bytes"), int):
|
||||
report.error(f"manifest: take {tid} has a file but no raw_bytes")
|
||||
return entries
|
||||
|
||||
|
||||
def summarize(m):
|
||||
"""What a pull request's description says about the export: takes, minutes, lighting,
|
||||
objects, controllers, the consent and tool versions."""
|
||||
if not isinstance(m, dict):
|
||||
return {}
|
||||
session = m.get("session") if isinstance(m.get("session"), dict) else {}
|
||||
checklist = session.get("checklist") if isinstance(session.get("checklist"), dict) else {}
|
||||
lighting = session.get("lighting") if isinstance(session.get("lighting"), dict) else {}
|
||||
takes = [t for t in m.get("takes") or [] if isinstance(t, dict)]
|
||||
seconds = sum(t.get("duration_s") or 0 for t in takes if isinstance(t.get("duration_s"), (int, float)))
|
||||
return {"session": m.get("session_id", ""), "contributor": m.get("contributor", ""),
|
||||
"takes": sum(1 for t in takes if t.get("file")), "takes_listed": len(takes),
|
||||
"sets": sum(t.get("sets") or 0 for t in takes if isinstance(t.get("sets"), int)),
|
||||
"minutes": round(seconds / 60, 1), "lighting": lighting.get("chosen", ""),
|
||||
"objects": list(checklist.get("objects") or []), "own_objects": len(checklist.get("own_objects") or []),
|
||||
"controllers": checklist.get("controllers", ""), "consent_version": m.get("consent_version", ""),
|
||||
"tool": m.get("tool", "")}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- all of it
|
||||
|
||||
def validate(root, progress=None, cancel=None):
|
||||
"""Check the export at root. progress(fraction 0..1, text) is called as it goes; cancel()
|
||||
(or cancel.is_set()) returning true raises Cancelled. Returns a Report."""
|
||||
root = os.path.abspath(root)
|
||||
report = Report(root)
|
||||
cancelled = getattr(cancel, "is_set", cancel) or (lambda: False)
|
||||
tell = progress or (lambda fraction, text: None)
|
||||
if not os.path.isdir(root):
|
||||
report.error(f"{root}: no such folder")
|
||||
return report
|
||||
files = walk_files(root, report)
|
||||
sizes = {rel: os.path.getsize(os.path.join(root, rel)) for rel in files}
|
||||
report.bytes = sum(sizes.values())
|
||||
# Work: each file's checksum, plus reading each sets.bin.zst again to decompress it.
|
||||
total = max(1, report.bytes + sum(n for rel, n in sizes.items() if rel.endswith(".zst")))
|
||||
done = 0
|
||||
|
||||
def step(n, text):
|
||||
nonlocal done
|
||||
done += n
|
||||
tell(min(done / total, 0.999), text)
|
||||
|
||||
tell(0.0, "Checking checksums")
|
||||
check_sums(root, files, report, lambda n: step(n, "Checking checksums"), cancelled)
|
||||
|
||||
manifest = None
|
||||
if "manifest.json" in files:
|
||||
manifest = read_json_file(os.path.join(root, "manifest.json"), "manifest.json", report)
|
||||
else:
|
||||
report.error("manifest.json is missing")
|
||||
entries = check_manifest(manifest, report, root) if manifest is not None else {}
|
||||
report.summary = summarize(manifest)
|
||||
|
||||
if "calibration.json" in files:
|
||||
cal = read_json_file(os.path.join(root, "calibration.json"), "calibration.json", report)
|
||||
if cal is not None:
|
||||
if not isinstance(cal, dict):
|
||||
report.error("calibration.json: not a JSON object")
|
||||
else:
|
||||
_, removed = strip_calibration(cal)
|
||||
if removed:
|
||||
report.error(f"calibration.json: identifying fields left in: {listed(removed)}")
|
||||
else:
|
||||
report.warn("no calibration.json: the recordings can't be used in 3D without it")
|
||||
if "device.json" in files:
|
||||
dev = read_json_file(os.path.join(root, "device.json"), "device.json", report)
|
||||
if dev is not None:
|
||||
check_device(dev, report)
|
||||
else:
|
||||
report.warn("no device.json (a session from before it was recorded): the labeller falls back to "
|
||||
"another unit's head frame")
|
||||
|
||||
take_dirs = sorted({rel.split("/")[1] for rel in files if rel.startswith("takes/") and rel.count("/") == 2})
|
||||
for tid in take_dirs:
|
||||
if tid not in entries:
|
||||
report.error(f"takes/{tid}/ isn't in the manifest")
|
||||
for tid, entry in entries.items():
|
||||
if tid not in take_dirs:
|
||||
report.error(f"take {tid} is in the manifest but has no folder")
|
||||
for tid in take_dirs:
|
||||
if cancelled():
|
||||
raise Cancelled()
|
||||
base = f"takes/{tid}/"
|
||||
title = (entries.get(tid) or {}).get("title") or tid
|
||||
for name in ("prompts.jsonl", "poses.jsonl"):
|
||||
rel = base + name
|
||||
if rel in sizes:
|
||||
check_jsonl(os.path.join(root, rel), rel, report, cancelled)
|
||||
else:
|
||||
report.warn(f"{rel} is missing")
|
||||
if base + "take.json" in sizes:
|
||||
tj = read_json_file(os.path.join(root, base + "take.json"), base + "take.json", report)
|
||||
if tj is not None and not isinstance(tj, dict):
|
||||
report.error(f"{base}take.json: not a JSON object")
|
||||
else:
|
||||
report.warn(f"{base}take.json is missing")
|
||||
rel = base + "sets.bin.zst"
|
||||
entry = entries.get(tid)
|
||||
if rel in sizes:
|
||||
check_sets(os.path.join(root, rel), rel, entry, report,
|
||||
lambda n, title=title: step(n, f"Checking {title}"), cancelled)
|
||||
elif entry is not None and entry.get("file"):
|
||||
report.error(f"{rel} is missing")
|
||||
|
||||
if report.bytes > MAX_BYTES:
|
||||
report.error(f"the export is {report.bytes / 1000 ** 3:.1f} GB, over the {MAX_BYTES / 1000 ** 3:.0f} GB limit")
|
||||
elif report.bytes > WARN_BYTES:
|
||||
report.warn(f"the export is large: {report.bytes / 1000 ** 3:.1f} GB")
|
||||
tell(1.0, "Checked")
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Check a hand recorder export (an exports/<session>/ folder).")
|
||||
ap.add_argument("dir")
|
||||
ap.add_argument("--json", action="store_true", help="print the result as JSON")
|
||||
a = ap.parse_args()
|
||||
if not os.path.isdir(a.dir):
|
||||
print(f"{a.dir}: no such folder", file=sys.stderr)
|
||||
return 2
|
||||
if hasattr(os, "setpriority"): # not on Windows
|
||||
try:
|
||||
os.setpriority(os.PRIO_PROCESS, 0, 19) # checksums and decompression: stay out of VR's way
|
||||
except OSError:
|
||||
pass
|
||||
report = validate(a.dir)
|
||||
if a.json:
|
||||
print(json.dumps(report.as_dict(), indent=2))
|
||||
else:
|
||||
for e in report.errors:
|
||||
print(f"error: {e}")
|
||||
for w in report.warnings:
|
||||
print(f"warning: {w}")
|
||||
s = report.summary
|
||||
if s:
|
||||
print(f"{s.get('session')}: {s.get('takes')} takes, {s.get('sets')} sets, {s.get('minutes')} min, "
|
||||
f"{report.bytes / 1000 ** 2:.1f} MB, consent {s.get('consent_version') or '-'}")
|
||||
print("OK" if report.ok else f"{len(report.errors)} errors")
|
||||
return 0 if report.ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -23,8 +23,9 @@ packages=(
|
||||
# tools (hands/tools) need NumPy and OpenCV, which aren't here: Fedora's python3-opencv pulls
|
||||
# in over a gigabyte (hands/README.md says how to get them)
|
||||
jsoncpp-devel glibc-static
|
||||
# hand recorder (hands/rec): its export compresses recordings with zstd
|
||||
zstd
|
||||
# hand recorder (hands/rec): its export compresses recordings with zstd; its Upload page and
|
||||
# hub.py upload to Hugging Face with huggingface_hub (also the hf command, for hf auth login)
|
||||
zstd python3-huggingface-hub
|
||||
# Frametop Input Settings app (Kirigami, PySide6)
|
||||
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland breeze-icon-theme plasma-breeze
|
||||
# Frametop remote desktop (VNC bridge through krdp)
|
||||
|
||||
Reference in new issue
Block a user