Hands: shorter recording sessions with pose sweeps

The second real session took 16 minutes, half of it 36 still poses. Labels
come from the auto-labeller, so what matters is variety, not clean holds.
A sweep step shows a strip of pose pictures and lights one every 4 s while
the hands move slowly near and far; each cue is a prompt event with
"cue": true. The core session is now 15 steps, about 5 minutes recorded.

The pose groups, the one-hand sweeps' groups and the cue order are shuffled
per session, seeded from its id and saved in session.json. A quick round
(--quick, or the checklist's choice) is about 2 minutes for extra lighting.
Touch the dot has 6 dots, the push sections two heights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-10-02 21:30:50 -06:00
1 parent 3ebae6a88f
commit 97b7fc837c
8 files changed
+631 -170

No files matched your search

+42 -19
View File
@@ -78,13 +78,14 @@ No name, email, or account. The contributor id is random, so several sessions fr
"checklist": {"objects": ["pencil", "phone", "cup", "keyboard", "mouse", "gamepad", "small"], "own_objects": ["..."],
"controllers": "straps|none", "sleeves": "short|long|", "rings": false, "watch": false, "notes": ""},
"device": {"steamos": "<VERSION_ID from /etc/os-release>", "steamvr": "<version if known>", "cameras": [{"name", "width", "height"}]},
"mode": "step|auto", "takes": ["01-hand-size", "..."],
"mode": "step|auto", "quick": false, "shuffle": {"seed": 123, "sweeps": {"pose-sweeps": [{"id", "hands", "cues"}]}},
"takes": ["01-hand-size", "..."],
"camera": {"status": "ok|unknown|degraded", "reason": "..."}, "stop_reason": "..."}
```
`camera` is the camera check's verdict at the start (no paths or log lines: those go to `session.log`). `stop_reason` is there when a session was stopped at the no-hands screen, with the check's result.
A session started before step mode existed has no `mode`: it ran as `auto`.
A session started before step mode existed has no `mode`: it ran as `auto`. `quick` and `shuffle` (below, "Sweeps" and "Quick round") are missing from sessions before the sweeps.
### calibration.json
@@ -132,7 +133,7 @@ prompt the hold: its labels start here
wait the hold is over: no labels from here; part N stops
```
The touch-the-dot targets after the first follow straight on: no `ready` or `wait` between them. `redo` marks a try done again (R): `from` is that step's `ready` (or its `prompt` if it had none), `to` its end. Its prompt is skipped; the sets stay. In auto mode there's no `ready` or `wait`, and the intro is a recorded prompt `<section>/intro`. `nohands` marks the first hand-size step stopped because no hand was seen (below, "Camera check"); a `redo` over the same range follows it, so the try gets no labels. Readers that knew only `prompt` and `end` keep working, but they'd give the countdown to the step before: `hub_review.py` (frame-hands `train/hub`) shows it as "(countdown)" and redone prompts as "(redone)"; `FORMAT.md` in the dataset repo has the rules.
The touch-the-dot targets after the first follow straight on: no `ready` or `wait` between them. `redo` marks a try done again (R): `from` is that step's `ready` (or its `prompt` if it had none), `to` its end. Its prompt is skipped; the sets stay. In auto mode there's no `ready` or `wait`, and the intro is a recorded prompt `<section>/intro`. `nohands` marks the first hand-size step stopped because no hand was seen (below, "Camera check"); a `redo` over the same range follows it, so the try gets no labels. A sweep step (below, "Sweeps") has a `prompt` per cue, each with its `pose`, `"cue": true` and `"step"`, the step's id; its `ready`, `wait` and `redo` carry the step's id. Readers that knew only `prompt` and `end` keep working, but they'd give the countdown to the step before: `hub_review.py` (frame-hands `train/hub`) shows it as "(countdown)", redone prompts as "(redone)" and cues as "[pose] text"; `FORMAT.md` in the dataset repo has the rules.
### poses.jsonl
@@ -171,6 +172,7 @@ Commands (UTF-8; `|` starts a new line in text):
| `big <text>` | Large cyan text under the instruction: the countdown (empty clears it). |
| `keys <text>` | The faint key hints along the bottom. |
| `rec on` / `rec off` | The red "Rec" by the step line. |
| `strip <cue> <path>\|<mode>\|<label>;...` / `strip off` | A sweep's row of pose pictures (path `-`: none; mode `-` or `mirror`), each with its label under it; the one at index `cue` (from 0, -1 for none) framed in cyan and bright, the others dim. It sits along the bottom of the space left, the where-to diagram at its right end if there is one, and the text above it; it takes the left column's place. Each picture is loaded once. A file that can't be read: `error can't read ...`, and that item shows an empty frame. |
| `target <x> <y> <z> [show\|hold <0..1>\|done]` / `target off` | The touch target: a small sphere-like dot about 2 cm across, in its own overlay (`frametop.handpanel.target`). The point is in the head frame (metres, +x right, +y up, -z forward). The first `target` with a new point places it in the room with the current HMD pose, and it stays there. Later commands with the same point change only the state. `hold` draws a filling ring, `done` turns it green. Reply: `ok <room x> <room y> <room z>`. |
| `poses start <path>` / `poses stop` | Log poses to the path (appending, `poses.jsonl` format above) from a thread at 250 Hz, until stopped. |
| `devices` | Reply: `ok hmd <r> left <r or -> right <r or ->`, the current `ETrackingResult` values (`-` for no device in that role). |
@@ -196,9 +198,13 @@ A pose with no entry, or whose file is missing, shows no picture: the text alone
```json
{"version": 1,
"cue_names": {"thumbs-up": "Thumbs up", "...": "..."},
"sections": [
{"id": "hand-size", "title": "Hand size", "requires": [], "intro": "text shown before the first prompt: 4 s, or until Next", "go": "Hold",
"prompts": [{"text": "...", "seconds": 8, "hands": "both", "pose": "flat", "distance": "near"}]},
"quick": true, "prompts": [{"text": "...", "cues": ["flat", "flat-back", "spread"], "cue_s": 5, "hands": "both", "distance": "mid"}]},
{"id": "pose-sweeps", "title": "Hand poses", "kind": "sweep", "quick": true, "cue_s": 4, "step_s": 20,
"groups": [["open", "fist", "point", "pinch"], ["ok", "thumbs-up", "spread", "claw"], ["count-1", "...", "count-5"]],
"sweeps": [{"hands": "both", "group": "next", "text": "..."}, {"hands": "left", "group": "any", "quick": false, "text": "..."}]},
{"id": "objects", "title": "Things you hold", "requires": ["objects"], "for_each": "object",
"prompts": [{"text": "Pick up the {object} and use it the way you normally would.", "seconds": 15, "hands": "both", "object": "{object}"}]},
{"id": "touch", "title": "Touch the dot", "kind": "targets", "hold_s": 1.0, "timeout_s": 8,
@@ -210,11 +216,29 @@ A pose with no entry, or whose file is missing, shows no picture: the text alone
- `requires`: `objects` (at least one object ticked), `controllers` (straps ticked). A section whose requirements aren't met is skipped and logged.
- `go`: the word the countdown ends on, "Go" unless set ("Hold" for the still poses).
- `quick`: the section is in a quick round; a step with `"quick": false` isn't (below).
- `kind`:
- `prompts` (the default): each prompt shows for its `seconds` with a countdown.
- `prompts` (the default): each prompt shows for its `seconds` with a countdown. A prompt with `cues` (and `cue_s`) is a sweep step with those cues in that order, `seconds` = cues x cue_s (hand size; the mouse and switch step).
- `sweep`: steps built from `sweeps` (below).
- `targets`: each target shows until the live index tip is within 3 cm of it for `hold_s`, or `timeout_s` passes.
- `bar`: the target marker sweeps near to far and back, `reps` times per height, at `period_s` per sweep. The current marker follows the live palm distance.
- Each section is one take. Prompts within it are marked in `prompts.jsonl`.
- `cue_names`: the short labels under the strip's pictures (otherwise the pose id).
### Sweeps
The second in-headset session (sessions/20261002-202734) took about 16 min and was "super long and kind of annoying": the 36 held poses alone took 7.9 min, 3.2 of it reading, and the person skipped the wrist turns and gave up on the bare push after 3 steps. The labels come from the auto-labeller (teacher model and triangulation, frame-hands `train/label`), not from the prompts, so what the recordings need is variety (shapes, distances, angles), not clean holds. So the poses are swept:
- A sweep step shows a strip of 2-5 pose pictures and asks for slow, continuous movement (near and far, all around) while the hands change shape with the lit picture. The light moves on every `cue_s` (4 s), cycling, over `step_s` (20 s): 5 cues for a group of 4 or 5. Each cue is a `prompt` event with that `pose`, `"cue": true` and `"step"` (the step's id); `distance` and `position` are "" (varied). So the timeline tags each pose roughly, and the countdown, `wait` and `redo` work per step as before. The time-left bar covers the whole step.
- `groups`: lists of poses. A step takes `"group": "next"` (the groups in turn) or `"any"` (one drawn at random, a different one for each "any" while there are groups left), or names its own `cues`. `"shuffle": false` (gestures) keeps the script's order; `"cycle": false` runs the cues once; `"fixed": true` keeps one step's order.
- The pose sweeps: 3 two-hand sweeps, one per group ([open, fist, point, pinch], [ok, thumbs-up, spread, claw], [count-1 ... count-5]); then a left-hand and a right-hand sweep (the other hand in the lap) on groups drawn from those three, which also turn the wrist as they go, so the wrist angles come for free (the old wrist-turns section is gone).
- **The shuffle, per session.** The groups' order, the "any" draws and the cues' order in each step are shuffled with a seed from the session's id (`session_seed`: the first 8 hex digits of its SHA-256), separately per section (`random.Random("<seed>/<section>")`). The sections' order isn't shuffled (the controller sections need their before and after). session.json records `"shuffle": {"seed", "sweeps": {section: [{"id", "hands", "cues"}]}}`, and `build_plan(script, checklist, seed=...)` gives the same again. `--seed N` overrides it; `--plan` without one shows the script's order.
- The strip has one picture per pose, a single hand: both hands make the same shape, and five pairs wouldn't fit. A left hand's pictures are flipped.
- Gestures are sweeps too, in order and once: tap then drag; grab, cross, overlap; near the face then a screen's distance. Hand size is one step of three held shapes (flat, backs, spread; 5 s each), still the no-hands check's first step.
### Quick round
For more lighting rounds: "Quick round (about 3 min)" on the checklist page, `session.py --quick`. It has the sections marked `quick` (hand size, the pose sweeps without the one-hand ones, touch the dot, no hands), about 2 min recorded in 6 steps. session.json gets `"quick": true`. The checklist page suggests it (and picks it) once a full session has gone to the end (`Backend.hasFullSession`: a session.json with status `done`, not quick, not a dry run).
### Step mode (the default) and auto mode
@@ -228,22 +252,21 @@ Steps that wait: each prompt, each bar height, and the first touch-the-dot targe
Auto mode ("Advance by itself" on the checklist page, `session.py --auto`) is the old timed flow: the welcome, the between and before screens, the intro (recorded) and each prompt for its seconds, one recording per take. R still works there: it restarts the step running.
Holds are 5 s for still poses (4 s counting fingers), 8-10 s for movements. The core session (no objects, no controllers) records about 11 min in 62 steps; with 5 s of reading a step that's about 16 min. Everything ticked: about 16.5 min recorded in 86 steps. The window and `session.py --plan` give these (`plan_summary`); in step mode they leave the reading time out and say so.
Steps are 20 s sweeps, 16-18 s gestures, 10-12 s desk and object steps, the touch targets (6) and the push heights (2, 3 reps of 6 s). The core session (no objects, no controllers) records about 5 min in 15 steps; with 5 s of reading a step that's about 6 min. Everything ticked (four objects, controllers): about 7.3 min recorded in 24 steps. A quick round: about 2 min in 6 steps. The window and `session.py --plan` give these (`plan_summary`); in step mode they leave the reading time out and say so.
The sections, in this order (see the plan):
1. hand size
2. static poses (both hands, then each hand: open, fist, point, pinch, OK, thumbs up, spread, claw, counting 1-5, at near, mid and far, and centre, left, right, up, down)
3. wrist rotations
4. gestures (pinch taps, pinch-drag, grabs, hands crossing and overlapping, hands near the face, hands at screen distance)
5. desk work (typing, mouse)
6. objects (one prompt per ticked object, own objects included)
7. touch the dot
8. controller depth, straps (push out and back at 3 heights and to each side, wrist turns, open and close). "Out and back" is straight away from the headset and back toward it: the first in-headset session took the left-right bar for sideways. The texts say so, the bar's ends read "At your chest" and "Arm out" (`near_label`, `far_label`, sent as `bar ... At your chest|Arm out`), and the picture is a side view.
9. bridge (one controller on, the bare fingertip touches the marked point on it at near, mid and far, then swap; then a controller on the desk, touched from several angles)
10. bare repeat of section 8, controllers off
11. no hands (10 s)
1. hand size (one step: flat, backs, spread)
2. pose sweeps (above: 3 with both hands, one with each hand)
3. gestures (3 sweeps: pinch taps and drags; grabs, crossing and overlapping; near the face, then pointing at screen distance)
4. desk work (typing or pretend typing; the mouse, with switching to the keyboard when both are there)
5. objects (one prompt per ticked object, own objects included)
6. touch the dot (6 dots spread near and far, left and right, low)
7. controller depth, straps (push out and back at chest and eye level, 3 reps each; then the wrists and fingers with the controllers on). "Out and back" is straight away from the headset and back toward it: the first in-headset session took the left-right bar for sideways. The texts say so, the bar's ends read "At your chest" and "Arm out" (`near_label`, `far_label`, sent as `bar ... At your chest|Arm out`), and the picture is a side view.
8. bridge (one controller on, the bare fingertip on its thumbstick while that hand moves from close to arm's length and back, then swap; then a controller on the desk, touched from the front and above, then from the sides: 4 steps, the controller changes during the ready screens)
9. bare repeat of section 7's pushes, controllers off
10. no hands (10 s)
Before section 8: "Put on both controllers and tighten the straps". Before section 10: "Take the controllers off and put them out of view".
Before section 7: "Put on both controllers and tighten the straps". Before section 9: "Take the controllers off and put them out of view".
### Feedback while recording
@@ -307,7 +330,7 @@ On 2026-10-02 a whole session showed "I can't see your hands": after the headset
Errors get a plain explanation: not logged in, a token Hugging Face rejects (401), a token that can't open a pull request (403), terms not accepted, dataset not found, network errors. `--dry-run` does everything except the network calls and the record, and lists what it would upload.
- **The page** shows the login (`whoami`, with "Check again"). If nobody is logged in, it explains how to run `distrobox enter dev -- hf auth login` in Konsole with a write token: the token goes only into that terminal. The page then has Upload and Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), and the pull request's link when it's done. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded.
- **While the texts are drafts**, Upload stays off unless `FT_HANDREC_ALLOW_UPLOAD=1`, so the maintainer can rehearse against a private test repo. `FT_HANDREC_DATASET` overrides `HF_DATASET`. `ft-handrec --hub-dry-run` makes Upload a dry run: no network, so it isn't held back by the drafts.
- **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start --next-after 0.3` records a two-section test script in step mode, three parts of 6 s (countdown and hold), about 360 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).
- **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start --next-after 0.3` records a three-section test script (a prompt, a two-cue sweep, no hands) in step mode, three parts of 6 s (countdown and hold), about 360 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).
## Licensing and consent (texts in `CONSENT.md`)
+20 -9
View File
@@ -6,8 +6,9 @@ A Kirigami (QML) app with a Python backend. It runs in the dev container:
changes. Agreeing writes profile.json with a random contributor id.
- Before you start: the checklist (objects, controller straps, lighting, sleeves, privacy,
free space), the camera check (hands/camcheck.py: Start stays off while the upper cameras
are off, with a Restart SteamVR button; --ignore-cameras overrides it) and what will happen.
Start hands it to the session runner (session.py).
are off, with a Restart SteamVR button; --ignore-cameras overrides it), a full session or a
quick round (suggested once a full one is done) and what will happen. Start hands it to
the session runner (session.py).
- Session: the runner's live status with the step's pose picture and where-to diagram, Next,
Pause/Resume, Redo, Skip section and Stop (Space: Next, P, R, S and Esc while the window has
focus; the button on the headset's right side is Next, pause and resume). The prompts appear
@@ -492,21 +493,31 @@ class Backend(QObject):
def sessionId(self):
return self._session_id
@Slot("QVariantMap", bool, result=str)
def planText(self, checklist, auto):
"""How long a session with these answers takes (session.plan_summary)."""
@Slot("QVariantMap", bool, bool, result=str)
def planText(self, checklist, auto, quick):
"""How long a session with these answers takes (session.plan_summary), or a quick round."""
mod = self._runner()
if not mod:
return ""
try:
script = mod.load_script(SCRIPT_PATH)
plan, _ = mod.build_plan(script, dict(checklist))
plan, _ = mod.build_plan(script, dict(checklist), quick=quick)
return mod.plan_summary(script, plan, auto=auto)
except Exception as e:
return f"Couldn't read the script: {e}"
@Slot("QVariantMap", str, bool, result=bool)
def startSession(self, checklist, lighting, auto):
@Property(bool, notify=sessionsChanged)
def hasFullSession(self):
"""A full session (not a quick round, not a dry run) went to the end: the window then
suggests a quick round, in another light."""
for s in self.store.sessions():
meta = takes.read_json(os.path.join(self.store.session_dir(s["id"]), "session.json"))
if meta.get("status") == "done" and not meta.get("quick") and not meta.get("dry_run"):
return True
return False
@Slot("QVariantMap", str, bool, bool, result=bool)
def startSession(self, checklist, lighting, auto, quick):
if self.sessionActive:
return False
mod = self._runner()
@@ -528,7 +539,7 @@ class Backend(QObject):
try:
self._session = mod.Session(self.store.base, self.store.profile(), checklist, lighting, SCRIPT_PATH,
on_status=lambda s: self._statusArrived.emit(dict(s)), auto=auto,
**self._session_options)
quick=quick, **self._session_options)
except Exception as e:
self.message.emit(f"Couldn't set up the session: {e}", True)
return False
+66 -3
View File
@@ -469,7 +469,8 @@ Kirigami.ApplicationWindow {
text: "Press Start, then put the headset on. A panel in the headset shows each step: a picture "
+ "of the hand pose, where to hold your hands, and what to do. The sections: hand poses, "
+ "gestures, typing and the mouse, your objects, touching a dot, and moves with the "
+ "controllers on and off. Each section is recorded as one take.\n\n"
+ "controllers on and off. Each section is recorded as one take. In the pose steps your "
+ "hands keep moving while a row of pictures lights up one shape after another: follow it.\n\n"
+ "Each step waits until you're ready: press the button on the right side of the headset, "
+ "or Space or Next in this window. A 3-2-1 "
+ "countdown follows, then hold the pose until the bar runs out. Nothing is recorded while "
@@ -478,6 +479,28 @@ Kirigami.ApplicationWindow {
+ "Nothing leaves the headset. Afterwards you watch the takes in Review, delete anything "
+ "you don't want to share, and only then export."
}
ColumnLayout {
Kirigami.FormData.label: "Round:"
Controls.RadioButton {
id: fullRound
text: "Full session"
checked: !backend.hasFullSession
}
Controls.RadioButton {
id: quickRound
text: "Quick round (about 3 min): hand size, poses, the dot, no hands"
checked: backend.hasFullSession
}
}
Kirigami.InlineMessage {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
Layout.fillWidth: true
visible: backend.hasFullSession
Layout.preferredHeight: visible ? implicitHeight : 0
type: Kirigami.MessageType.Information
text: "You've recorded a full session already. A quick round in a different light (dim, "
+ "daylight) adds the most now."
}
Controls.CheckBox {
id: autoAdvance
Kirigami.FormData.label: "Pace:"
@@ -490,14 +513,15 @@ Kirigami.ApplicationWindow {
text: (autoAdvance.checked
? "Each step shows for a few seconds and the next follows by itself. Quicker if you "
+ "know the steps already. "
: "") + "Length: " + backend.planText(checklist.answers(), autoAdvance.checked) + "."
: "") + "Length: " + backend.planText(checklist.answers(), autoAdvance.checked, quickRound.checked) + "."
}
Controls.Button {
text: "Start"
icon.name: "media-record"
enabled: checklist.ready && !backend.camerasBlockStart && !backend.restartingSteamVR
onClicked: {
if (backend.startSession(checklist.answers(), lighting.currentValue, autoAdvance.checked))
if (backend.startSession(checklist.answers(), lighting.currentValue, autoAdvance.checked,
quickRound.checked))
root.show(sessionPage)
}
}
@@ -778,6 +802,45 @@ Kirigami.ApplicationWindow {
font.pointSize: Kirigami.Theme.defaultFont.pointSize * 1.4
text: sessionView.st.prompt || ""
}
// A sweep's row of pictures, the lit one framed, as on the panel
Row {
visible: (sessionView.st.strip || []).length > 0
spacing: Kirigami.Units.largeSpacing
Repeater {
model: sessionView.st.strip || []
Column {
required property var modelData
required property int index
readonly property bool lit: index === sessionView.st.cue
spacing: Kirigami.Units.smallSpacing
Rectangle {
width: Kirigami.Units.gridUnit * 5.5
height: width
radius: Kirigami.Units.smallSpacing * 2
color: Qt.rgba(Kirigami.Theme.textColor.r, Kirigami.Theme.textColor.g,
Kirigami.Theme.textColor.b, 0.06)
border.width: parent.lit ? 3 : 0
border.color: "#4cd9ff"
Image {
anchors.fill: parent
anchors.margins: 4
source: modelData.image ? "file://" + modelData.image : ""
fillMode: Image.PreserveAspectFit
mirror: modelData.mode === "mirror"
opacity: parent.parent.lit ? 1 : 0.4
smooth: true
mipmap: true
}
}
Controls.Label {
anchors.horizontalCenter: parent.horizontalCenter
text: modelData.label || ""
color: parent.lit ? "#4cd9ff" : Kirigami.Theme.textColor
opacity: parent.lit ? 1 : 0.6
}
}
}
}
// The countdown's 3, 2, 1, then the hold's word, big
Controls.Label {
visible: !!sessionView.st.big && sessionView.state !== "paused"
+89 -6
View File
@@ -36,6 +36,12 @@
// big <text> large, under the instruction: the countdown's 3, 2, 1, then "Hold"
// keys <text> a small faint line along the bottom: the window's keys
// rec on|off a red "Rec" dot by the step line while recording
// strip <cue> <path>|<mode>|<label>;... / strip off
// a sweep's row of pose pictures under the text (path "-": none,
// mode "-" or "mirror"), each with its label; the one at index
// cue (from 0; -1: none) lit, framed in cyan, the others dim. The
// where-to diagram, if any, goes at the row's right end. It takes
// the place of the left column (image is ignored meanwhile)
// target <x> <y> <z> [show|hold <0..1>|done] / target off
// the touch target, about 2 cm across. The point is in the head
// frame (metres, +x right, +y up, -z forward). The first command
@@ -344,14 +350,27 @@ const Picture &Scaled(Image &img, int side, bool mirror) {
return out;
}
void DrawPicture(Picture &dst, const Picture &src, int x0, int y0) {
void DrawPicture(Picture &dst, const Picture &src, int x0, int y0, double opacity = 1) {
for (int y = 0; y < src.h; ++y)
for (int x = 0; x < src.w; ++x) {
const uint8_t *q = &src.px[(size_t(y) * src.w + x) * 4];
if (q[3]) Blend(dst, x0 + x, y0 + y, q[0] / 255.0, q[1] / 255.0, q[2] / 255.0, q[3] / 255.0);
if (q[3]) Blend(dst, x0 + x, y0 + y, q[0] / 255.0, q[1] / 255.0, q[2] / 255.0, opacity * q[3] / 255.0);
}
}
// A sweep's strip: its pictures, loaded once each (a strip changes only its lit one as it goes).
struct StripItem {
std::string path, mode, label;
};
std::map<std::string, Image> g_stripImages; // path -> picture (empty if it can't be read)
Image &StripImage(const std::string &path) {
auto [it, fresh] = g_stripImages.try_emplace(path);
if (fresh && !LoadImage(it->second, path))
std::fprintf(stderr, "ft-handpanel: can't read %s: %s\n", path.c_str(), stbi_failure_reason());
return it->second;
}
// The where-to diagram's cells: a front view, 3 x 3, (column, row) from the top left.
bool WhereCell(const std::string &pos, int &col, int &row) {
col = 1, row = 1;
@@ -434,6 +453,8 @@ struct Panel {
Image image; // the pose picture, if image.path isn't empty
std::string imageMode; // "", "mirror" or "both"
std::string wherePos, whereDist; // the diagram's, "" for none
std::vector<StripItem> strip; // a sweep's pictures, or none
int stripCue = -1; // the lit one
// What the last Draw laid out, for --no-vr's printout.
std::vector<std::string> lines, noteLines;
double textDeg = 0;
@@ -508,13 +529,34 @@ void Draw(Panel &p) {
bottom = ty - int(ring) - int(ppd * 0.8);
}
// The pose picture and the diagram in a column on the left, centred together; the text
// takes the rest. "both": the picture flipped (the left hand) beside it as it is.
// A sweep's strip: a row of pictures along the bottom of what's left, labels under them, the
// diagram at its right end; the text goes above it.
int textX0 = margin, textX1 = kW - margin;
int col, row;
const bool imageOn = !p.image.path.empty();
const bool imageOn = !p.image.path.empty() && p.strip.empty();
const bool whereOn = WhereCell(p.wherePos, col, row) || DistanceStep(p.whereDist) >= 0;
if (imageOn || whereOn) {
if (!p.strip.empty()) {
const int n = int(p.strip.size()), gap = int(ppd * 0.6), whereW = whereOn ? int(ppd * 9) : 0;
const int labelSize = int(ppd * 0.85), labelH = int(labelSize * 1.6), frame = std::max(3, int(ppd * 0.14));
const int rowW = width - (whereOn ? whereW + gap : 0);
const int side = std::max(8, std::min({int(ppd * 6), (rowW - (n - 1) * gap) / n, (bottom - top) / 2 - labelH}));
const int y0 = bottom - side - labelH, totalW = n * side + (n - 1) * gap;
int x = margin + (rowW - totalW) / 2;
for (int k = 0; k < n; ++k, x += side + gap) {
const StripItem &it = p.strip[k];
const bool lit = k == p.stripCue;
RoundRect(pic, x - frame, y0 - frame, x + side + frame, y0 + side + frame, ppd * 0.5, kCyan[0], kCyan[1],
kCyan[2], lit ? 0.95 : 0);
RoundRect(pic, x, y0, x + side, y0 + side, ppd * 0.4, 0.16, 0.16, 0.18, 1);
Image *img = it.path.empty() ? nullptr : &StripImage(it.path);
if (img && !img->path.empty()) DrawPicture(pic, Scaled(*img, side, it.mode == "mirror"), x, y0, lit ? 1.0 : 0.4);
if (lit) Text(pic, it.label, labelSize, x + side / 2, y0 + side + labelH / 2 + frame, kCyan[0], kCyan[1], kCyan[2]);
else Text(pic, it.label, labelSize, x + side / 2, y0 + side + labelH / 2 + frame, 0.6);
}
if (whereOn) DrawWhere(pic, kW - margin - whereW, y0 + (side + labelH - int(ppd * 4.4)) / 2, whereW, int(ppd * 4.4),
p.wherePos, p.whereDist);
bottom = y0 - frame - int(ppd * 0.8);
} else if (imageOn || whereOn) {
const int colW = int(ppd * 14), gap = int(ppd * 0.6);
const int whereH = whereOn ? int(ppd * 4.4) : 0, between = imageOn && whereOn ? gap : 0;
const bool both = p.imageMode == "both";
@@ -650,6 +692,15 @@ void Print(const Panel &p, const Target &t, bool visible, int number) {
if (!p.wherePos.empty() || !p.whereDist.empty())
std::printf("where: %s %s\n", p.wherePos.empty() ? "-" : p.wherePos.c_str(), p.whereDist.empty() ? "-" : p.whereDist.c_str());
else std::printf("where: off\n");
if (!p.strip.empty()) {
std::printf("strip: cue %d:", p.stripCue);
for (const StripItem &it : p.strip)
std::printf(" %s(%s%s)", it.label.c_str(), it.path.empty() ? "no picture" : it.path.c_str(),
it.mode.empty() ? "" : (" " + it.mode).c_str());
std::printf("\n");
} else {
std::printf("strip: off\n");
}
if (t.on)
std::printf("target: %s %.2f, head %.3f %.3f %.3f, room %.3f %.3f %.3f\n", t.state.c_str(), t.progress, t.head[0],
t.head[1], t.head[2], t.room[0], t.room[1], t.room[2]);
@@ -1049,6 +1100,38 @@ int main(int argc, char **argv) {
} else {
p.imageMode = mode, dirty = true;
}
} else if (Is(buf, "strip", &rest)) {
// strip <cue> <path>|<mode>|<label>;...
int cue = -1, used = 0;
if (!std::strcmp(rest, "off")) {
p.strip.clear(), p.stripCue = -1, dirty = true;
} else if (std::sscanf(rest, "%d %n", &cue, &used) >= 1 && rest[used]) {
std::vector<StripItem> items;
std::string all = rest + used;
for (size_t at = 0; at <= all.size();) {
const size_t end = std::min(all.find(';', at), all.size());
const std::string item = all.substr(at, end - at);
at = end + 1;
if (item.empty()) continue;
const size_t a1 = item.find('|'), a2 = a1 == std::string::npos ? a1 : item.find('|', a1 + 1);
StripItem it;
it.path = item.substr(0, a1);
if (a1 != std::string::npos) it.mode = item.substr(a1 + 1, a2 == std::string::npos ? a2 : a2 - a1 - 1);
if (a2 != std::string::npos) it.label = item.substr(a2 + 1);
if (it.path == "-") it.path.clear();
if (it.mode == "-") it.mode.clear();
items.push_back(it);
}
if (items.empty()) {
reply = "error usage: strip <cue> <path>|<mode>|<label>;... | strip off";
} else {
for (const StripItem &it : items)
if (!it.path.empty() && StripImage(it.path).path.empty()) reply = "error can't read " + it.path;
p.strip = items, p.stripCue = cue, dirty = true;
}
} else {
reply = "error usage: strip <cue> <path>|<mode>|<label>;... | strip off";
}
} else if (Is(buf, "where", &rest)) {
char pos[16] = "", dist[16] = "";
if (!std::strcmp(rest, "off")) {
+7 -6
View File
@@ -11,10 +11,10 @@
# drafts that also needs FT_HANDREC_ALLOW_UPLOAD=1 in the environment.
# --capture DIR the recording to play (default the frame-hands capture
# rec-20260930-103803-lit); --from S: start this far into it (60)
# --prompt-seconds N each test prompt's length (3). The session runs in step mode, Next
# pressed by itself (--next-after): each prompt is a recording part of
# its own, with its 3 s countdown, so about 18 s are recorded in all, a
# few hundred MB before compression
# --prompt-seconds N each test step's length (3): a prompt, a two-cue sweep, no hands. The
# session runs in step mode, Next pressed by itself (--next-after): each
# step is a recording part of its own, with its 3 s countdown, so about
# 18 s are recorded in all, a few hundred MB before compression
# --keep keep the temporary folders (camera images of a room: delete them after)
#
# It runs in the dev container (where ft-hands, ft-handpanel, zstd and huggingface_hub are),
@@ -152,8 +152,9 @@ json.dump({"version": 1, "intro_s": 1, "between_s": 1,
"sections": [
{"id": "hand-size", "title": "Hand size", "intro": "Rehearsal: hands.",
"prompts": [{"text": "Both hands flat, palms toward you.", "seconds": secs, "hands": "both",
"pose": "flat", "distance": "mid"},
{"text": "Fingers spread wide.", "seconds": secs, "hands": "both", "pose": "spread"}]},
"pose": "flat", "distance": "mid"}]},
{"id": "pose-sweeps", "title": "Hand poses", "kind": "sweep", "intro": "", "cue_s": secs / 2,
"groups": [["open", "fist"]], "sweeps": [{"hands": "both", "group": "next", "text": "Keep moving."}]},
{"id": "no-hands", "title": "No hands", "intro": "",
"prompts": [{"text": "Hands out of view.", "seconds": secs, "hands": "none"}]}]},
open(out, "w"))
+44 -104
View File
@@ -8,83 +8,45 @@
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
"stopped": {"title": "Stopped", "seconds": 4,
"text": "What was recorded so far is saved. Review it in the Hand recorder window."},
"cue_names": {"flat": "Flat", "flat-back": "Backs", "spread": "Spread", "open": "Open", "fist": "Fist",
"point": "Point", "pinch": "Pinch", "ok": "OK", "thumbs-up": "Thumbs up", "claw": "Claw", "count-1": "1",
"count-2": "2", "count-3": "3", "count-4": "4", "count-5": "5", "pinch-tap": "Tap", "pinch-drag": "Drag",
"grab": "Grab", "cross": "Cross", "overlap": "Overlap", "near-face": "Face", "screen-point": "Screen",
"mouse": "Mouse", "switch": "Switch"},
"objects": {"pencil": "pencil or pen", "phone": "phone", "cup": "cup", "keyboard": "keyboard",
"mouse": "mouse", "gamepad": "gamepad", "small": "small thing"},
"sections": [
{"id": "hand-size", "title": "Hand size", "requires": [], "go": "Hold",
"intro": "First, your hand size. Hold each pose still until the bar runs out.",
{"id": "hand-size", "title": "Hand size", "requires": [], "go": "Hold", "quick": true,
"intro": "First, your hand size. Hold each shape still while its picture is lit.",
"defaults": {"hands": "both", "distance": "mid", "position": "centre"},
"prompts": [
{"text": "Both hands flat, palms toward you, about 40 cm away. Fingers together.", "seconds": 6, "pose": "flat"},
{"text": "Turn them around: backs of your hands toward you.", "seconds": 5, "pose": "flat-back"},
{"text": "Palms toward you again, fingers spread wide.", "seconds": 5, "pose": "spread"}
{"text": "Both hands about 40 cm away, still. Make each lit shape: flat with palms toward you, then the backs toward you, then fingers spread wide.",
"cues": ["flat", "flat-back", "spread"], "cue_s": 5}
]},
{"id": "static-poses", "title": "Hand poses", "requires": [], "go": "Hold",
"intro": "Now some poses. Close means a hand's length away. Far means your arm stretched out.",
"prompts": [
{"text": "Both hands: open and relaxed. Halfway out, in front of you.", "seconds": 5, "hands": "both", "pose": "open", "distance": "mid", "position": "centre"},
{"text": "Both hands: make fists. Close to you, off to your left.", "seconds": 5, "hands": "both", "pose": "fist", "distance": "near", "position": "left"},
{"text": "Both hands: point with your index fingers. Arms out, off to your right.", "seconds": 5, "hands": "both", "pose": "point", "distance": "far", "position": "right"},
{"text": "Both hands: pinch, thumb and index tips touching. Halfway out, up high.", "seconds": 5, "hands": "both", "pose": "pinch", "distance": "mid", "position": "up"},
{"text": "Both hands: make an OK sign. Close to you, down low.", "seconds": 5, "hands": "both", "pose": "ok", "distance": "near", "position": "down"},
{"text": "Both hands: thumbs up. Arms out, in front of you.", "seconds": 5, "hands": "both", "pose": "thumbs-up", "distance": "far", "position": "centre"},
{"text": "Both hands: spread your fingers wide. Halfway out, off to your left.", "seconds": 5, "hands": "both", "pose": "spread", "distance": "mid", "position": "left"},
{"text": "Both hands: curl your fingers like claws. Close to you, off to your right.", "seconds": 5, "hands": "both", "pose": "claw", "distance": "near", "position": "right"},
{"text": "Now your left hand only. Rest your right hand in your lap.", "seconds": 4, "hands": "left"},
{"text": "Left hand: open and relaxed. Close to you, off to your right.", "seconds": 5, "hands": "left", "pose": "open", "distance": "near", "position": "right"},
{"text": "Left hand: a fist. Arm out, up high.", "seconds": 5, "hands": "left", "pose": "fist", "distance": "far", "position": "up"},
{"text": "Left hand: point with your index finger. Halfway out, down low.", "seconds": 5, "hands": "left", "pose": "point", "distance": "mid", "position": "down"},
{"text": "Left hand: pinch. Close to you, in front of you.", "seconds": 5, "hands": "left", "pose": "pinch", "distance": "near", "position": "centre"},
{"text": "Left hand: an OK sign. Arm out, off to your left.", "seconds": 5, "hands": "left", "pose": "ok", "distance": "far", "position": "left"},
{"text": "Left hand: thumbs up. Halfway out, off to your right.", "seconds": 5, "hands": "left", "pose": "thumbs-up", "distance": "mid", "position": "right"},
{"text": "Left hand: spread your fingers wide. Close to you, up high.", "seconds": 5, "hands": "left", "pose": "spread", "distance": "near", "position": "up"},
{"text": "Left hand: curl your fingers like a claw. Arm out, down low.", "seconds": 5, "hands": "left", "pose": "claw", "distance": "far", "position": "down"},
{"text": "Left hand, halfway out: show one finger.", "seconds": 4, "hands": "left", "pose": "count-1", "distance": "mid", "position": "centre"},
{"text": "Two fingers.", "seconds": 4, "hands": "left", "pose": "count-2", "distance": "mid", "position": "centre"},
{"text": "Three fingers.", "seconds": 4, "hands": "left", "pose": "count-3", "distance": "mid", "position": "centre"},
{"text": "Four fingers.", "seconds": 4, "hands": "left", "pose": "count-4", "distance": "mid", "position": "centre"},
{"text": "Five fingers.", "seconds": 4, "hands": "left", "pose": "count-5", "distance": "mid", "position": "centre"},
{"text": "Now your right hand only. Rest your left hand in your lap.", "seconds": 4, "hands": "right"},
{"text": "Right hand: open and relaxed. Arm out, down low.", "seconds": 5, "hands": "right", "pose": "open", "distance": "far", "position": "down"},
{"text": "Right hand: a fist. Halfway out, in front of you.", "seconds": 5, "hands": "right", "pose": "fist", "distance": "mid", "position": "centre"},
{"text": "Right hand: point with your index finger. Close to you, off to your left.", "seconds": 5, "hands": "right", "pose": "point", "distance": "near", "position": "left"},
{"text": "Right hand: pinch. Arm out, off to your right.", "seconds": 5, "hands": "right", "pose": "pinch", "distance": "far", "position": "right"},
{"text": "Right hand: an OK sign. Halfway out, up high.", "seconds": 5, "hands": "right", "pose": "ok", "distance": "mid", "position": "up"},
{"text": "Right hand: thumbs up. Close to you, down low.", "seconds": 5, "hands": "right", "pose": "thumbs-up", "distance": "near", "position": "down"},
{"text": "Right hand: spread your fingers wide. Arm out, in front of you.", "seconds": 5, "hands": "right", "pose": "spread", "distance": "far", "position": "centre"},
{"text": "Right hand: curl your fingers like a claw. Halfway out, off to your left.", "seconds": 5, "hands": "right", "pose": "claw", "distance": "mid", "position": "left"},
{"text": "Right hand, halfway out: show one finger.", "seconds": 4, "hands": "right", "pose": "count-1", "distance": "mid", "position": "centre"},
{"text": "Two fingers.", "seconds": 4, "hands": "right", "pose": "count-2", "distance": "mid", "position": "centre"},
{"text": "Three fingers.", "seconds": 4, "hands": "right", "pose": "count-3", "distance": "mid", "position": "centre"},
{"text": "Four fingers.", "seconds": 4, "hands": "right", "pose": "count-4", "distance": "mid", "position": "centre"},
{"text": "Five fingers.", "seconds": 4, "hands": "right", "pose": "count-5", "distance": "mid", "position": "centre"}
{"id": "pose-sweeps", "title": "Hand poses", "kind": "sweep", "requires": [], "quick": true,
"intro": "Now keep your hands moving slowly while they change shape. A picture lights up every few seconds: make that shape wherever your hands are. Close, far, high, low: all of it helps.",
"cue_s": 4, "step_s": 20,
"groups": [["open", "fist", "point", "pinch"], ["ok", "thumbs-up", "spread", "claw"],
["count-1", "count-2", "count-3", "count-4", "count-5"]],
"sweeps": [
{"hands": "both", "group": "next", "text": "Both hands: move them slowly and keep going, from close to you out to arm's length and all around in front of you. Change shape with the lit picture."},
{"hands": "both", "group": "next", "text": "Both hands again: slowly near and far, high and low, left and right. Change shape with the lit picture."},
{"hands": "both", "group": "next", "text": "Both hands, still moving slowly all around in front of you. Change shape with the lit picture."},
{"hands": "left", "group": "any", "quick": false, "text": "Left hand only, the right one in your lap. Move it slowly near and far and all around, change shape with the lit picture, and turn your wrist as you go: palm toward you, then away."},
{"hands": "right", "group": "any", "quick": false, "text": "Right hand only, the left one in your lap. Move it slowly near and far and all around, change shape with the lit picture, and turn your wrist as you go: palm toward you, then away."}
]},
{"id": "wrist-rotations", "title": "Wrist turns", "requires": [],
"intro": "Slow wrist turns. Keep your fingers relaxed.",
"defaults": {"pose": "open", "distance": "mid"},
"prompts": [
{"text": "Both hands open, palms toward you. Slowly turn them palms away, then back.", "seconds": 10, "hands": "both", "pose": "turn-in-out"},
{"text": "Left hand, fingers up. Slowly turn your wrist: palm in, palm out.", "seconds": 9, "hands": "left", "pose": "turn-in-out"},
{"text": "Right hand, fingers up. Slowly turn your wrist: palm in, palm out.", "seconds": 9, "hands": "right", "pose": "turn-in-out"},
{"text": "Both hands flat, palms down. Slowly turn them palms up, then down again.", "seconds": 10, "hands": "both", "pose": "turn-up-down"},
{"text": "Bend both wrists slowly up and down, then side to side.", "seconds": 10, "hands": "both", "pose": "bend"}
]},
{"id": "gestures", "title": "Gestures", "requires": [],
"intro": "Now some movements. Go at a calm, normal pace.",
{"id": "gestures", "title": "Gestures", "kind": "sweep", "requires": [], "shuffle": false, "cycle": false,
"intro": "Now some movements, at a calm, normal pace. Follow the lit picture.",
"defaults": {"hands": "both"},
"prompts": [
{"text": "Tap your thumb and index finger together a few times, both hands.", "seconds": 8, "pose": "pinch-tap"},
{"text": "Pinch, move your hand slowly to the side, then let go. Do it again.", "seconds": 10, "pose": "pinch-drag"},
{"text": "Close your hands as if grabbing a bar, then open them. A few times.", "seconds": 8, "pose": "grab"},
{"text": "Cross your hands in front of you, then uncross them. Slowly.", "seconds": 10, "pose": "cross"},
{"text": "Hold one hand over the other, palms toward you, and slide them around.", "seconds": 8, "pose": "overlap"},
{"text": "Bring your hands close to your face and move them around it. Don't touch the headset.", "seconds": 8, "pose": "near-face", "distance": "near"},
{"text": "Reach out to where a screen would be, about arm's length, and point at things on it.", "seconds": 8, "pose": "screen-point", "distance": "far"}
"sweeps": [
{"id": "pinch", "cues": ["pinch-tap", "pinch-drag"], "cue_s": 8,
"text": "Both hands: tap thumb and index finger together a few times. Then pinch, drag slowly to the side and let go, again and again."},
{"id": "grab-cross", "cues": ["grab", "cross", "overlap"], "cue_s": 6,
"text": "Grab as if holding a bar and open again. Then cross your hands and uncross them. Then hold one over the other and slide them around."},
{"id": "face-screen", "cues": ["near-face", "screen-point"], "cue_s": 8,
"text": "Move your hands close around your face, without touching the headset. Then reach out to where a screen would be and point at things on it."}
]},
{"id": "desk-work", "title": "Desk work", "requires": [],
@@ -93,10 +55,9 @@
"prompts": [
{"when": "objects:keyboard", "text": "Type on your keyboard the way you normally do. Any text is fine.", "seconds": 10, "pose": "typing", "object": "keyboard"},
{"when": "!objects:keyboard", "text": "Rest your hands on the desk and pretend to type.", "seconds": 10, "pose": "typing"},
{"when": "objects:keyboard", "text": "Lift your hands off the keyboard and put them back, a few times.", "seconds": 8, "pose": "lift", "object": "keyboard"},
{"when": "objects:mouse", "text": "Use your mouse: move it around and click a few times.", "seconds": 10, "hands": "right", "pose": "mouse", "object": "mouse"},
{"when": "!objects:mouse", "text": "Put one hand flat on the desk and slide it around, as if using a mouse.", "seconds": 10, "hands": "right", "pose": "mouse"},
{"when": "objects:keyboard", "text": "Switch between typing and the mouse a few times.", "seconds": 10, "pose": "switch", "object": "keyboard"}
{"when": ["objects:mouse", "objects:keyboard"], "text": "Use your mouse: move it and click. When the picture changes, switch between the mouse and the keyboard a few times.", "cues": ["mouse", "switch"], "cue_s": 6, "object": "mouse"},
{"when": ["objects:mouse", "!objects:keyboard"], "text": "Use your mouse: move it around and click a few times.", "seconds": 10, "hands": "right", "pose": "mouse", "object": "mouse"},
{"when": "!objects:mouse", "text": "Put one hand flat on the desk and slide it around, as if using a mouse.", "seconds": 10, "hands": "right", "pose": "mouse"}
]},
{"id": "objects", "title": "Things you hold", "requires": ["objects"], "for_each": "object",
@@ -107,69 +68,48 @@
{"text": "Pick up the {object}, use it for a moment, then put it down.", "seconds": 12, "pose": "hold", "object": "{object}"}
]},
{"id": "touch", "title": "Touch the dot", "kind": "targets", "requires": [],
{"id": "touch", "title": "Touch the dot", "kind": "targets", "requires": [], "quick": true,
"intro": "A small dot will appear. Touch it with your index fingertip and hold still until it turns green.",
"defaults": {"hands": "any", "pose": "point"},
"hold_s": 1.0, "timeout_s": 8,
"text": "Touch the dot with your index fingertip and hold still.",
"targets": [[0.00, -0.15, -0.40], [-0.20, -0.10, -0.40], [0.20, -0.10, -0.40], [0.00, -0.05, -0.50],
[-0.15, -0.25, -0.35], [0.15, -0.25, -0.35], [0.00, -0.20, -0.25], [-0.25, -0.05, -0.50],
"targets": [[0.00, -0.20, -0.25], [-0.20, -0.10, -0.40], [0.20, -0.10, -0.40], [-0.15, -0.25, -0.35],
[0.25, -0.05, -0.50], [0.05, -0.10, -0.55]]},
{"id": "controller-push", "title": "Depth with controllers", "kind": "bar", "requires": ["controllers"],
"before": {"text": "Put on both controllers and tighten the straps. Take your time.", "seconds": 25},
"intro": "Push both hands straight out in front of you, away from the headset, then pull them back toward it. Only nearer and farther, not sideways. The marker on the bar shows how far: right is arm out, left is at your chest.",
"defaults": {"hands": "both", "pose": "open", "picture": "push-controller", "controller": true, "controllers": ["left", "right"]},
"near_m": 0.2, "far_m": 0.6, "near_label": "At your chest", "far_label": "Arm out", "period_s": 6, "lead_s": 3, "reps": 5,
"near_m": 0.2, "far_m": 0.6, "near_label": "At your chest", "far_label": "Arm out", "period_s": 6, "lead_s": 3, "reps": 3,
"heights": [
{"id": "chest", "text": "Chest height, palms out like a stop sign. Push straight out, away from the headset, then pull back to your chest. Follow the marker."},
{"id": "desk", "text": "Now at desk height, palms out. Push straight out, away from you, then pull back toward you. Follow the marker."},
{"id": "eye", "text": "Now at eye level, palms out. Push straight out, away from the headset, then pull back toward it. Follow the marker."},
{"id": "left", "text": "Now hold both hands off to your left. From there, push straight out and pull back toward you, not sideways. Follow the marker.", "reps": 2},
{"id": "right", "text": "Now hold both hands off to your right. From there, push straight out and pull back toward you, not sideways. Follow the marker.", "reps": 2}
{"id": "eye", "text": "Now at eye level, palms out. Push straight out, away from the headset, then pull back toward it. Follow the marker."}
],
"prompts": [
{"text": "Slowly turn your wrists: palms out, palms toward you, and back.", "seconds": 10, "pose": "turn-in-out", "distance": "mid"},
{"text": "Open and close your hands a few times.", "seconds": 8, "pose": "open-close", "distance": "mid"}
{"text": "Controllers on: slowly turn your wrists, and open and close your hands, a few times.", "seconds": 10, "pose": "open-close", "distance": "mid"}
]},
{"id": "bridge", "title": "Controller and fingertip", "requires": ["controllers"],
"intro": "Now one controller on and one hand bare. The thumbstick is the point to touch.",
"defaults": {"controller": true},
"prompts": [
{"text": "Keep the left controller on. Take the right one off and put it on the desk.", "seconds": 8, "hands": "both", "controllers": ["left"]},
{"text": "Left hand close to you. Touch the left thumbstick with your right index fingertip.", "seconds": 6, "hands": "both", "pose": "touch-stick", "distance": "near", "controllers": ["left"]},
{"text": "Keep touching it while you move your left hand halfway out.", "seconds": 6, "hands": "both", "pose": "touch-stick", "distance": "mid", "controllers": ["left"]},
{"text": "Keep touching it with your left arm stretched out.", "seconds": 6, "hands": "both", "pose": "touch-stick", "distance": "far", "controllers": ["left"]},
{"text": "Swap: put the right controller on, then take the left one off.", "seconds": 12, "hands": "both"},
{"text": "Right hand close to you. Touch the right thumbstick with your left index fingertip.", "seconds": 6, "hands": "both", "pose": "touch-stick", "picture_mirror": true, "distance": "near", "controllers": ["right"]},
{"text": "Keep touching it while you move your right hand halfway out.", "seconds": 6, "hands": "both", "pose": "touch-stick", "picture_mirror": true, "distance": "mid", "controllers": ["right"]},
{"text": "Keep touching it with your right arm stretched out.", "seconds": 6, "hands": "both", "pose": "touch-stick", "picture_mirror": true, "distance": "far", "controllers": ["right"]},
{"text": "Take off the right controller and put it on the desk in front of you.", "seconds": 8, "hands": "both"},
{"text": "Touch its thumbstick with your index fingertip, from the front.", "seconds": 5, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "down"},
{"text": "Now touch the thumbstick reaching from the left.", "seconds": 5, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "left"},
{"text": "Now from the right.", "seconds": 5, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "right"},
{"text": "Now from straight above, with your other hand.", "seconds": 5, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "up"}
{"text": "Left controller on, the right one off on the desk. Touch the left thumbstick with your right index fingertip, and keep touching it while you move your left hand slowly from close to you out to arm's length and back.", "seconds": 12, "hands": "both", "pose": "touch-stick", "controllers": ["left"]},
{"text": "Swap: right controller on, the left one off. Touch the right thumbstick with your left index fingertip, and keep touching it while you move your right hand slowly from close to you out to arm's length and back.", "seconds": 12, "hands": "both", "pose": "touch-stick", "picture_mirror": true, "controllers": ["right"]},
{"text": "Put the right controller on the desk in front of you. Touch its thumbstick with your index fingertip from the front, then from above.", "seconds": 8, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "down"},
{"text": "Now touch it reaching from the left, then from the right.", "seconds": 8, "hands": "any", "pose": "touch-stick", "picture": "touch-stick-desk", "position": "down"}
]},
{"id": "bare-push", "title": "Depth, bare hands", "kind": "bar", "requires": [],
"before": {"when": "controllers", "text": "Take the controllers off and put them out of view.", "seconds": 15},
"intro": "Push both hands straight out in front of you, away from the headset, then pull them back toward it. Only nearer and farther, not sideways. The marker on the bar shows how far: right is arm out, left is at your chest.",
"defaults": {"hands": "both", "pose": "open", "picture": "push"},
"near_m": 0.2, "far_m": 0.6, "near_label": "At your chest", "far_label": "Arm out", "period_s": 6, "lead_s": 3, "reps": 4,
"near_m": 0.2, "far_m": 0.6, "near_label": "At your chest", "far_label": "Arm out", "period_s": 6, "lead_s": 3, "reps": 3,
"heights": [
{"id": "chest", "text": "Chest height, palms out like a stop sign. Push straight out, away from the headset, then pull back to your chest. Follow the marker."},
{"id": "desk", "text": "Now at desk height, palms out. Push straight out, away from you, then pull back toward you. Follow the marker."},
{"id": "eye", "text": "Now at eye level, palms out. Push straight out, away from the headset, then pull back toward it. Follow the marker."},
{"id": "left", "text": "Now hold both hands off to your left. From there, push straight out and pull back toward you, not sideways. Follow the marker.", "reps": 2},
{"id": "right", "text": "Now hold both hands off to your right. From there, push straight out and pull back toward you, not sideways. Follow the marker.", "reps": 2}
],
"prompts": [
{"text": "Slowly turn your wrists: palms out, palms toward you, and back.", "seconds": 10, "pose": "turn-in-out", "distance": "mid"},
{"text": "Open and close your hands a few times.", "seconds": 8, "pose": "open-close", "distance": "mid"}
{"id": "eye", "text": "Now at eye level, palms out. Push straight out, away from the headset, then pull back toward it. Follow the marker."}
]},
{"id": "no-hands", "title": "No hands", "requires": [],
{"id": "no-hands", "title": "No hands", "requires": [], "quick": true,
"intro": "Last one: put your hands down, out of view.",
"prompts": [
{"text": "Keep your hands out of view and look slowly around the room.", "seconds": 10, "hands": "none", "picture": "no-hands"}
+187 -23
View File
@@ -26,10 +26,12 @@ All _ns times are CLOCK_MONOTONIC nanoseconds.
"""
import argparse
import glob
import hashlib
import json
import math
import mmap
import os
import random
import re
import select
import shutil
@@ -742,9 +744,12 @@ PROMPT_KEYS = ("text", "seconds", "hands", "pose", "distance", "position", "obje
def _when_ok(cond, ctx):
"""A prompt's or block's "when": "controllers", "objects:keyboard", or either negated with "!"."""
"""A prompt's or block's "when": "controllers", "objects:keyboard", either negated with "!",
or a list of them, all to hold."""
if not cond:
return True
if isinstance(cond, list): # all of them
return all(_when_ok(c, ctx) for c in cond)
neg = cond.startswith("!")
c = cond.lstrip("!")
value = c[len("objects:"):] in ctx["objects"] if c.startswith("objects:") else bool(ctx.get(c))
@@ -790,9 +795,67 @@ def load_script(path):
return script
def build_plan(script, checklist):
def session_seed(session_id):
"""The shuffle's seed for a session: from its id, so the same id gives the same plan."""
return int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16)
def _sweep_prompts(s, raw, ctx, rng):
"""A sweep section's steps (DESIGN.md, "Sweeps"): each a prompt with "cues", the pose of each
cue_s slot in order. A step names its cues, or takes a group: "next" (the groups in turn, in
the shuffled order) or "any" (a group drawn at random, different for each "any" while there
are groups left). With rng (the session's shuffle) the groups' order, the "any" draws and the
cues' order within a step are shuffled, unless the section says "shuffle": false or the
step "fixed": true; without it, everything goes in the script's order."""
groups = [list(g) for g in raw.get("groups") or []]
shuffle = rng is not None and raw.get("shuffle", True)
order = list(range(len(groups)))
pool = list(range(len(groups)))
if shuffle:
rng.shuffle(order)
rng.shuffle(pool)
taken, drawn, counts = 0, 0, {}
out = []
for step in raw.get("sweeps") or []:
if not _when_ok(step.get("when"), ctx):
continue
g = step.get("group")
if step.get("cues"):
cues = list(step["cues"])
elif g == "next" and groups:
cues, taken = list(groups[order[taken % len(order)]]), taken + 1
elif g == "any" and groups:
cues, drawn = list(groups[pool[drawn % len(pool)]]), drawn + 1
elif isinstance(g, int) and 0 <= g < len(groups):
cues = list(groups[g])
else:
raise ValueError("%s: a sweep needs cues or a group" % s["id"])
if shuffle and not step.get("fixed"):
rng.shuffle(cues)
cue_s = float(step.get("cue_s", raw.get("cue_s", 4)))
slots = max(len(cues), int(math.ceil(float(step.get("seconds", raw.get("step_s", cue_s * len(cues)))) / cue_s
- 1e-9))) if step.get("cycle", raw.get("cycle", True)) else len(cues)
p = _prompt(s, {k: v for k, v in step.items() if k not in ("group", "cues", "fixed", "cycle", "quick")}, ctx)
p["cues"] = [cues[k % len(cues)] for k in range(slots)]
p["cue_s"] = cue_s
p["seconds"] = cue_s * slots
p["pose"] = p["cues"][0]
if not step.get("id"):
counts[p["hands"]] = counts.get(p["hands"], 0) + 1
p["id"] = "%s/%s-%d" % (s["id"], p["hands"] or "any", counts[p["hands"]])
else:
p["id"] = "%s/%s" % (s["id"], step["id"])
if step.get("quick") is False:
p["quick"] = False
out.append(p)
return out
def build_plan(script, checklist, seed=None, quick=False):
"""The sections this session runs, prompts expanded for the checklist, and the ones
skipped: (plan, skipped) with skipped = [{"section", "reason"}]."""
skipped: (plan, skipped) with skipped = [{"section", "reason"}]. seed: the shuffle of the
sweeps (session_seed; None: the script's order). quick: the quick round, only sections
marked "quick" and, in those, no step marked "quick": false."""
objects = [o for o in (checklist.get("objects") or []) if o]
own = [o for o in (checklist.get("own_objects") or []) if str(o).strip()]
ctx = {"objects": set(objects) | set(own), "controllers": checklist.get("controllers") == "straps"}
@@ -800,6 +863,9 @@ def build_plan(script, checklist):
plan, skipped = [], []
for raw in script["sections"]:
sid = raw["id"]
if quick and not raw.get("quick"):
skipped.append({"section": sid, "reason": "not in a quick round"})
continue
missing = [r for r in raw.get("requires") or [] if not ctx.get(r)]
if missing:
skipped.append({"section": sid, "reason": "needs " + ", ".join(missing)})
@@ -808,7 +874,12 @@ def build_plan(script, checklist):
s["kind"] = raw.get("kind", "prompts")
s["intro_s"] = float(raw.get("intro_s", script.get("intro_s", 4)))
prompts = []
if raw.get("for_each") == "object":
if s["kind"] == "sweep":
rng = random.Random("%d/%s" % (seed, sid)) if seed is not None else None
prompts = _sweep_prompts(s, raw, ctx, rng)
if quick:
prompts = [p for p in prompts if p.get("quick") is not False]
elif raw.get("for_each") == "object":
skip = set(raw.get("skip_objects") or [])
for o in [o for o in objects if o not in skip] + own:
label = clean_text(names.get(o, o)).replace("|", "/")
@@ -816,7 +887,14 @@ def build_plan(script, checklist):
if _when_ok(p.get("when"), ctx):
prompts.append(_prompt(s, p, ctx, {"object": label, "object_key": clean_text(o)}))
else:
prompts = [_prompt(s, p, ctx) for p in raw.get("prompts") or [] if _when_ok(p.get("when"), ctx)]
prompts = [_prompt(s, p, ctx) for p in raw.get("prompts") or [] if _when_ok(p.get("when"), ctx)
and not (quick and p.get("quick") is False)]
for p in prompts: # a prompt may have cues too (hand size): its own, in its order
if p.get("cues"):
cue_s = float(p.get("cue_s", raw.get("cue_s", 4)))
p["cues"], p["cue_s"] = list(p["cues"]), cue_s
p["seconds"] = cue_s * len(p["cues"])
p["pose"] = p["cues"][0]
_set_ids(s, prompts)
s["prompts"] = prompts
if s["kind"] == "bar":
@@ -829,7 +907,7 @@ def build_plan(script, checklist):
s["heights"] = heights
before = raw.get("before")
s["before"] = before if before and _when_ok(before.get("when"), ctx) else None
if s["kind"] == "prompts" and not prompts:
if s["kind"] in ("prompts", "sweep") and not prompts:
skipped.append({"section": sid, "reason": "nothing to do"})
continue
if s["kind"] == "targets" and not raw.get("targets"):
@@ -839,6 +917,16 @@ def build_plan(script, checklist):
return plan, skipped
def plan_record(plan):
"""What the shuffle chose, for session.json: each section's sweep steps and their cues."""
out = {}
for s in plan:
steps = [{"id": p["id"], "hands": p["hands"], "cues": p["cues"]} for p in s["prompts"] if p.get("cues")]
if steps:
out[s["id"]] = steps
return out
def section_steps(s):
"""The steps that wait for Next in step mode: each prompt and bar height, and the first target."""
return (len(s["prompts"]) + (len(s.get("heights") or []) if s["kind"] == "bar" else 0)
@@ -875,7 +963,7 @@ def plan_steps(plan):
def plan_summary(script, plan, auto=False):
"""The length in words, for the window and --plan."""
"""The length in words, for the window and --plan (the plan of a quick round or a full one)."""
minutes = max(1, round(plan_seconds(script, plan, auto=auto) / 60))
if auto:
return "about %d min, each step advancing by itself" % minutes
@@ -971,12 +1059,13 @@ class Session:
headset's button (not in a dry run unless button_device, a test hook, names the device or
a FIFO of input_event structs). check_cameras: run the camera check first and don't start
if it fails (not in a dry run, nor with ring). hands_reader (a test hook): an object whose
read() stands in for the hands file, read in a dry run too."""
read() stands in for the hands file, read in a dry run too. quick: the quick round (a few
sections, about 3 min). seed: the sweeps' shuffle (default: from the session's id)."""
def __init__(self, base_dir, profile, checklist, lighting_choice, script_path, *, ring=None,
start_processes=True, dry_run=False, speed=1.0, on_status=None, hands_dir=None, panel_bin=None,
auto=False, next_after=None, poses_dir=None, button=True, button_device=None,
check_cameras=True, hands_reader=None):
check_cameras=True, hands_reader=None, quick=False, seed=None):
self.base_dir = os.path.abspath(os.path.expanduser(base_dir or BASE_DIR))
self.profile = dict(profile or {})
self.checklist = dict(checklist or {})
@@ -993,13 +1082,16 @@ class Session:
self.next_after = next_after
self.button, self.button_device = bool(button), button_device
self.check_cameras = bool(check_cameras)
self.quick = bool(quick)
self.seed = seed # set from the session's id when it starts, unless given
# the camera check (a test hook: tests swap it); a dry run looks at no real cameras
self.camera_check_fn = (lambda: None) if dry_run else camera_check
self.input_devices = INPUT_DEVICES # where mice are looked for (a test hook)
self._button = None # the ButtonReader
self.print = print # where dry-run panel commands go (the CLI's stdout)
self.script = load_script(self.script_path)
self.plan, self.skipped = build_plan(self.script, self.checklist)
# the plan in the script's order; the session's own shuffle comes with its id (_make_dir)
self.plan, self.skipped = build_plan(self.script, self.checklist, seed=seed, quick=self.quick)
self._poses = load_poses(poses_dir or POSES_DIR)
self.session_dir = ""
self._thread = None
@@ -1016,7 +1108,7 @@ class Session:
"take": None, "error": "", "waiting": False, "countdown": 0, "big": "", "can_redo": False,
"image": "", "image_mode": "", "caption": "", "position": "", "distance": "",
"ready_text": READY_TEXT, "button": False, "mouse": True,
"camera": None, "nohands": False}
"camera": None, "nohands": False, "strip": [], "cue": -1, "quick": self.quick}
self._last_emit = 0.0
self._log_file = None
self._panel = None
@@ -1176,6 +1268,10 @@ class Session:
path = os.path.join(sessions, "%s-%d" % (sid, n))
os.mkdir(os.path.join(path, "takes"))
self.session_dir = path
if self.seed is None:
self.seed = session_seed(os.path.basename(path))
# the same sections and steps, the sweeps shuffled for this session
self.plan, self.skipped = build_plan(self.script, self.checklist, seed=self.seed, quick=self.quick)
self._log_file = open(os.path.join(path, "session.log"), "a", buffering=1)
self._log("session %s%s, script %s" % (os.path.basename(path), " (dry run)" if self.dry_run else "",
self.script_path))
@@ -1219,7 +1315,9 @@ class Session:
"calibration_removed": removed,
"script": {"version": self.script.get("version"), "sections": [s["id"] for s in self.plan],
"skipped": self.skipped},
"mode": "auto" if self.auto else "step", "takes": [], "status": "recording"}
"mode": "auto" if self.auto else "step", "quick": self.quick,
"shuffle": {"seed": self.seed, "sweeps": plan_record(self.plan)},
"takes": [], "status": "recording"}
cam = self._status.get("camera")
if cam:
self._session_json["camera"] = {"status": cam.get("status"), "reason": cam.get("reason", "")}
@@ -1236,7 +1334,7 @@ class Session:
self._panel.cmd("paused off")
for key, c in (("note", "note "), ("countdown", "countdown off"), ("hands", "hands off off"),
("bar", "bar off"), ("target", "target off"), ("image", "image off"), ("where", "where off"),
("big", "big "), ("action", "action "), ("rec", "rec off")):
("big", "big "), ("action", "action "), ("rec", "rec off"), ("strip", "strip off")):
self._panel.set(key, c)
self._hints(action=False)
@@ -1416,7 +1514,7 @@ class Session:
try:
self._panel.cmd("paused off")
for key, c in (("big", "big "), ("action", "action "), ("rec", "rec off"), ("keys", "keys "),
("bar", "bar off"), ("target", "target off")):
("bar", "bar off"), ("target", "target off"), ("strip", "strip off")):
self._panel.set(key, c)
self._screen(state, screen, controls=False)
except (_Stop, _Skip, _Fail):
@@ -1430,7 +1528,7 @@ class Session:
prompt=error or (self._stop_note if state == "stopped" else "") or screen.get("text", "").replace("|", "\n"),
hands={"left": None, "right": None}, nohands=False,
waiting=False, countdown=0, big="", can_redo=False, image="", image_mode="", caption="",
position="", distance="")
position="", distance="", strip=[], cue=-1)
# --- the timing loop
def _controls(self):
@@ -1634,13 +1732,36 @@ class Session:
"""The prompt's picture and where-to diagram, on the panel and in the status (sent with
the next _emit); none without a prompt."""
p = p or {}
path, mode, caption = pose_view(self._poses, p)
path, mode, caption = ("", "", "") if p.get("cues") else pose_view(self._poses, p)
self._strip(p)
pos, dist = p.get("position") or "", p.get("distance") or ""
self._panel.set("image", "image %s%s" % (path, " " + mode if mode else "") if path else "image off")
self._panel.set("where", "where %s %s" % (clean_text(pos) or "-", clean_text(dist) or "-")
if pos or dist else "where off")
self._status.update(image=path, image_mode=mode, caption=caption, position=pos, distance=dist)
def _cue_name(self, pose):
return clean_text((self.script.get("cue_names") or {}).get(pose) or pose.replace("-", " ").capitalize())
def _strip(self, p):
"""A sweep's strip of pictures, the cue's highlighted (panel "strip", status strip and
cue), or none. One hand per picture, as both hands make the same shape: a left hand's
flipped, the rest as drawn."""
poses = list(dict.fromkeys(p.get("cues") or [])) # each once, in the step's order
if not poses:
self._panel.set("strip", "strip off")
self._status.update(strip=[], cue=-1)
return
items = []
for pose in poses:
path, mode, _ = pose_view(self._poses, {"pose": pose, "hands": p.get("hands")})
items.append({"image": path, "mode": "" if mode == "both" else mode, "label": self._cue_name(pose)})
cue = poses.index(p["pose"]) if p.get("pose") in poses else -1
self._panel.set("strip", "strip %d %s" % (cue, ";".join(
"%s|%s|%s" % (e["image"] or "-", e["mode"] or "-", e["label"].replace("|", "/").replace(";", ","))
for e in items)))
self._status.update(strip=items, cue=cue)
def _hints(self, action=True):
"""The Next hint (with action) and the key line for what's there now: the headset button
leads when no mouse is connected. Looked at again for each step, so a mouse plugged in counts."""
@@ -1799,9 +1920,12 @@ class Session:
self._log("take %s %s" % (t["id"], status))
def _prompt_event(self, p):
extra = {"controllers": p["controllers"]} if p["controllers"] else {}
if p.get("cue"): # a sweep's cue: the pose highlighted from now, within the step
extra.update(cue=True, step=p["step"])
return self._event("prompt", id=p["id"], text=p["text"], hands=p["hands"], pose=p["pose"],
distance=p["distance"], position=p["position"], object=p["object"],
controller=bool(p["controller"]), **({"controllers": p["controllers"]} if p["controllers"] else {}))
controller=bool(p["controller"]), **extra)
def _begin_prompt(self, s, p, step, state="running"):
self._prompt = p
@@ -1822,7 +1946,7 @@ class Session:
# --- steps: a section's prompts, bar heights and targets
def _step_list(self, s):
"""[{"kind": "prompt"|"bar"|"target", "p": prompt, "ready": waits for Next in step mode, ...}]."""
"""[{"kind": "prompt"|"sweep"|"bar"|"target", "p": prompt, "ready": waits for Next in step mode, ...}]."""
out = []
defaults = s.get("defaults") or {}
if s["kind"] == "targets":
@@ -1841,7 +1965,7 @@ class Session:
"controllers": list(defaults.get("controllers") or []),
"seconds": lead + h["reps"] * float(s.get("period_s", 6))}
out.append({"kind": "bar", "p": p, "h": h, "lead": lead, "ready": True})
out += [{"kind": "prompt", "p": p, "ready": True} for p in s["prompts"]]
out += [{"kind": "sweep" if p.get("cues") else "prompt", "p": p, "ready": True} for p in s["prompts"]]
return out
def _steps(self, i, s, label):
@@ -1873,7 +1997,7 @@ class Session:
retry = False
self._redo_ok = True
self._emit(force=False, can_redo=True)
check = self._read_hands and s["id"] == HANDS_CHECK_SECTION and k == 0 and st["kind"] == "prompt"
check = self._read_hands and s["id"] == HANDS_CHECK_SECTION and k == 0 and st["kind"] in ("prompt", "sweep")
self._hold_watch = {"reads": 0, "published": 0, "seen": 0} if check else None
try:
self._run_step(s, st, where)
@@ -2041,9 +2165,43 @@ class Session:
self._target(s, st, where)
elif st["kind"] == "bar":
self._bar(s, st, where)
elif st["kind"] == "sweep":
self._sweep(s, st, where)
else:
self._run_prompt(s, st["p"], where)
def _sweep(self, s, st, where):
"""A sweep: the strip's pictures highlighted in turn, one every cue_s, while the hands
keep moving. Each cue is a prompt event of its own, with that pose, "cue": true and the
step's id, so the timeline tags each pose roughly; the time-left bar covers the step."""
p = st["p"]
cues, cue_s = p["cues"], p["cue_s"]
seen = {}
def cue_prompt(j):
pose = cues[j]
seen[pose] = seen.get(pose, 0) + 1
return dict(p, id="%s/%s%s" % (p["id"], pose, "#%d" % seen[pose] if seen[pose] > 1 else ""),
pose=pose, cue=True, step=p["id"])
self._begin_prompt(s, cue_prompt(0), where)
state = {"t": 0.0, "j": 0}
def tick(dt):
state["t"] += dt
j = min(len(cues) - 1, int(state["t"] / cue_s + 1e-9))
if j != state["j"]:
state["j"] = j
cp = cue_prompt(j)
self._prompt = cp
self._strip(cp)
self._prompt_event(cp)
self._emit()
self._log(" cue %s" % cp["id"])
return False
self._wait(p["seconds"], tick)
def _target(self, s, st, where):
hold_s, timeout_s = float(s.get("hold_s", 1.0)), float(s.get("timeout_s", 8))
p, pt = st["p"], st["pt"]
@@ -2162,6 +2320,9 @@ def main():
help="don't read the headset's button (gpio-keys KEY_SELECT: Next, pause, resume)")
ap.add_argument("--button-device", metavar="PATH",
help="test: read the button from this event device or FIFO of input_event structs (also in a dry run)")
ap.add_argument("--quick", action="store_true",
help="a quick round (about 3 min, for another lighting): hand size, the two-hand sweeps, touch, no hands")
ap.add_argument("--seed", type=int, help="the sweeps' shuffle (default: from the session's id; --plan: the script's order)")
ap.add_argument("--ignore-cameras", action="store_true",
help="start even if the camera check (hands/camcheck.py) finds the upper cameras off")
a = ap.parse_args()
@@ -2211,11 +2372,11 @@ def main():
s = Session(base, profile, checklist, a.lighting, a.script, ring=a.ring, start_processes=not a.no_start,
dry_run=a.dry_run, speed=a.speed, on_status=on_status, hands_dir=a.hands_dir, panel_bin=a.panel,
auto=a.auto, next_after=a.next_after, poses_dir=a.poses, button=not a.no_headset_button,
button_device=a.button_device, check_cameras=not a.ignore_cameras)
button_device=a.button_device, check_cameras=not a.ignore_cameras, quick=a.quick, seed=a.seed)
est = plan_seconds(s.script, s.plan, auto=a.auto)
worst = plan_seconds(s.script, s.plan, worst=True, auto=a.auto)
print("%d sections, %s mode: about %.1f min%s (at most %.1f)%s" % (
len(s.plan), "auto" if a.auto else "step", est / 60, "" if a.auto else " recorded", worst / 60,
print("%d sections%s, %s mode: about %.1f min%s (at most %.1f)%s" % (
len(s.plan), " (a quick round)" if a.quick else "", "auto" if a.auto else "step", est / 60, "" if a.auto else " recorded", worst / 60,
", %gx speed" % a.speed if a.speed != 1 else ""))
if not a.auto:
print(" %d steps wait for Next: add your reading time (at 5 s a step, %.1f min)"
@@ -2226,6 +2387,9 @@ def main():
for sec in s.plan:
print(" %-18s %-9s %3d prompts %3d steps %5.0f s" % (sec["id"], sec["kind"], len(sec["prompts"]),
section_steps(sec), section_seconds(sec, auto=a.auto)))
for p in sec["prompts"]:
if p.get("cues"):
print(" %-26s %-5s %3.0f s %s" % (p["id"], p["hands"], p["seconds"], " ".join(p["cues"])))
return 0
if not a.dry_run and not a.ring:
cam = camera_check()
+176
View File
@@ -204,6 +204,182 @@ class SessionTest(SessionBase):
self.assertIn("2 steps", session.plan_summary(SCRIPT, plan))
SWEEP_SCRIPT = {
"version": 1, "intro_s": 1, "between_s": 1,
"welcome": {"title": "Test", "seconds": 1, "text": "A test."},
"done": {"title": "Done", "seconds": 0, "text": "Done."},
"stopped": {"title": "Stopped", "seconds": 0, "text": "Stopped."},
"cue_names": {"thumbs-up": "Thumbs up"},
"sections": [
{"id": "sweeps", "title": "Poses", "kind": "sweep", "intro": "Keep moving.", "quick": True,
"cue_s": 2, "step_s": 6,
"groups": [["open", "fist"], ["ok", "thumbs-up", "claw"]],
"sweeps": [{"hands": "both", "group": "next", "text": "Both hands."},
{"hands": "left", "group": "any", "quick": False, "text": "Left hand."}]}]}
class SweepTest(SessionBase):
def setUp(self):
super().setUp()
with open(self.script, "w") as f:
json.dump(SWEEP_SCRIPT, f)
def events(self, s):
with open(os.path.join(s.session_dir, "takes", "01-sweeps", "prompts.jsonl")) as f:
return [json.loads(line) for line in f]
def test_sweep_flow(self):
s = self.session(next_after=0.0, speed=10, seed=1)
s.start()
s.join(20)
self.assertEqual(s.state, "done")
ev = self.events(s)
names = [e["event"] for e in ev]
self.assertEqual(names, ["take", "ready", "prompt", "prompt", "prompt", "wait",
"ready", "prompt", "prompt", "prompt", "wait", "end"])
plan = {p["id"]: p["cues"] for p in s.plan[0]["prompts"]}
for step, first in (("sweeps/both-1", 2), ("sweeps/left-1", 7)):
cues = [e for e in ev[first:first + 3]]
self.assertEqual([e["pose"] for e in cues], plan[step]) # one prompt per cue, in order
self.assertTrue(all(e["cue"] and e["step"] == step for e in cues))
self.assertTrue(all(e["distance"] == e["position"] == "" for e in cues))
self.assertEqual(ev[first - 1], dict(ev[first - 1], event="ready", id=step))
gaps = [(b["t"] - a["t"]) / 1e9 for a, b in zip(cues, cues[1:])]
self.assertTrue(all(0.15 < g < 0.4 for g in gaps), gaps) # cue_s 2 at 10x speed
# a cue repeated within a step (6 s of 2 s cues from a group of 2) gets its own id
both = [e["id"] for e in ev[2:5]]
self.assertEqual(len(set(both)), 3)
# the strip: every picture of the step, the cue lit
strips = [c for c in self.panel if c.startswith("panel: strip ")]
self.assertIn("panel: strip off", strips)
self.assertTrue(any(c.startswith("panel: strip 1 ") for c in strips))
with open(os.path.join(s.session_dir, "session.json")) as f:
meta = json.load(f)
self.assertEqual(meta["shuffle"]["seed"], 1)
self.assertEqual({x["id"]: x["cues"] for x in meta["shuffle"]["sweeps"]["sweeps"]}, plan)
self.assertFalse(meta["quick"])
def test_seed_from_session_id(self):
s = self.session(next_after=0.0, speed=20)
s.start()
s.join(20)
with open(os.path.join(s.session_dir, "session.json")) as f:
meta = json.load(f)
sid = os.path.basename(s.session_dir)
self.assertEqual(meta["shuffle"]["seed"], session.session_seed(sid))
again, _ = session.build_plan(SWEEP_SCRIPT, {}, seed=meta["shuffle"]["seed"]) # reproducible
self.assertEqual(session.plan_record(again), meta["shuffle"]["sweeps"])
def test_quick_session(self):
s = self.session(next_after=0.0, speed=20, quick=True)
s.start()
s.join(20)
self.assertEqual(s.state, "done")
self.assertEqual([e["event"] for e in self.events(s)].count("ready"), 1) # the left sweep left out
with open(os.path.join(s.session_dir, "session.json")) as f:
self.assertTrue(json.load(f)["quick"])
def test_auto(self):
s = self.session(auto=True, speed=20)
s.start()
s.join(20)
self.assertEqual(s.state, "done")
names = [e["event"] for e in self.events(s)]
self.assertEqual(names, ["take", "prompt"] + ["prompt"] * 6 + ["end"]) # intro, 3 cues a step
class ShuffleTest(unittest.TestCase):
def setUp(self):
self.script = session.load_script(session.SCRIPT_PATH)
def cues(self, seed, quick=False, checklist=None):
plan, _ = session.build_plan(self.script, checklist or {}, seed=seed, quick=quick)
return session.plan_record(plan)
def test_deterministic(self):
self.assertEqual(self.cues(42), self.cues(42))
self.assertNotEqual(self.cues(42)["pose-sweeps"], self.cues(43)["pose-sweeps"])
self.assertEqual(session.session_seed("20261002-202734"), session.session_seed("20261002-202734"))
self.assertNotEqual(session.session_seed("20261002-202734"), session.session_seed("20261002-202735"))
def test_groups(self):
groups = [sorted(g) for g in self.script["sections"][1]["groups"]]
for seed in range(20):
steps = self.cues(seed)["pose-sweeps"]
self.assertEqual([x["hands"] for x in steps], ["both", "both", "both", "left", "right"])
used = [sorted(set(x["cues"])) for x in steps]
self.assertEqual(sorted(used[:3]), sorted(groups)) # each group once with both hands
self.assertTrue(all(u in groups for u in used[3:]))
self.assertNotEqual(used[3], used[4]) # the one-hand sweeps differ
self.assertTrue(all(len(x["cues"]) == 5 for x in steps)) # 20 s of 4 s cues
# unshuffled: the script's order
steps = self.cues(None)["pose-sweeps"]
self.assertEqual(steps[0]["cues"], ["open", "fist", "point", "pinch", "open"])
# gestures and hand size keep their order
for seed in (1, 2, 3):
self.assertEqual([x["cues"] for x in self.cues(seed)["gestures"]],
[["pinch-tap", "pinch-drag"], ["grab", "cross", "overlap"], ["near-face", "screen-point"]])
self.assertEqual(self.cues(seed)["hand-size"][0]["cues"], ["flat", "flat-back", "spread"])
def test_plans(self):
full, skipped = session.build_plan(self.script, {}, seed=5)
self.assertEqual([s["id"] for s in full],
["hand-size", "pose-sweeps", "gestures", "desk-work", "touch", "bare-push", "no-hands"])
self.assertEqual(session.plan_steps(full), 15)
self.assertLess(session.plan_seconds(self.script, full, auto=False), 6 * 60)
everything, _ = session.build_plan(self.script, {"objects": ["pencil", "keyboard", "mouse"],
"controllers": "straps"}, seed=5)
self.assertEqual(len(everything), 10)
quick, skipped = session.build_plan(self.script, {"objects": ["pencil"], "controllers": "straps"},
seed=5, quick=True)
self.assertEqual([s["id"] for s in quick], ["hand-size", "pose-sweeps", "touch", "no-hands"])
self.assertEqual([p["hands"] for p in quick[1]["prompts"]], ["both"] * 3)
self.assertEqual(len(quick[2]["targets"]), 6)
self.assertLess(session.plan_seconds(self.script, quick, auto=False), 150)
self.assertIn({"section": "objects", "reason": "not in a quick round"}, skipped)
# desk work: the mouse-and-keyboard step only with both ticked
desk = [s for s in everything if s["id"] == "desk-work"][0]["prompts"]
self.assertEqual([p.get("cues") for p in desk], [None, ["mouse", "switch"]])
class StripPanelTest(unittest.TestCase):
"""The panel's strip command on ft-handpanel --no-vr (skipped if it isn't built)."""
def test_strip(self):
import subprocess
binary = session.PANEL_BIN
if not os.access(binary, os.X_OK):
self.skipTest("ft-handpanel isn't built")
name = "hrtest-strip-%d" % os.getpid()
proc = subprocess.Popen([binary, "--no-vr", "--socket", name], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True)
try:
panel = session.Panel(name=name)
for _ in range(50):
if panel.cmd("ping", reply=True, timeout=0.1):
break
if proc.poll() is not None:
self.skipTest("ft-handpanel doesn't run here: " + proc.stderr.read().strip()[-200:])
time.sleep(0.1)
else:
self.fail("ft-handpanel --no-vr doesn't answer")
poses = session.POSES_DIR
ok = panel.cmd("strip 1 %s/fist.png|mirror|Fist;-|-|No picture;%s/ok.png|-|OK" % (poses, poses), reply=True)
bad = panel.cmd("strip 1", reply=True)
missing = panel.cmd("strip 0 /nonexistent.png|-|Gone", reply=True)
panel.cmd("show", reply=True)
panel.cmd("strip off", reply=True)
panel.close()
finally:
proc.terminate()
out = proc.communicate(timeout=5)[0]
self.assertEqual(ok, "ok")
self.assertTrue(bad.startswith("error usage: strip"))
self.assertTrue(missing.startswith("error can't read"))
self.assertIn("strip: cue 0: Gone(/nonexistent.png)", out)
self.assertIn("strip: off", out)
# /proc/bus/input/devices as the Frame has it (2026-10-02), trimmed, plus a USB mouse.
DEVICES = """I: Bus=0019 Vendor=0001 Product=0001 Version=0100
N: Name="gpio-keys"