Commit Graph
103 Commits
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 e16b7588e2 Gaze mode is a mouse feature: drop its controller parts
Steam reads the Frame controllers itself, outside SteamVR's bindings, and every
press and release it sees takes SteamVR out of laser mode, so controller clicks
at the gaze can't be done cleanly (docs/gaze-controllers.md, from the gaze-first
branch's tests).

- Gaze precision and Gaze drag take a mouse button or a key combination only;
  the controller source, its aim steering, and POINTER_PRECISION_GAIN,
  POINTER_PRECISION_DEADZONE and POINTER_GAZE_DRAG_GAIN are gone.
- The gaze actions (including Gaze pointer on/off) can't be mapped to controller
  buttons: the relay ignores them there and doesn't ask the helper for those
  buttons, and Input Settings no longer offers them.
- Last used wins in gaze mode too: picking up a controller hands it the laser,
  as docs/design.md already said.
- The gaze dot shows all the time (POINTER_GAZE_DOT=always, the default;
  moving brings back the old behaviour), with a switch on the Gaze page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:11:24 -06:00
DeeJanuzandClaude Opus 5.5 d5c2ec65d7 Take the user, host, and home from the machine, not this Frame
The local RDP login between krdp and the VNC bridge uses the account's
own name instead of "steamos", its certificate the hostname instead of
"steam-frame", and the ft_pointer driver installs under the user's home
instead of /home/steamos. The remote-access address already came from
the tailnet (tailscale0 and tailscaled's local API).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:57:33 -06:00
DeeJanuzandClaude Opus 5.5 1f19732aaf Add Frametop Remote Access, a settings app for the VNC view
A GTK 4 / libadwaita app (remote/ft-remote-settings, on the host's own
Python like the gaze probe): remote access on and off (REMOTE, applied at
once when the desktop allows it), the tailnet name and address to connect
to, and the VNC password shown, copied, or replaced. The password stays
random and made on the Frame, in ~/.config/frametop-remote; none is in
the code.

session/remote-ctl.sh starts, stops, and reports remote access; the
session uses it and leaves a remote-capable marker, since KWin allows the
capture only in a desktop that started with REMOTE=1. The password
between krdp and the VNC bridge (local only, but on krdp's command line)
is now new at every start. The installer adds the app to the menu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:55:49 -06:00
DeeJanuz 9820e7996f Serve only the desktop's primary screen over VNC
krdp streams every screen, so the VNC screen is now the primary's size and the
FreeRDP window is shifted so the primary fills it (ft-layout remote-view gives
the offset). It resizes and reconnects when the layout changes. With remote
access on, KWin's D-Bus screenshot interface is open too, for scripts that look
at the screens without the headset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6f5a23c80f2ecd26e6a96c86b102be36d518ee25)
2026-09-30 15:50:46 -06:00
DeeJanuzandClaude Opus 5.5 97f853130d Merge eye-tracking into experimental
Our own eye tracker joins the gaze folder: ft-eyegrab (the root frame
grabber, frametop-eyegrab.service) and ft-eyes, run by the gaze service
when GAZE_TRACKER=own, with its lab tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:45:38 -06:00
DeeJanuzandClaude Opus 5.5 55d92b411b Input Settings: drop the pointer role choice
The hand role the pointer takes isn't something to choose by hand. The
helper still reads POINTER_ROLE (right by default) for experiments.

Tried and reverted (not committed): in gaze mode, taking the stylus role
by reconnecting. SteamVR gave the device no role at all (hint 5, role
none) and kept the dashboard laser on the held controllers, so the mouse
couldn't click either. The dashboard laser needs a hand role, and a held
Frame controller takes its hand's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:35:08 -06:00
DeeJanuzandClaude Opus 5.5 d666fa031f Gaze first: precision and drag buttons, key combinations, pointer role
Two new actions for mouse buttons, controller buttons, and key
combinations: gaze_precision (hold: the pointer stops where you look and
the button's device steers it, a controller by where it points at
POINTER_PRECISION_GAIN, the mouse by its moves; release: click there) and
gaze_drag (the same with a real press at once, dragging until the
release). The relay sends "precision|gazedrag <source> 1|0" to the
helper, which runs them through the same holds as pinches and grips.

- Key combinations on any keyboard ("key_bindings" in the rules, e.g.
  Ctrl+Alt+G for gaze on/off): the last key isn't typed.
- POINTER_GAZE_MOUSE: in gaze mode the left button is a precision button
  (the default, as before) or clicks right away (direct).
- In gaze mode a moving controller no longer takes the pointer away.
- POINTER_ROLE (right, left, stylus): the driver takes a "role" command,
  so the pointer can stay off the hand holding the precision controller,
  which would otherwise take the role back. Needs the rebuilt driver.
- Input Settings: the actions on the Controllers and Buttons pages; on the
  Gaze page the mouse choice, the role, the precision sliders, and key
  combinations (captured from any keyboard).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:14:57 -06:00
DeeJanuzandClaude Opus 5.5 c2739cb0df Hands: park hand tracking behind ft-handsctl
Hand tracking no longer starts with SteamVR: hands/run.sh install leaves
the units disabled and links hands/ft-handsctl into ~/.local/bin, which
turns it on and off (on | off | status | log | cutouts on|off | gestures).
With it on, hands show through the screens; pinches and grips move the
pointer only with POINTER_HANDS=1, now off by default. ft-camd's service
runs the mono cameras only: while the headset is worn, the colour module
writes just a half-size image into the top-left quarter of its buffers,
which ft-camd can't use yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:08:54 -06:00
DeeJanuzandClaude Opus 5.5 5ca4160a7e Hands: fixes from the second headset test (pinches)
- The palm-down filter is off by default: the user's deliberate pinches,
  hand raised, read 0.90-0.99, like typing.
- Typing is told apart by the keyboard instead: the input relay sends the
  pointer helper "typing" on key presses, and it takes no pinch within
  POINTER_PINCH_TYPING (1 s) of one.
- Grips are still held to hands raised (POINTER_GRIP_BELOW, 0.35 m below
  the eyes); pinches aren't, since the user's own sat 0.35-0.45 m below,
  elbow resting.
- A grip doesn't begin with the thumb on the index tip: that's a pinch
  with the other fingers curled, which was taken for a grip.
- A pinch's point is the index and middle knuckles: the tips' midpoint
  moved 1-2 cm as the pinch opened, dragging every release off its press.
- ft-hands --gesture-log prints what the detectors measure, 10 times a
  second.

Pinches still aren't reliable enough to use; hand tracking is parked for
now in favour of the controllers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:07:58 -06:00
DeeJanuzandClaude Opus 5.5 fbd188f5f9 ft-pointer: without gaze mode, a pinch is a real press
Pressed when the pinch closes, released when it opens, its hand dragging
the pointer in between (at the pinch gain, past the dead zone), like the
mouse's button. That's what the gaze probe's Click practice needs: it
freezes its own gaze dot at the press and drags it by the pointer's
movement until the release. With gaze mode on, the pinch still holds the
press back and clicks on the release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:42:59 -06:00
DeeJanuzandClaude Opus 5.5 6d1b04e032 ft-pointer: pinch to click, grip to drag
The pointer helper reads ft-hands' gestures. A pinch holds the pointer
where the gaze put it and clicks on the release; held, the hand nudges
the pointer at half its angle past a 1.5 degree dead zone, which also
teaches the gaze tracker. A grip presses where the pointer is, drags with
the hand, and releases when the hand opens, unless it began more than
30 cm below the eyes (hands on a desk). Hand movement is taken in the
room with the head pose at capture time. Hand use keeps the pointer from
the relay's idle release, as gaze mode does. POINTER_HANDS and friends in
frametop.conf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:31:17 -06:00
DeeJanuzandClaude Opus 5.5 50c14545ed Hands: pick the cameras by the light, and detect grips
ft-hands tracks with the mono IR cameras in dim light and with every
camera (or the colour pair, HANDS_BRIGHT) in bright light, going by the
colour frames' mean brightness with hysteresis and a 2 s hold
(HANDS_CAMERAS=auto, the default; mono, color and all fix it). Colour
frames are placed on the mono cameras' clock by their dequeue time, and a
view in a camera a step lacks waits for that camera's next frame.

A grip (a closed hand) is a second gesture next to the pinch, in version 2
of the gestures file: every fingertip curled toward the wrist, beginning
only on a hand seen open within a second and held up in front. On the
2026-09-30 lit recording that leaves 6 false grips of 14, all with the
hands on the desk; pinch counts are unchanged. ft-handreplay logs grips
and finger curl, watch_gestures.py shows them, and tools/cut_sets.py
copies a few sets out of a recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:31:17 -06:00
DeeJanuzandClaude Opus 5.5 40a2f39b2a ft-camd: keep colour trouble away from the mono cameras, and idle colour
A colour camera probes at most 4 buffers a frame (each probe syncs a
~9 MB buffer's cache, which made the mono cameras miss frames), and one
that goes stale twice in a row is paused (10 s, doubling to 160 s) and
learned again instead of ft-camd exiting. The colour cameras run at 2 fps
until a reader asks for more in frametop-hands/color-fps, which saves
most of their decoding while only their brightness is needed. Each mono
camera's latest near-black frame's mean goes in the ring (dark_mean), a
measure of the room's IR light. The service now starts with --with-color;
HANDS_CAMERAS=mono leaves the colour cameras out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:31:16 -06:00
DeeJanuzandClaude Opus 5.5 8bc6739dca Merge hands-migration into experimental
Hand tracking (ft-camd, ft-hands, and their tools) joins the desktop. The
hands file and ring move to /run/user/UID/frametop-hands/, which ft-screens'
hand cutouts now read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:02:01 -06:00
DeeJanuzandClaude Opus 5.5 e2abaa06b6 Hands: fixes from the first headset test
- The runtime files move to /run/user/UID/frametop-hands/: the desktop
  session deletes /run/user/UID/frametop at every start.
- The cutout copy shader runs at highp: mediump (16-bit on Adreno)
  stepped 1.7 texels across a 3440-pixel screen.
- One hand no longer pinches both sides after its left/right call flips
  mid-pinch, and --pinch-palm-down (0.6) holds back pinches with the palm
  facing down (typing on a lap keyboard).
- ft-camd judges a colour frame fresh by its luma rows only, and logs
  per-buffer changes at stale colour frames with FT_CAMD_DEBUG=1.
- hands/run.sh caps skips the setcap when ft-camd already has them.
- The replay tool dumps poses (--poses), and its pinch events carry the
  hand id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:00:54 -06:00
DeeJanuzandClaude Opus 5.5 4493b789fd Merge floating-windows into experimental
Floating windows join the keyboard and the hand cutouts. Floating panels
don't get cutouts yet: their panel and popups show crops of the client
buffer (texture bounds), which the side-by-side cutout buffer doesn't
match. KWin gets both the spare outputs and our input method, and the
pointer helper's frametop. prefix already covers the float panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:53:28 -06:00
DeeJanuzandClaude Opus 5.5 5714978e65 Add a keyboard for text fields on the desktop
Fixes #7: SteamVR's keyboard never came up for the desktop's apps, and
opening it for our panels doesn't work well on the Frame (it's Steam's own
panel, mounted in the dashboard's scene, it follows the laser between
panels, and it takes the controllers over to SteamVR's laser).

- KWin starts input/ft-textinput as the desktop's input method. It tells
  the input relay when a text field gains or loses focus, and the relay
  asks ft-screens to open or close the keyboard. The session drops the
  QT_IM_MODULE=xim and GTK_IM_MODULE=xim that the gamescope session sets,
  or Qt and GTK apps never report text fields.
- The keyboard is ft-screens' own panel (screens/keyboard.cpp): a US laptop
  layout, typed with a controller's laser or the 3D mouse. It opens 0.7 m
  in front of you, below your eyes and facing you. It has a grab bar to
  move it, a Close key, latching Shift, Ctrl and Alt, and repeat on a held
  key. It's drawn into shared DMA-BUFs, so it doesn't flicker. Its keys
  reach the focused screen as key presses, so every app takes them.
- It steps aside while the Steam menu or Steam's own keyboard is up and
  comes back after. A layout reset closes it, and it doesn't open without
  a head pose.
- Frametop Input Settings has a Keyboard page: open it for every text
  field, only while no keyboard is connected (the default), only from a
  mapped button (the new Open/close keyboard action, for mice and
  controllers), or never. A switch keeps it open until you close it.
- The pointer helper treats every frametop.* overlay as a real panel. The
  keyboard's shared texture reports 0x0 like SteamVR's scene-graph
  controls, and the helper had given it their wide catch radius.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:41:47 -06:00
DeeJanuzandClaude Opus 5.5 c8fc6351bb Floating windows: fix the scale crash, the click offset, and placement
- KWin's nested backend makes an output the size it's configured to times its
  scale, so after Meta+scroll every size ft-floatd sent was multiplied again,
  and an odd result disconnected KWin (buffer not divisible by its scale).
  ft-floatd now asks for sizes in the output's scaled terms (kwin_size), and
  asks again after each scale change.
- SteamVR reports mouse positions on a panel with texture bounds in the whole
  texture, not the crop, so clicks on a floating window landed up to ~200 px
  off. The mouse scale is now the buffer's size, as on a screen.
- A floated window starts 30 cm in front of its screen (was 5 cm), so it's
  easy to point at apart from the screen behind it.
- No 1 s wait before a spare turns on (a disabled output never commits), and
  the login splash on the spares isn't taken for floating windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:57:36 -06:00
Nikita Koptelov 4e5131d8ce Start Frametop's SteamVR clients only once SteamVR is up (#6)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
2026-09-30 10:02:46 -06:00
Nikita Koptelov 2a9fbdebb4 Start Frametop's SteamVR clients only once SteamVR is up (#6)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
2026-09-30 10:02:46 -06:00
Nikita Koptelov c3375d32d3 Start Frametop's SteamVR clients only once SteamVR is up (#6)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
2026-09-30 10:02:33 -06:00
Nikita Koptelov 3aea571496 Start Frametop's SteamVR clients only once SteamVR is up (#6)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
2026-09-30 10:02:33 -06:00
DeeJanuzandClaude Opus 5.5 ed75614f62 Bring our own eye tracker into the gaze folder, run by the gaze service
frame-eyes, a separate project until now, becomes gaze/tracker:
- ft-eyegrab (was fe-bufprobe) copies the eye-camera frames, read-only, out of
  SteamVR's eyetracking process. It runs as the system service
  frametop-eyegrab.service, which gaze/tracker/install.sh installs to
  /etc/frametop with sudo. It keeps only CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH, and
  CAP_CHOWN, and copies frames only while /dev/shm/frametop-eyes-want is fresh,
  holding none of the tracker's buffers otherwise.
- ft-eyes (was fe-trackd) runs under ft-gazed in the dev container, with
  build/venv's pinned numpy and OpenCV: while Eye tracker is Own tracker, or on
  the probe's lease ("eyes SECONDS"). No sudo password or fe-live script at
  run time any more.
- lab/ holds the research tools (ft-eyes-score, -e2e, -record, -replay,
  -session) and findings.md. Recordings live outside the repo, in
  ~/.local/share/frametop/eyes/captures; .gitignore catches stray frame dumps.

Its socket is now @ft_eyes, its output /dev/shm/frametop-eyes-gaze, and its state
~/.local/state/frametop/gaze/eyes. On practice1 -> practice2 the whole live path
(ft-eyes-e2e) gives 1.30 deg median and 3.18 for the worst tenth, as before the
move (1.30, 3.21).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:00:07 -06:00
DeeJanuzandClaude Opus 5.5 369736f0ca Read the hands file through the shared header, at its Frametop path
ft-screens' hand cutouts read /run/user/UID/frametop/hands through
hands/include/fh_hands.h instead of their own copy of its offsets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:15:07 -06:00
DeeJanuzandClaude Opus 5.5 499035216c Make hand tracking a Frametop component
- Programs: ft-camd (the camera broker), ft-hands (the tracker), and
  ft-handreplay and ft-ringplay for recordings, built by hands/build.sh
  into hands/build/ with one Makefile. The first build fetches ncnn at
  frame-hands' pinned tag and builds it with the same options.
- ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE,
  CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with
  sudo, and drops them once set up. It still works under sudo. It runs
  on the host, linked statically, as frametop-camd.service. ft-hands
  runs in the dev container as frametop-hands.service. Both start and
  stop with SteamVR.
- Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures),
  not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has
  its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands.
- The calibration is read through /run/host in the container.
- Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf.
- install.sh offers hand tracking as an optional last step.
- The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV
  for the Python tools.
- tools/ring.py reads the ring, and models/NOTICE credits the
  Apache-2.0 models.

Checked: ft-handreplay gives identical summaries and byte-identical
depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:15:07 -06:00
DeeJanuzandClaude Opus 5.5 5565a25444 Let the gaze pointer use our own eye tracker, and weight the eyes
Frametop Input Settings' Gaze page gets two settings, saved in frametop.conf and
read again by ft-gazed when the file changes: Eye tracker (GAZE_TRACKER: SteamVR's
or our own, frame-eyes' fe-trackd) and Eye bias (GAZE_EYE: auto, left, right).

ft-gazed now combines the eyes, each calibrated on its own: SteamVR's set 2 eyes
with the probe's Left eye and Right eye calibrations, or our tracker's eyes as
they come. Without per-eye calibrations, or with --source, it keeps the older
one-source path. A pointer nudge finds its look from the raw gaze the helper
echoes back; with our tracker it goes to fe-trackd as a click.

The bias leans instead of choosing (gazecal.EyeWeights): on 306 live clicks the
eyes' errors partly cancelled, both together 0.65 deg off against 0.96 and 1.11
for either alone. Left or Right counts that eye twice; auto weights each eye by
its RMS miss at its last 20 nudges, since the calibration's fit picked the wrong
eye on SteamVR's test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:11:54 -06:00
DeeJanuz 3440ec8b90 Merge branch 'main' into experimental 2026-09-30 09:11:24 -06:00
DeeJanuzandClaude Opus 5.5 3e3d31c728 Lay out hands/ like Frametop's other components
trackd/ becomes track/, and the calibration helper the Python tools
import moves from the prototype's folder into tools/. The model
development tools (nettest and the scripts that compare it with the
Python models or cut int8 calibration crops) stay in frame-hands with
the prototype they need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:01:10 -06:00
DeeJanuzandClaude Opus 5.5 1a76d1560b Bring in frame-hands' hand tracking under hands/
The history of frame-hands (~/Desktop/Projects/frame-hands on the
Frame), filtered to what moves: the camera broker (camd/), the tracker
and its offline tools (trackd/), the shared file layouts (include/), the
ncnn models, the analysis tools, and the calibration and model helpers
they import from the Python prototype. The reverse-engineering notes,
probes, camprobe, and the rest of the prototype stay in frame-hands.
Unchanged here: the renames to ft- names and Frametop paths follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:59:32 -06:00
DeeJanuzandClaude Opus 5.5 9f6ce5cc59 Plan the move of hand tracking into Frametop
frame-hands (the headset cameras' hand tracking, a separate project so
far) becomes a native component under hands/. docs/hands-migration.md
has what it is, where each part goes, names, build, how ft-camd gets
its privileges (file capabilities), interfaces, open items, and steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:59:27 -06:00
DeeJanuzandClaude Opus 5.5 2031b9ca8f Add colour cameras, pinch gestures, and depth measures
- fh-camd --with-color publishes the Arcturus colour pair (luma, half
  size, 30 fps) to the ring, flagged FH_CAM_COLOR. fh-tracker records
  them, and fh-replay --cams mono|color|all tracks with them, using the
  module's EEPROM calibration (load_color_calibration).
  tools/check_color.py checks which node is left and how the crop maps.
- Pinch detection per hand (trackd/pinch.h), published to
  $XDG_RUNTIME_DIR/frame-hands/gestures (include/fh_gestures.h). It has
  begin and end counters, times, the pinch point, and the begin point
  for drags. tools/watch_gestures.py shows it live, and fh-replay
  reports it.
- fh-replay --depth and tools/depth_report.py measure the depth without
  ground truth: noise along the line of sight against across it, the
  one-camera guess, and a simulated camera loss.
- fh-tracker --swap-sides, --ring, --keep-presence, --record-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:58:30 -06:00
DeeJanuzandClaude Opus 5.5 fc15a8a8a6 Record what's built of floating windows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:32:03 -06:00
DeeJanuzandClaude Opus 5.5 691b66cd87 Stop ft-screens without an abort
wlroots asserts that nothing still listens to its xdg-shell and decoration
globals when the display goes, so every desktop stop ended with ft-screens
aborting and leaving a core dump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:31:54 -06:00
DeeJanuzandClaude Opus 5.5 6122b7eb15 Floating windows: full screen, per-window scale, and drags across panels
Full screen fills the window's own panel (the margin drops to zero).
Meta+scroll over a floating window changes its scale at the same size in
pixels. Spare outputs are sized to a multiple of KWin's buffer scale, and
screens to even sizes: an odd buffer at a fractional scale is a protocol
error that disconnected KWin. The 3D mouse's drag lock now crosses onto
other Frametop panels unless the pressed one is being carried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:31:23 -06:00
DeeJanuzandClaude Opus 5.5 c2e5e861c8 Show floating windows as panels of their own
ft-screens makes a panel for each spare output (frametop.float.N), hidden
until ft-floatd floats a window on it. The panel shows only the window's
rectangle of the buffer at the density of the screen it came from, its
popups and dialogs get small panels over it, pressing its title bar
carries it while KWin's pointer stays put, the corner tab resizes the
window in pixels, and two more buttons close it and put it back on the
desktop. The session adds FLOAT_SLOTS spare outputs to KWin and starts
ft-floatd; ft-layout arranges only the screens' outputs, and the pointer
helper treats the new panels like screens. Not yet tried in the headset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:26:20 -06:00
DeeJanuzandClaude Opus 5.5 87a402c68d Add ft-floatd and the frametop-float KWin script
ft-floatd loads the script into the desktop's KWin, which reports windows
over D-Bus and takes commands through a long poll. Floating a window (the
window menu's Float in VR, Meta+Shift+F, or ft-float) turns on a spare
output sized to the window plus a margin, moves the window onto it, and
tells ft-screens the panel's crop, density, and place; docking puts it
back and turns the spare off. Tested on the headless test desktop; the
panel side in ft-screens comes next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:20:26 -06:00
DeeJanuzandClaude Opus 5.5 fcc8d96246 Record the headless phase 0 results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:14:22 -06:00
DeeJanuzandClaude Opus 5.5 4859412126 Put the first click after crossing onto a screen where the pointer is
KWin's nested backend ignores the position in wl_pointer.enter, and
wlroots drops a motion to the position it entered at, so KWin kept its old
pointer until the next move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:14:09 -06:00
DeeJanuzandClaude Opus 5.5 42a755b9f4 Add a headless test mode to ft-screens and a throwaway test desktop
ft-screens --no-vr runs without SteamVR and leaves the input relay alone;
--control names its control socket; "toplevels" lists KWin's windows with
their titles and sizes; "input" feeds pointer events as if from a panel.
screens/test/headless.sh starts it with a bare nested KWin next to the
running desktop, and loads KWin scripts, runs apps, and takes screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:14:09 -06:00
DeeJanuzandClaude Opus 5.5 76f3cdd2dc Ask for one look at a centre dot after the headset was off
With the Own tracker, the probe polls its status each second. After the headset was off
(or the tracker restarted), it shows one dot at the centre until you look at it and press
(S skips): that click resets both eyes' shifts, so the first real clicks aren't 10-17
degrees off. The screen-to-direction geometry the calibration used is shared with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:08:07 -06:00
DeeJanuzandClaude Opus 5.5 0a88a6b7c9 Release a button held on a screen even when the laser lets go between panels
While a button is held on a screen, the laser leaving it no longer takes
KWin's pointer, and an invisible catcher overlay sits on the laser whenever
it's off every panel, so the release reaches KWin at the pointer's last
spot. The pointer helper also reports the mouse's left release as a
backstop. Not yet tested in the headset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:05:54 -06:00
DeeJanuz f17a5a66ab Merge remote-tracking branch 'frame/layouts-headpin' into floating-windows
# Conflicts:
#	README.md
#	display-settings/ft_display_settings.py
#	display-settings/main.qml
#	docs/reference.md
2026-09-29 23:02:12 -06:00
DeeJanuz d5dbe23cae Merge remote-tracking branch 'frame/pointer-ignore' into floating-windows 2026-09-29 23:01:46 -06:00
DeeJanuzandClaude Opus 5.5 389878b9fd Settle the floating-windows design with the user
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:01:46 -06:00
DeeJanuzandClaude Opus 5.5 0af087c776 Plan floating windows: any desktop app in a VR panel of its own
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:00:33 -06:00
DeeJanuzandClaude Opus 5.5 3022d7de9d Run the tracker on CPUs 5-7 by default
probes/core_ab.py with the headset on (3 rounds, the same replayed frames in
every block): on 5-7 a step took 8.4 ms against 13.2 ms on 2-4, where XRService's
head tracking also runs, and latency fell from 14.1 to 9.6 ms. The compositor's
late frames and CPU/GPU time per frame didn't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:59:19 -06:00
DeeJanuzandClaude Opus 5.5 c37bee27c9 Keep the Own tracker's calibration in reach, and let it learn after a re-seat
The calibration dots for the Own tracker go out to the Calibration ring angle each way on
an oval, not to the window's corners, which were too far to look at while facing the
centre. The Own tracker learns from drags up to 25 degrees (after taking the headset off
and on, its first clicks were 11-17 degrees off, and the 6-degree limit blocked them). The
probe starts on the Own tracker when it's running, and the calibration header names the
tracker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:51:37 -06:00
DeeJanuzandClaude Opus 5.5 355f30f335 Add a live replay harness for the CPU-placement test
fh-ringplay plays a recording into a frame ring in real time, and fh-tracker
--ring reads it (cameras without a device node are mapped by name). With the
same frames in every run, probes/core_ab.py compares the tracker on CPUs 2-4,
on 5-7, and not running, measuring the tracker's step time and latency, the
compositor's late frames and CPU/GPU time, XRService's timing warnings, CPU
temperature and clocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:42:24 -06:00
DeeJanuzandClaude Opus 5.5 067a03ce38 Hand tracking for Frametop's hand cutouts
fh-camd (camd/) borrows XRService's camera buffers and publishes the tracking
cameras' frames to a shared ring. fh-tracker (trackd/) finds hands in them with
MediaPipe's palm and landmark models on ncnn, triangulates them in 3D, and
publishes them for ft-screens. fh-replay replays recordings offline. tracker/ is
the earlier Python version; tools/ and probes/ hold the checks and experiments.

As of this commit: crop contrast defaults to CLAHE for the palm search and plain
crops for the landmarks, --swap-sides works around fh-camd naming the side
cameras backwards after some XRService restarts (tools/check_sides.py detects
it), and --record-only, --with-dark, --cpus and --keep-presence support the
bright-light and CPU-placement tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:36:48 -06:00
DeeJanuzandClaude Opus 5.5 b76a8c50e0 Spread the Own tracker's calibration dots over the practice area
Its fit goes wrong past its dots, and the ring (limited by the window's height) never
reached the sides, where frame-eyes' worst practice clicks were. With the Own tracker
the calibration now uses the centre, corners, and side middles of the practice area.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:04:06 -06:00