mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 05:00:08 +02:00
Frametop: a multi-screen desktop and universal 3D mouse for the Steam Frame
Several KDE Plasma screens floating in SteamVR, each a real monitor of any resolution and shape, shown by our own compositor (ft-screens), with a layout, wrist pinning, and visibility modes; a Bluetooth mouse that drives all of SteamVR as a room-anchored 3D pointer (input relay, ft-pointer helper, ft_pointer SteamVR driver); two settings apps; and Bluetooth LE fixes. Installs on the headset with ./install.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
commit
d439bc3f25
61 files changed
+8785
No files matched your search
+201
@@ -0,0 +1,201 @@
|
||||
# Frametop design
|
||||
|
||||
Last updated: 2026-09-25.
|
||||
|
||||
## Goal
|
||||
|
||||
- Two or more desktop screens shown as panels in SteamVR, placed anywhere around the user (arc, stacked, tilted, behind).
|
||||
- One physical mouse drives a cursor through that 3D layout. Moving the mouse moves the cursor across panels in the direction you expect from where they are in space.
|
||||
- Works with a Bluetooth mouse. The keyboard should follow the focused panel.
|
||||
- Desktops come from Linux on the Frame itself. PC streaming is out of scope for now.
|
||||
|
||||
## Approach: nested Plasma in ft-screens (2026-09-26; before: a PerWindow gamescope)
|
||||
|
||||
Since 2026-09-26 the nested KWin runs inside ft-screens (`screens/`), our own wlroots compositor, instead of gamescope: gamescope draws every window into one canvas of at most 1920x1080 pixels and leaves the panels to the dashboard, which caps their size. ft-screens sizes each KWin screen on its own and shows it as its own SteamVR overlay that we place and size (grab bar, resize handle, pin to a hand, hide/show). The findings log (2026-09-26) has the details. The gamescope notes below still describe `BACKEND=gamescope`.
|
||||
|
||||
Everything it uses ships with SteamOS.
|
||||
|
||||
- `session/frametop-session.sh` starts its own `gamescope --backend openvr --virtual-connector-strategy PerWindow`. Inside it runs a full Plasma session whose KWin has `--output-count N`. The nested KWin opens one window per output, and PerWindow makes each window its own SteamVR overlay.
|
||||
- It's modeled on `/usr/bin/steamos-nested-desktop`, SteamOS's single-screen desktop in VR, and runs alongside it. It has its own `XDG_RUNTIME_DIR` (`/run/user/1000/frametop`), config (`~/.config/frametop`), and state (`~/.local/state/frametop`).
|
||||
- It's controlled from the PC with `desktops.sh`.
|
||||
|
||||
Verified in the headset on 2026-09-25:
|
||||
- Two 1920x1080 panels, each with wallpaper, sharp.
|
||||
- A taskbar, and apps can be launched.
|
||||
- Panels can be moved in VR.
|
||||
|
||||
Still open:
|
||||
- Do windows drag between screens? This needs a mouse or keyboard connected.
|
||||
- What overlay keys does gamescope give each window? Read gamescope's OpenVR backend source (ValveSoftware/gamescope).
|
||||
- Does the `Gamescope WSI Layer Error: Creating swapchain for non-Gamescope swapchain` in the session log affect any apps? Apps inside the nested desktop inherit `ENABLE_GAMESCOPE_WSI=1`.
|
||||
|
||||
## Launcher, config, and panel placement
|
||||
|
||||
- Launcher: `desktops.sh install` writes `~/.local/share/applications/deckard-nested-desktop.desktop`, which overrides the stock `/usr/share/applications/deckard-nested-desktop.desktop` (`Exec=steamos-nested-desktop`) by filename. It keeps `X-Steam-Special=Desktop`, which the Frame's non-Steam app launcher uses to pick out the Desktop entry. `uninstall` removes it.
|
||||
- Config: `~/.config/frametop.conf` (`SCREENS`, `WIDTH`, `HEIGHT`, `PHYS_WIDTH`).
|
||||
- Placement (researched 2026-09-25): gamescope's OpenVR backend (upstream `src/Backends/OpenVRBackend.cpp`) creates every panel with `CreateDashboardOverlay`, keyed `gamescope.<wl_display>.window.<n>` for non-Steam windows, and never sets a transform. The SteamVR dashboard (web UI, `resources/webinterface/dashboard`) owns placement. Each panel is a "frame" with a dock location: `Dashboard`, `Theater`, `World` (the "Float In World" menu action), `LeftHand`, `RightHand`, or `Boot`. No `/settings/dashboard/*` key sets the initial dock location, and dock state and transforms aren't persisted, only `lastAccessedExternalOverlayKey`.
|
||||
- Default floating and a layout (decided 2026-09-26: automating the dashboard won over a patched gamescope, which would have lost SteamVR's window controls): `layout/ft-layout` floats each screen with `vrcmd --dock-overlay` and the pointer helper carries it into place with the invisible controller. See the 2026-09-26 findings and `README.md`. With `--virtual-connector-strategy PerWindow` and `--vr-overlay-key frametop`, the keys are `frametop.app.<window seq>`; `.app.0` is gamescope's default connector and never has a window.
|
||||
- Resolution per screen isn't possible with stock gamescope: it never sets its windows' sizes (no `xdg_toplevel` configure sizes), and draws each window into the one `-W`×`-H` composition, letterboxed. KWin 6.2 would resize a nested output on a configure, and `kscreen-doctor` offers only the one mode. Per-screen scale (KWin output scale, kept in the session's `kwinoutputconfig.json`) stands in.
|
||||
|
||||
## The universal 3D mouse
|
||||
|
||||
Goal (decided 2026-09-25): run all of SteamVR from inside the headset with a Bluetooth mouse and keyboard. That covers the dashboard, Steam, overlays, the Frametop desktop, and flatscreen apps. It works like the Apple Vision Pro's mouse: a small cursor floats in the room and "collides" with any panel, then acts like a controller laser on it.
|
||||
|
||||
Decisions:
|
||||
- Universal, not only Frametop panels.
|
||||
- The cursor is anchored in the room (world space), with recenter on demand.
|
||||
- Controllers stay fully usable. Last used wins.
|
||||
- Hands off VR games. A scene app with the dashboard closed gets nothing from the pointer. Flatscreen mouse games in Theater get the raw mouse passed through.
|
||||
|
||||
### Cursor model
|
||||
|
||||
1. The relay grabs the mouse (done: `input/input-relay.py`).
|
||||
2. Relative motion becomes yaw and pitch of a direction anchored in the room. Recenter puts it straight ahead of the current head pose. Sensitivity is in degrees per count, with optional acceleration.
|
||||
3. The pointer ray starts at the head (or a point just below the eyes) and runs along that direction. Anything it hits is the target.
|
||||
4. The cursor sits on the hit surface when there is one. Otherwise it sits on a sphere at a set distance (1.5 m default).
|
||||
5. Settings: distance, cursor size, snap margin (with hysteresis so it doesn't flicker at panel edges), sensitivity and acceleration, recenter key.
|
||||
|
||||
### Delivery: a virtual SteamVR controller
|
||||
|
||||
SteamVR's dashboard, and every overlay it hosts, is driven by the vrcompositor "lasermouse" action set:
|
||||
|
||||
- `Pointer`: the `/pose/tip` pose.
|
||||
- `leftclick`, `rightclick`, `middleclick`, `back`, and `home`.
|
||||
- `scroll_discrete` and `scroll_smooth`: `scroll`.
|
||||
- `system`: `ToggleDashboard`.
|
||||
- `quickrecenter`: `Recenter`.
|
||||
|
||||
The Frame controller's defaults are in `/opt/steamvr/drivers/frame_controller/resources/input/vrcompositor_bindings_frame_controller.json`.
|
||||
|
||||
A small OpenVR driver (`ft_pointer`) adds a virtual controller with no render model or laser of its own:
|
||||
- Pose: the pointer ray (origin at the head, aimed at the cursor). A laser that starts at the eye and runs along your line of sight shows up as a dot, so SteamVR's own hit cursor becomes the floating mouse on panels.
|
||||
- Inputs: mouse left to `trigger` (leftclick), right to rightclick, middle to middleclick, wheel to scroll, side buttons to back. A keyboard shortcut maps to system (ToggleDashboard) and to recenter.
|
||||
- Its own `controller_type` (`ft_pointer`) with default bindings for `openvr.component.vrcompositor` and `steam.client`.
|
||||
|
||||
Open questions (spike 1):
|
||||
- Role. The right hand collides with the real controller. The stylus role (`TrackedControllerRole_Stylus`, path `/user/stylus`) might be bindable without taking a hand.
|
||||
- Does the Frame's dashboard follow a third pointer device? Is last used wins automatic (`lasermouse_secondary/switchlaserhand` exists)?
|
||||
- Hiding the device from VR games: report the pose as invalid, or deactivate, whenever a scene app has focus and the dashboard is closed.
|
||||
|
||||
### Processes
|
||||
|
||||
- `input-relay.py` (host, user service): in pointer mode it sends mouse deltas and buttons to the helper over a Unix socket. In passthrough mode (pointer off, or a flatscreen game in Theater) it forwards to the virtual uinput devices as now.
|
||||
- `ft_pointer` driver (inside vrserver, host): the virtual controller. It takes its pose and buttons from the helper over a Unix socket. It's built for the host ABI (glibc 2.39; the `dev` container has 2.43) in a Fedora 40 build container.
|
||||
- `ft-pointer` helper (OpenVR client, in the container): cursor state, recenter, mode switching (IVROverlay `IsDashboardVisible` and the scene app's focus), the free-space cursor overlay, and settings.
|
||||
|
||||
Keyboard: still open. SteamVR opens keyboards (vrserver held the Z3 Keyboard), but routing physical keys into dashboard text fields and gamescope panels needs its own spike.
|
||||
|
||||
### Spikes
|
||||
|
||||
1. Driver: a minimal `ft_pointer` virtual controller with a fixed pose in front of the HMD and a scripted trigger. Does the dashboard laser follow it, and do clicks work? Try the right-hand and stylus roles.
|
||||
2. Mouse-driven pose: the relay feeds the helper, the helper feeds the driver. Room-anchored cursor, recenter.
|
||||
3. Free-space cursor overlay, and hiding it when SteamVR's hit dot is on a panel.
|
||||
4. Modes: VR game focus (off), Theater flatscreen (passthrough), dashboard and overlays (pointer).
|
||||
5. Keyboard routing.
|
||||
|
||||
## Phases
|
||||
|
||||
1. MVP: two panels. The mouse cursor crosses between them in 3D, with click and scroll working.
|
||||
2. Layout: N panels, save and restore panel placement, recenter.
|
||||
3. Daily use: keyboard focus follows the cursor, launch apps onto a chosen panel, cursor visuals, sensitivity settings, autostart.
|
||||
|
||||
## Rejected approaches
|
||||
|
||||
- WayVR (wayvr-org/wayvr), tried 2026-09-25 and removed. It built for aarch64 in the `dev` container and connected to SteamVR, but we couldn't see or open it in the headset. It has no bindings for `frame_controller`, so it relies on SteamVR's Oculus remap, and its KDE screen capture needs `xdg-desktop-portal-kde`, which the Frame lacks. A GitHub fork, `DeeJanuz/wayvr`, was created for it.
|
||||
- A custom capture app (headless KWin, then `zkde_screencast`, then DMA-BUF, then OpenVR overlays). It's workable, but gamescope PerWindow does the same job with no code.
|
||||
|
||||
## Findings log
|
||||
|
||||
- 2026-09-26: Programs that stop their own helpers when their window closes still lose them on a desktop restart, even after keep-apps.sh moved them out of the unit (22 processes moved, all gone 7 s later when the compositor went): nothing outside the app can keep them. Run such work somewhere independent of the desktop, like a systemd user service. Controls: shown only while some controller ray passes within max(1.5 x button, 12% of the bar) of a control (5 points along the bar, each button, the tab), 0.4 s linger; a zone covering the lower quarter of the screen showed them whenever a screen was in use.
|
||||
|
||||
- 2026-09-26: Restarting the desktop killed everything started in it, including background servers and the jobs they ran: `systemctl --user stop frametop-desktop` kills the unit's whole cgroup, and apps launched in the nested session never get scopes of their own (KIO uses systemd scopes only when systemd is on the session bus, and the session runs on a private dbus-run-session bus). Now `desktops.sh stop` first runs session/keep-apps.sh, which moves every process in the unit except the session's own (session script, dbus, KWin, Plasma, the session services it starts; matched by comm name) into a new transient scope (StartTransientUnit with PIDs, via busctl). GUI apps still exit when the compositor goes; whether an app's children outlive it is up to the app. Controls now fade in only while some controller-class device's ray (the 3D mouse's virtual controller included) meets the screen's plane in its lower quarter or just below it, and linger ~0.8 s.
|
||||
|
||||
- 2026-09-26: Roll button and quieter controls (feedback: tilting screens sideways was hard; the new corner tab was too big; controls should be smaller and translucent like SteamVR's). Roll is a knob: at the press the laser's angle around the screen's centre (in the plane of the pose at the press) is recorded, and the screen turns by how far that angle has moved (RollZ about its own front axis; pinned screens roll their controller->screen transform). Within 2.5° of level (the right edge's slope) it snaps level; scrolling on the button steps 5°. Controls: bar sqrt(0.012 x distance x width), buttons and tab max(13% of the bar, 1.8% of the distance); textures are a translucent light pill and dark translucent discs with white glyphs (3x supersampled), and every control sits at 55% overlay alpha until a laser is on it (VREvent_MouseMove / FocusEnter / FocusLeave on the control) or it's being dragged.
|
||||
|
||||
- 2026-09-26: Screen controls and pointer depth (feedback: the wrist ring was too big; the bar and corner handle only followed distance, looked detached from flat screens; the pointer was unreliable between things close together in view at different depths). Wrist ring 6 cm (leave at 9 cm). Controls: bar width = sqrt(0.02 x distance x width), at least 4% of the distance, at most 60% of the width; the resize control is a quarter-disc tab whose corner sits on the screen's corner (the old L floated 5 cm off it); on curved screens the bar gets the screen's radius and the button and tab are placed on the cylinder (angle u/r, facing the axis). They're re-sized every half second when the distance changes by more than 8%. OpenVR's intersection mask (SetOverlayIntersectionMask) doesn't affect ComputeOverlayIntersection, so it can't be checked from code; the tab stays entirely outside the screen instead. ft-pointer: the cursor's ray starts at the recenter anchor, not the eye, so after leaning it could land on a panel that a nearer one covers from the eye; a second test along the eye's line of sight to the cursor point now takes the nearer thing.
|
||||
|
||||
- 2026-09-26: Wrist pinning, third version (in testing, the target wasn't visible, and a screen that pinned mid-carry was stuck at the angle the carrying hand had while pointing at the wrist). Now the laser entering a controller's 10 cm ring (leaving it past 14 cm, so it doesn't flicker) toggles an armed state, and the pin happens on release with the pose at that moment. Grabbing a pinned screen starts armed for its wrist, so moving it re-pins it. While a screen is carried, every other hand controller gets a ring overlay (its zone, facing the head, blue when armed) and a dot at the laser's closest point to it (within 35 cm, blue inside the ring); both are non-interactive overlays above the screens (sort order 20/21). Frametop Display Settings' pages moved from a collapsed side drawer, which was easy to miss, to tabs.
|
||||
|
||||
- 2026-09-26: Wrist pinning, second version (feedback: pinning should work by aiming, not touching, so big and far screens can ride on a wrist, and a pinned screen should only show from the front). While a screen is carried, the segment from the carrying device to its bar is tested against the other hand controllers (not ft_pointer); within 10 cm for 0.3 s, the screen pins as it is (controller->screen transform kept). A pinned screen's alpha follows the angle between its front and the direction to the head: 1 inside the wrist angle minus 10°, 0 beyond it (then hidden). Visibility modes (always / dashboard / gesture: gaze within N° of a controller / toggle) in ft-screens, with g_manual as "hidden" in always and "shown anyway" in the others. Poses are read once per tick. `get` reports the pin (hand and transform) so `ft-layout capture` keeps pins and `apply` restores them.
|
||||
|
||||
- 2026-09-26: ft-screens fixes after the first session in the headset (moving windows between screens worked; text is as sharp as the headset allows, so screens need to be bigger and nearer).
|
||||
- The curved preset spaced screens by angle (2 atan(w/2d)), which assumes every screen sits on the circle. A flat 3.6 m screen's edges are 2.7 m away when its centre is 2 m, so its neighbours landed in front of its edges. Now each row is chained edge to edge like monitors on a desk: the middle screen (or seam) straight ahead at the distance, each neighbour hinged at the previous one's outer edge plus the gap and turned until it faces the eye (dot(centre, right) = 0, solved by scan and bisection; a fixed-point iteration diverges for screens nearly as wide as twice their distance).
|
||||
- Resize took the larger of the ray's reach in x and y (y scaled by the aspect). The handle sits below the bottom edge, so moving inward without moving up never shrank a screen. Now the corner follows the ray along the diagonal, keeping the grab offset; minimum 15 cm.
|
||||
- Push/pull scaled the device-to-screen offset. The 3D mouse's device sits just in front of the bar (the laser origin is near the cursor), below the screen's centre, so that offset points mostly up. Now it moves along the head-to-screen line.
|
||||
- Pinning checked the screen's centre within 35 cm of a controller, but a big screen's centre is far from the edge you bring to your wrist. Now it's the nearest point of the screen's rectangle within 20 cm; the pinned screen shrinks to 32 cm, 12 cm from the controller, and gets its width back when grabbed.
|
||||
- Curvature: `SetOverlayCurvature` (the fraction of a full cylinder the width covers, width / 2πr). The curve button uses the head's distance as the radius, so the screen wraps around you; the width can change and the radius stays. ComputeOverlayIntersection and the mouse coordinates follow the curve.
|
||||
- A button release is delivered to the panel under the laser, maybe another screen's, so any release on any of our panels ends that device's drags.
|
||||
|
||||
- 2026-09-26: ft-screens, a gamescope replacement (decided: our own panels, always visible with a hide hotkey, switch as soon as it works; acceptance test: an ultrawide between two portrait screens). Spike works: three native panels, 1080×1920 / 3440×1440 / 1080×1920.
|
||||
- OpenVR's public `IVRIPCResourceManagerClient_003` (`VRIPCResourceManager()`: `GetDmabufModifiers`, `ImportDmabuf`, `UnrefResource`) is how gamescope hands SteamVR its frames; a texture of type `TextureType_SharedTextureHandle` then shows it. No size limit. The Frame's runtime supports it (and `IVROverlay_028`), but SteamVR's bundled `hellovr` header predates it, so builds fetch Valve's public header (v2.15.6). SteamVR imports XRGB8888/ARGB8888 with `LINEAR` and `0x0500000000000001` (Qualcomm compressed).
|
||||
- First try, KWin screencast virtual outputs (`zkde_screencast_unstable_v1.stream_virtual_output`, v3 in KWin 6.2.5): KWin crashed in `WorkspaceSceneOpenGL::textureForOutput` (std::out_of_range) streaming one while nested in gamescope, three times (the wrapper restarted it, plasmashell didn't come back until restarted by hand). In 6.2.5 only the DRM and nested Wayland backends implement `createVirtualOutput`, not `--virtual`. The nested backend's version is just another host window: `createOutput(name, size * scale, scale)`.
|
||||
- So ft-screens is a minimal wlroots 0.20 compositor (`screens/compositor.c`, C) that hosts the nested KWin, plus an OpenVR side (`screens/vr.cpp`). KWin's nested backend needs `wl_compositor` v4+, `wl_shm`, `wl_seat`, `xdg_wm_base`, and `zwp_linux_dmabuf_v1` v4 (feedback: main device, format table); pointer constraints/gestures, relative pointer, and xdg-decoration are optional. We configure each KWin window's size on its first commit (`wlr_xdg_toplevel_set_size`) and KWin sizes that screen to match; each committed DMA-BUF goes straight to SteamVR, held until the next one. Frame callbacks at 90 Hz. Idle cost: ft-screens 1% CPU, KWin 1%.
|
||||
- wlroots details: `wlr_shm_create` wants DRM format codes; KWin asks for server-side decorations before its first commit, and setting the mode then asserts (`surface->initialized`), so it's answered on the first commit.
|
||||
|
||||
- 2026-09-26: Three screens (3440×1440 centre, portrait sides) meet two gamescope limits.
|
||||
- One size and shape for every screen: a 540×960 test window opened straight in Frametop's gamescope (`WAYLAND_DISPLAY=/run/user/1000/gamescope-1`) became a landscape 16:9 panel (0.59 × 0.33 m): gamescope composes each window into the shared `-W`×`-H` canvas, letterboxed. Portrait screens are rotated KWin outputs (`kscreen-doctor output.N.rotation.left`) with their panels rolled 90° in VR (`place ... roll`); a rolled placement lands exactly (it took two moves).
|
||||
- At most 1920×1080 worth of pixels: `upload_buffer_size = 1920 * 1080 * 4` (rendervulkan.hpp), and the OpenVR backend uploads a flat texture of the whole output at start (`vulkan_create_flat_texture(g_nOutputWidth, g_nOutputHeight, ...)`). 3440×1440 aborted gamescope at start (`uploadBufferData: Assertion 'size <= upload_buffer_size' failed`). The largest 21:9-ish size is about 2224×928. Beyond that needs a patched gamescope.
|
||||
- A new window starts docked in the dashboard, where `--dock-overlay dashboard` is ignored as redundant and doesn't open the dashboard, so `world` then fails. `float_screen` docks to theater first.
|
||||
- For scale: a 1.18 m panel at 1.2 m spans about 52°, and the Frame resolves roughly 20-25 pixels per degree, so about 1,200 pixels across; more pixels only help on bigger or nearer panels.
|
||||
|
||||
- 2026-09-26: Screens float and arrange themselves (tested with one screen; two screens pending).
|
||||
- `vrcmd --dock-overlay <dashboard|world|theater|lefthand|righthand> <key>` sends the dashboard `vrcmd_dock_overlay`. An unknown key moves the dashboard's active frame instead (`GetFramesWithAssociatedSummonKeys(key)[0] ?? activeFrame`), and a key already at that location is ignored ("redundant"). `world` takes its first transform from the dashboard's position (`setInitialTransformForLocation` → `requestSGTransform(GetDockLocationTransformID(Dashboard))`), which fails while the dashboard is closed ("Invalid transform ID"): the panel floats with no position and shows only with the dashboard. Working order: dock `dashboard` (opens the dashboard), dock `world`, `--hidedashboard`.
|
||||
- Floating panels' transforms can't be read (DashboardTab, type 5), but `ComputeOverlayIntersection` works on them and returns UVs (v bottom to top). Casting rays from the head over the sphere and fitting point = O + u·U + v·V gives centre, size, and frame exactly; 230,000 rays take 0.1 s (`md::ScanPanel` in `pointer/common/vrmath.h`, `vrprobe --scan`). A floating 16:9 panel measured 1.181 × 0.664 m with `PHYS_WIDTH=1.6`.
|
||||
- The grab bar: below a floating panel SteamVR's laser hits three bands, 2-4, 6-9, and 14-26 cm below the bottom edge; 7.5 cm is the grab bar (`UndockedOverlay`: `onMouseDown: startFloatingWindowMove` on a 350 px bar at `{y: -0.26}` from the frame controls). `valve.steam.gamepadui.floatingfooter` never became visible there.
|
||||
- The move (`startFloatingWindowMove`): the panel is parented to the device that clicked (head, left, or right hand by the mouse event's input path) with the relative transform at the press, and at the release it's `device × relative × pushTransform` (push = scroll, along the panel normal, whole notches of ~7 cm; fractional scroll does nothing). Drops within 0.3 m of the open dashboard or 0.4 m of the other hand snap there, so the dashboard is closed first. Rotations about the device origin carry the panel exactly (±20° yaw, ±8° pitch: 0.0° error). Slow translations do too (0.1 m in 1.5 s: 9.9 cm, no rotation). The first try, 0.1 m in 12 jerky 25 ms steps right after the press, moved it 0.19 m and turned it 8.5°. With a hover before the press, rotation first, then a smooth 60 Hz slide, placement was exact (0.0 cm, 0.0°) at every slide speed tried, 0.07 to 1 m/s, including a 66° swing and a panel facing away from the eye. `place` takes 2.5-3.5 s for a floating screen and about 8 s from docked.
|
||||
- The helper answers `place`/`measure`/`head` by datagram to the sender's abstract address (`recvfrom`). The first version reset the sender length before replying, so replies went nowhere.
|
||||
|
||||
- 2026-09-26: The pointer with the dashboard closed (user-tested: "exactly how I want the pointer to work").
|
||||
- gamescope's app panels (`frametop.app.N`, the desktops) report a 0x0 texture like scene-graph overlays, but their transform type is DashboardTab (5), so `GetOverlayTransformAbsolute` fails and the plane test skipped them. `ComputeOverlayIntersection` hits them normally. Only absolutely placed 0x0 overlays are scene-graph now. They stay visible when the dashboard closes.
|
||||
- Off a panel, the cursor jumped to `POINTER_DISTANCE` (1.5 m), behind the panel (~1 m), and the laser started behind the panel's resize margins and window controls. It now stays on the last panel's plane within `POINTER_EDGE_REACH` (0.3 m). The floating-window controls only show while the panel is hovered, so the overlay list includes hidden overlays, and visibility is re-read every 50 ms.
|
||||
- With the dashboard closed, SteamVR's laser mouse is off until a click, even while our device is the primary dashboard device (`GetPrimaryDashboardDevice` stayed ours; `system.pointer` stayed hidden until a trigger press). So the first click on a panel only turned the laser on, and leaving every panel turned it off again. A held Frame controller keeps it on by itself. Fix: `VROverlayFlags_MakeOverlaysInteractiveIfVisible` ("the system-wide laser mouse mode will be activated whenever this overlay is visible") on a transparent 1 mm overlay, `frametop.pointer.lasermode`, 50 m below the head, shown only while the pointer is awake. vrcompositor's strings (`overlaysForcingLaserMouseOn`, `force_activate_laser_mouse`) led to it. Re-sending the claim pulse when the primary device went invalid didn't help and was removed.
|
||||
|
||||
- 2026-09-25: The tilt was also lost on the left release. SteamVR's dashboard finishes a floating move up to 150 ms after mouseup (`endFloatingWindowMove` races `updatePushDistance` against `s_flFinalPushMeasurementMS` = 150) and re-reads the controller pose, which had already gone back to plain pointing. The helper now holds the drag pose (tilt and frozen distance) for 0.5 s after the left release.
|
||||
|
||||
- 2026-09-25: Tilt works (user-tested), but it reverted when the right button was released: the device pose went back to plain pointing and the still-grabbed panel followed. Now the tilt angles accumulate per drag and stay applied (about the current cursor point) until left is released. They reset on each left press and release.
|
||||
|
||||
- 2026-09-25: Small controls explained by the helper debug log (`debug` command). Over the undock-type controls the helper saw FREE space, put the catcher at 1.5 m and the laser origin at 1.39 m, while the controls were about 1.1-1.2 m away. `valve.steam.gamepadui.bar` (dock) and `valve.steam.gamepadui.floatingfooter` (floating-window controls) are visible absolute overlays with texture 0x0 and a placeholder width of 1.0 m. The dashboard draws them through its scene graph, so `ComputeOverlayIntersection` never hits them. The helper now plane-tests texture-less overlays within `POINTER_SCENE_RADIUS` (0.5 m) and puts the catcher 5 cm behind that plane. The drag lock (freeze the cursor distance while left is held) fixed resize snap-back; the user confirmed it.
|
||||
|
||||
- 2026-09-25: User feedback round.
|
||||
- The Buttons page hid bindings whenever the Z3 slept, because it listed only connected devices. The app now also lists devices with saved rules or bindings.
|
||||
- "Toggle dashboard" did nothing: the relay sent the system button's press and release together. It now wakes the pointer and holds the button for 0.12 s.
|
||||
- Small floating controls (undock, frame buttons) sit a few cm in front of their panel. With the laser origin at 0.95-0.98 of the way, the laser started behind them. `POINTER_ORIGIN_MARGIN` (0.15 m) keeps the origin in front.
|
||||
- Tilt: the driver takes `posq` (full quaternion), and the helper rotates the device pose around the grab point while left and right are held. SteamVR's floating move (`UndockedOverlay.startFloatingWindowMove`) keeps the panel rigid with the controller (`m_sMoveDevicePath`), so this should turn the panel. Not yet tested; it needs a SteamVR restart to load the driver.
|
||||
|
||||
- 2026-09-25: Input relay v2 and the settings app. The relay has per-device roles (pointer, passthrough, ignore) keyed by Bluetooth address (EVIOCGUNIQ) or USB ids. It no longer grabs keyboards by default: they used to be swallowed into the virtual keyboard, which nothing types from. It has per-device button maps to named actions and a control socket `@frametop_relay` (devices, watch, reload; reload also reaches the helper, which now re-reads `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, and `POINTER_ORIGIN_FRACTION` live). `input-settings` (Kirigami and PySide6, in the container) was tested headless against the live relay with fake uinput devices: listing, roles, capture, mapping, role change, settings, and Bluetooth all work. The UI was checked through screenshots of the VNC display. Clicking through VNC, then RDP, then KWin fake input is too lossy for scripted UI tests.
|
||||
|
||||
- 2026-09-25: Controller handoff works for all three devices (left, right, mouse). The driver switches its role hint (Right while connected, OptOut while not), because SteamVR keeps a hand role reserved for a disconnected device that still hints it. The helper releases the pointer when a real controller moves, or when ours hasn't got the hand role within 1 s: SteamVR gives a contested role to the most recently used device, and a held Frame controller counts as used through its touch sensors.
|
||||
- 2026-09-25: The beam width can't be switched live. `dashboard.laserRayWidthScale` set by any client (IVRSettings, `vrcmd`, even followed by a `laserLength` nudge or `VREvent_DashboardSectionSettingChanged`) is saved but not applied. Only the dashboard's own Settings screen, or a SteamVR restart, applies it. So the width stays at whatever it was set to. The helper starts the laser at 0.95 of the eye-to-cursor line (a few cm of beam along the line of sight, and SteamVR's hit dot is tiny), and draws its own white dot everywhere: `frametop.pointer.marker` (not interactive) on panels, `frametop.pointer.cursor` (interactive, catches the laser) in free space.
|
||||
|
||||
- 2026-09-25, spike 3 (helper) works. It looks right: only a dot, anchored to panel surfaces, with a floating white dot in free space.
|
||||
- Bug fixed: driver poses are in raw tracking space, and client math is in the standing universe (on the Frame, standing is ~1.6 m above raw). Sending standing coordinates put the laser origin 1.6 m above the head, which also inflated SteamVR's hit dot. The helper now converts via the HMD pose in both universes each frame (`TrackingUniverseRawAndUncalibrated`).
|
||||
- SteamVR's hit dot (`system.pointer`, transform type 4, not readable as absolute) is sized by distance from the laser origin. The origin sits `POINTER_ORIGIN_FRACTION` (0.5) along the eye-to-cursor line, which stays invisible and halves the dot.
|
||||
- Controller laser not returning: the dashboard pointer flipped 2 → 1. Suspected cause: the relay re-claimed the laser on tiny mouse movement after a pause (sensor jitter). Fix: `POINTER_WAKE_COUNTS` (40 counts in 1 s) before waking or re-claiming. Clicks and scroll wake immediately.
|
||||
|
||||
- 2026-09-25, spike 2 (the mouse drives the pointer): it works end to end. The Z3's motion aims SteamVR's laser, and left click, right click, and scroll act on the dashboard. The invisible render model works: `{ft_pointer}/rendermodels/ft_pointer_invisible` is one tiny triangle with a transparent texture. The claim button (`/input/a` bound to `lasermouse_secondary/switchlaserhand`) takes the laser without clicking.
|
||||
- Laser looks: `dashboard.laserRayWidthScale` (not in the Settings UI; default 1.0) = 0 hides the beam while it hits a panel. `dashboard.laserLength` is the Settings UI's "Laser Pointer Length" (0.5 = 50%, default). Neither hides the laser when it hits nothing. With an eye-origin ray the beam is still visible, because of stereo (each eye is ~31 mm off the ray).
|
||||
- `vrcmd --overlays` lists every overlay (key, visibility, type, flags, handle): `system.systemui` (Steam UI), `valve.steam.desktopgame.*`, `gamescope.*`, `system.pointer` (the laser hit dot), and so on. `vrcmd --compositorcmd dump_laser_overlays` isn't handled by this build.
|
||||
- Next (spike 3): the pointer helper. Relay → helper → driver. The helper does collision (enumerate visible overlays, `ComputeOverlayIntersection` from the anchor, cursor snaps onto the surface). It adds a laser-catching dot overlay at the cursor point in empty space, so the laser always hits something. It toggles the laser width by who owns the dashboard pointer.
|
||||
|
||||
- 2026-09-25, spike 1 results:
|
||||
- Holding the right-hand role while SteamVR starts leaves the Steam UI stuck on its loading icon. So the driver starts disconnected (`deviceIsConnected=false`) and only connects on `show`.
|
||||
- Once connected it becomes SteamVR's right hand (`GetTrackedDeviceIndexForControllerRole(Right)` = our device, even with the real controllers on). The real controllers still had their lasers.
|
||||
- The dashboard's pointer device (`IVROverlay::GetPrimaryDashboardDevice`) goes to whichever device summoned the dashboard or last pressed its trigger. The headset's side button selects the HMD head pointer (device 0).
|
||||
- With `/pose/raw` bound as `lasermouse/Pointer` and `lasermouse_secondary/switchlaserhand` on the trigger: a virtual trigger press moved the pointer to our device, and our system button closed and reopened the dashboard with ours as the pointer. `/pose/tip` didn't work, because tip comes from a render model and ours has none.
|
||||
- Still to see in the headset: our dot anchored in the room (`install.sh aimhere`), and a click landing on a target.
|
||||
- `pointer/probe/vrprobe` (OpenVR background client) prints devices, roles, the dashboard pointer, and head yaw and pitch. `vrcmd --info` also prints "Dashboard pointer device".
|
||||
|
||||
- 2026-09-25: Spike 1 prep. `pointer/driver/` holds the `ft_pointer` driver: a virtual controller with room-anchored `aim`/`gaze` pose from the head, buttons over the abstract datagram socket `@ft_pointer`, and default vrcompositor bindings to the lasermouse actions for both hands. It builds in `dev` for the host with `-static-libstdc++ -static-libgcc -Wl,--exclude-libs,ALL -fno-math-errno`: libm's `sqrtf` is versioned `GLIBC_2.43` in the container, which is newer than the host's 2.39. Its only export is `HmdDriverFactory`. Installed to `~/.local/share/frametop/ft_pointer` and registered with `vrpathreg adddriver`. Not loaded yet: that needs a SteamVR restart.
|
||||
|
||||
- 2026-09-25: The Z3 stopped reaching the desktop after it reconnected at 18:00:20 (sleep), seven minutes after SteamVR started. `vrserver` and `vrcompositor` held `/dev/input/event5`-`event8` as `(deleted)`. Restarting SteamVR fixed it once. The durable fix is `input/input-relay.py`: uinput virtual mouse and keyboard, created before SteamVR, plus EVIOCGRAB relay of USB and Bluetooth mice and keyboards with hotplug. Tested on the Frame with `--no-grab` and a fake USB uinput mouse: motion and BTN_LEFT relayed, unplug released cleanly, Z3 Mouse and Z3 Keyboard picked up, and the Z3 joystick nodes ignored. Enabled as a user service, not yet started.
|
||||
|
||||
- 2026-09-25: Chromium (Flatpak `org.chromium.Chromium`, system install) didn't launch from the taskbar. Plasma ran `kde-open appstream://org.chromium.Chromium`, which opens Discover, because the nested session had no `XDG_DATA_DIRS` and so no Flatpak exports. Fix: the session script sources `/etc/profile.d/flatpak.sh` (with a default `XDG_DATA_DIRS` first, since the script runs `set -u`). With the right env, Chromium runs fine in the nested session over Xwayland. Harmless log noise: `vaInitialize failed` (no VA-API) and a GCM `DEPRECATED_ENDPOINT`.
|
||||
|
||||
- 2026-09-25: VNC only. RealVNC Viewer can't speak RDP. Bridge: krdp on `127.0.0.1:3390`, then `xfreerdp` full screen inside `Xvnc :20`, then VNC on `<tailnet ip>:5900`. Verified with a screenshot of `:20`.
|
||||
- 2026-09-25: Changing `SCREENS` can orphan Plasma panels. With 2 screens the taskbar panels were saved with `lastScreen=1`. After switching to 1 screen, no taskbar showed. Fixed by moving `~/.config/frametop/plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` aside (`*.bak-2screens`). TODO: handle this in the session script when the screen count shrinks.
|
||||
- 2026-09-25: Remote desktop. `krfb` needs `xdg-desktop-portal-kde` (plugins `pw` and `xdp` only on Wayland), and `wayvnc` is wlroots-only. `krdpserver --plasma` (krdp 6.7.5, Fedora) uses KWin's screencast and fake-input protocols directly. It works against the nested KWin 6.2.5 once `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` is set, which is needed because KWin can't match a container binary to a desktop file. Tested with `xvfb-run xfreerdp`: the right password connects and streams H.264 (OpenH264, no VA-API), and a wrong password is rejected at PostConnect. Reachable from the Mac over the tailnet.
|
||||
- 2026-09-25: gamescope doesn't always exit on SIGTERM when started from the Steam launcher. `desktops.sh stop` waits 10 s, then sends SIGKILL.
|
||||
- 2026-09-25: A test of gamescope PerWindow with a bare nested `kwin_wayland --output-count 2` produced two separate overlays. The full Plasma version then showed two sharp 1080p desktops with wallpaper, a taskbar, and movable panels.
|
||||
- 2026-09-25: A headless `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1 kwin_wayland --virtual --output-count 2` runs next to the VR session. With the env var it exposes `zkde_screencast_unstable_v1` and `org_kde_kwin_fake_input`. Without it, KWin hides them. That's useful if the helper injects input through fake_input.
|
||||
- 2026-09-25: SteamOS runs its own nested Plasma session inside gamescope (`/run/user/1000/nested_plasma`, socket `wayland-0`, X display `:2`). This is the built-in single-screen desktop in VR. Don't touch it.
|
||||
- 2026-09-25: On the Frame, `gamescope --virtual-connector-strategy` accepts `SingleApplication`, `SteamControlled`, `PerAppId`, and `PerWindow`. The main VR session uses `PerAppId`.
|
||||
@@ -0,0 +1,132 @@
|
||||
# Frametop
|
||||
|
||||
Several desktop screens floating in SteamVR on the Steam Frame, driven by a physical mouse (Bluetooth or USB) whose cursor moves through 3D space. The cursor crosses from one screen to the next based on where the screens actually sit around you, not on a flat monitor layout.
|
||||
|
||||
How it works: a nested Plasma desktop runs inside ft-screens (`screens/`), our own small Wayland compositor. KWin opens a window per screen; ft-screens gives each one its own size, so every screen is a real monitor of any resolution and shape (ultrawide, portrait, 4K), and shows each as its own SteamVR panel, with KWin's frames passed to SteamVR as they are (no copy). The panels are ours: any size in metres, placed exactly by the layout, with a grab bar to move them, a handle to resize them, and pinning to a hand. See `docs/design.md`. (The older gamescope backend is still there: `BACKEND=gamescope`.)
|
||||
|
||||
Status: the multi-screen desktop works on ft-screens (acceptance test: a 3440 × 1440 ultrawide between two 1080 × 1920 portrait screens, wallpaper on each, the taskbar on the ultrawide). The universal 3D mouse works with the SteamVR dashboard, Steam, overlays, and the screens.
|
||||
|
||||
## Run
|
||||
|
||||
From the headset: open "Launch a program", then "Desktop". After `install`, that entry starts Frametop instead of the stock single-screen desktop.
|
||||
|
||||
From a terminal, on the Frame or from a PC over SSH:
|
||||
|
||||
```
|
||||
desktops.sh install # launcher "Desktop" starts Frametop (writes ~/.local/share/applications/deckard-nested-desktop.desktop)
|
||||
desktops.sh uninstall # launcher gets the stock SteamOS desktop back
|
||||
desktops.sh screens 3 # default screen count
|
||||
desktops.sh start [screens] | stop | restart | status | log [lines]
|
||||
```
|
||||
|
||||
Settings: the screens (resolution, width in metres, scale, taskbar screen) and the layout are in `~/.config/frametop-layout.json`; `BACKEND`, `REMOTE`, and the pointer settings in `~/.config/frametop.conf` (see `session/frametop.conf.example`). Frametop Display Settings (below) edits both.
|
||||
|
||||
The session script is `session/frametop-session.sh`. It runs on the Frame host and starts ft-screens in the `dev` container (`/tmp/frametop-screens.log`), then KWin and Plasma on the host inside it. Only one instance runs at a time, and `desktops.sh start` runs it in its own systemd unit (`frametop-desktop`). It keeps its own Plasma config in `~/.config/frametop`, separate from the stock desktop.
|
||||
|
||||
Restarting the desktop (`desktops.sh restart`, or Restart desktop in Frametop Display Settings) closes its windows, but programs started in it keep running when they can: `session/keep-apps.sh` moves them out of the desktop's systemd unit first. Background work like servers, tmux, and builds survives. An app that stops its own helpers when its window closes still loses them; for work that must survive, run it outside the desktop, for example as a systemd user service.
|
||||
|
||||
## ft-screens (the compositor)
|
||||
|
||||
`screens/compositor.c` (wlroots 0.20) hosts the nested KWin; `screens/vr.cpp` is the SteamVR side. Build: `screens/build.sh` (the installer does it).
|
||||
|
||||
- Screens: each KWin window is a screen. ft-screens sizes it (`xdg_toplevel` configure) and KWin resizes that screen to match, live. Frames arrive as DMA-BUFs and go to SteamVR with `ImportDmabuf` (OpenVR's `IVRIPCResourceManagerClient`), no copy, no size limit.
|
||||
- Panels: `frametop.screen.N`, with `.bar` (move), `.curve` (the round button next to it), `.roll` (the next one: drag it sideways like a knob to roll the screen, snapping level within 2.5°, or scroll on it for 5° steps), and `.resize` (the tab on the bottom right corner; screens go down to 15 cm wide). The controls are sized from both the screen's width and its distance from you, sit on its surface when it's curved, and are translucent like SteamVR's own until a laser is on them. They're invisible until a laser or the 3D mouse's cursor comes very close to one of them (about 1.5 times a button's size), and fade out a moment after it leaves. Drag the bar with any laser (a controller, or the 3D mouse, whose right-drag tilt works too); scroll while dragging to push it away or pull it closer (along the line from your head). The curve button bends the screen into a cylinder around you (radius: your distance to it), or flat again. Pin to a wrist: while carrying a screen, sweep the laser (the line from whatever carries it to its bar) across your other controller. A ring around that controller shows the target and a dot shows where the laser passes; entering the ring arms the pin (ring and bar turn blue), entering it again disarms it. Let go while armed and the screen rides on that controller as it is then, at its size and distance (a 3.6 m screen 5 m away works; so does pinning all screens), so you can arm it first and then turn it the way you want. Grab a pinned screen's bar to adjust it: it comes back to the same wrist when you let go, unless you sweep across the ring to disarm. A pinned screen shows only while you see its front within the wrist angle, fading over the last 10°.
|
||||
- Visibility (Frametop Display Settings → Visibility & wrist tab): always (Meta+Shift+H, the Hide/Show Screens menu entry, or a mapped mouse button hides them), only with the SteamVR dashboard open, while you look at a chosen controller (the wrist gesture), or only when shown with the hotkey. In the last three, the hotkey shows them anyway. Visible screens keep SteamVR's laser on; hidden, VR games get their triggers back. (A controller button to show them isn't there yet: in a game the game owns the buttons.)
|
||||
- Input: pointer from the panels to KWin through our seat; keys from the input relay (every keyboard it doesn't grab, and keys a pointer device passes through) to the screen that was clicked last, but not while the SteamVR dashboard is open.
|
||||
- Control socket `@ft_screens` (datagrams, replies to the sender): `place N x y z yaw pitch roll`, `width N metres`, `curve N radius|on|off`, `pin N|all left|right [12 numbers]`, `unpin N|all`, `size N w h` (live resolution), `get N`, `screens`, `head`, `visibility always|dashboard|gesture|toggle`, `wrist degrees`, `gesture left|right degrees`, `hide | show | toggle`, `state`, `key code value`.
|
||||
|
||||
## Input relay (Bluetooth mice and keyboards)
|
||||
|
||||
SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps and reconnects gets new device nodes, and SteamVR keeps reading the dead ones, so the mouse stops working until SteamVR restarts. `input/input-relay.py` fixes that:
|
||||
|
||||
- It creates `frametop virtual mouse` and `frametop virtual keyboard` through `/dev/uinput` before SteamVR starts.
|
||||
- It grabs every USB or Bluetooth mouse and keyboard as they come and go, and forwards their events. SteamVR only sees the virtual devices, which never go away.
|
||||
- Service: `frametop-input-relay.service` (user unit, `Before=steamvr.service`, wanted by `steamvr.service` and `default.target`).
|
||||
|
||||
```
|
||||
desktops.sh relay install # enable (starts on the next reboot or SteamVR start)
|
||||
desktops.sh relay status | log | uninstall
|
||||
```
|
||||
|
||||
- The first time, start it before SteamVR (reboot, or restart SteamVR after `relay install`), so SteamVR opens its virtual devices. After that, restarting the relay is safe: systemd keeps the virtual devices in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices.
|
||||
- Test without disturbing SteamVR: `input-relay.py --no-grab`.
|
||||
- This is where the 3D mouse will hook in.
|
||||
|
||||
## Universal 3D mouse
|
||||
|
||||
A Bluetooth mouse drives SteamVR like a controller laser, but it looks like a small dot floating in the room. It snaps onto panels and works on the dashboard, Steam, overlays, and this desktop. Details and findings are in `docs/design.md`, "The universal 3D mouse".
|
||||
|
||||
- `input/input-relay.py` (pointer mode, `POINTER=1`) sends mouse motion, clicks, and scroll to the helper. Deliberate movement or a click wakes it; 30 s idle releases it.
|
||||
- `pointer/helper/ft-pointer` (`frametop-pointer.service`, runs in the `dev` container, starts with SteamVR) holds the room-anchored cursor. It does collision against every visible overlay, draws the white dot, and sends the driver an exact pose.
|
||||
- `pointer/driver/` (`ft_pointer`, loaded by SteamVR) is an invisible virtual right-hand controller whose laser follows the cursor.
|
||||
- Last used wins: picking up a controller hands the laser back at once, and moving the mouse takes it again. While you hold a controller the mouse steps aside.
|
||||
- Moving panels: left-drag a floating panel's grab bar, and it follows the pointer around you (SteamVR's own move). The scroll wheel during a drag pushes and pulls it. **Tilt**: while left-dragging, hold the right button and move the mouse to rotate the panel around the grab point. The right press isn't sent as a right-click. The tilt stays for the rest of the drag: release right and keep moving the tilted panel, press right again to tilt further. Releasing left drops the panel as it is.
|
||||
- Toggle dashboard (a mapped button or a Meta tap) wakes the pointer if needed and holds the virtual system button for 0.12 s. SteamVR ignores a press and release in the same instant.
|
||||
|
||||
```
|
||||
pointer/driver/build.sh && pointer/driver/install.sh install # then restart SteamVR
|
||||
pointer/helper/build.sh && pointer/helper/run.sh install # user service
|
||||
pointer/helper/run.sh status | log | restart
|
||||
pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer
|
||||
```
|
||||
|
||||
Settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, and `POINTER_LASER_WIDTH`. See `session/frametop.conf.example`. Restart the relay or the helper after changing them.
|
||||
|
||||
## Frametop Input Settings (app)
|
||||
|
||||
A Plasma app (Kirigami, Python backend) to choose and map input devices. It's in the Plasma menu under Settings on the Frametop desktop (and the stock desktop). It runs in the `dev` container and talks to the relay's control socket `@frametop_relay`.
|
||||
|
||||
- **Devices**: every USB or Bluetooth mouse and keyboard, with a live activity light (move or press a device to find its row). Roles: **3D pointer** (grabbed, drives the pointer; default for anything with a mouse node), **Pass through** (not grabbed; default for keyboards; a Meta tap still toggles the dashboard), **Ignore**. A physical device is identified by its Bluetooth address (or USB ids and name), so all its nodes share a role.
|
||||
- **Buttons**: pick a pointer device (devices with saved bindings are listed even while asleep or disconnected, marked "not connected"; **Forget** on the Devices page drops all of a device's saved settings), press **Capture a button**, press the button or key, then choose an action: left, right, or middle click, back, scroll up or down, toggle dashboard, recenter, pointer on or off, faster or slower, pass through as key, or do nothing. The Z3's extra buttons arrive as keys from its keyboard node. Each row has **Remove** (your own binding; the button passes through again), **Reset** (a changed built-in button goes back to its default), or **Unbind** (a built-in button does nothing). **Remove all** clears the device's custom bindings. `FT_INPUT_PAGE=buttons` opens the app on that page.
|
||||
- **Pointer**: sliders for the `POINTER_*` settings, applied live (the relay and helper reload), plus Recenter.
|
||||
- **Bluetooth**: paired devices, and **Apply Bluetooth fixes** (runs `/etc/steamframe/bt-fixups.sh` through `pkexec`) after pairing an LE device. Pair new devices in Steam.
|
||||
|
||||
Rules are saved to `~/.config/frametop-input.json` and pointer settings to `~/.config/frametop.conf`.
|
||||
|
||||
```
|
||||
input-settings/install.sh # menu entry (ft-input-settings.desktop) -> host launcher ft-input-settings
|
||||
```
|
||||
|
||||
The launcher gives podman the real `XDG_RUNTIME_DIR` and user bus, and gives the app the session's Wayland socket (absolute path) and bus. Without the real user bus, podman fails with `crun: ... cgroup.procs: Permission denied`, because the Frametop session runs on a private bus from `dbus-run-session`.
|
||||
|
||||
## Screens and layout (Frametop Display Settings, ft-layout)
|
||||
|
||||
When the desktop starts, its screens arrange themselves around where you're facing. Move them by hand any time; put them back with **Meta+Shift+R** in the desktop, the **Reset Screen Layout** menu entry, **Arrange now** in the app, or a mouse button mapped to **Reset desktop screen layout** (Frametop Input Settings → Buttons).
|
||||
|
||||
**Frametop Display Settings** (Plasma menu, Settings; Kirigami app in the `dev` container, like Frametop Input Settings):
|
||||
|
||||
- **Screens**: add and remove screens; each has a resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), a width in VR in metres (0.5 to 6), a scale, **Curved**, and **Taskbar here**. Resolution, width, and curve apply at once; adding or removing a screen when the desktop starts again (the app offers the restart).
|
||||
- **Layout**: curved around you (screens hinged edge to edge like monitors on a desk, each turned to face you) or a flat wall, with rows, distance, gap, and height; or **Save current arrangement** to keep where you put the screens by hand (and their sizes). A preview shows it from above and from the front. **When the desktop starts** turns auto-arrange on or off.
|
||||
|
||||
`layout/ft-layout` does the work (Python standard library, on the host):
|
||||
|
||||
```
|
||||
layout/ft-layout apply # arrange every screen (instant with ft-screens)
|
||||
layout/ft-layout capture # save the current arrangement (and sizes) as the layout
|
||||
layout/ft-layout plan # the arrangement as JSON (no VR needed)
|
||||
layout/ft-layout scale # per-screen scale, side-by-side positions, taskbar screen to KWin
|
||||
layout/ft-layout toggle # hide or show all screens
|
||||
display-settings/install.sh # menu entries and the Meta+Shift+R / Meta+Shift+H shortcuts
|
||||
```
|
||||
|
||||
The layout is saved in `~/.config/frametop-layout.json`, relative to your head when it's applied. `/tmp/frametop-layout.log` has the startup run. With `BACKEND=gamescope`, `ft-layout` floats each dashboard panel with `vrcmd --dock-overlay` and the pointer helper carries it into place (`place`), since SteamVR's dashboard owns those panels.
|
||||
|
||||
## Remote desktop (VNC)
|
||||
|
||||
With `REMOTE=1` in the config (`desktops.sh remote on`), the session also serves the VR desktop over VNC. Use RealVNC Viewer or macOS Screen Sharing.
|
||||
|
||||
- Address: the Frame's tailnet name or address, port 5900 (`desktops.sh remote info` prints it). It listens on the tailnet address only, not the LAN. It needs Tailscale on the Frame ([deck-tailscale](https://github.com/tailscale-dev/deck-tailscale)).
|
||||
- Password: in `~/.config/frametop-remote/vnc-password` on the Frame. VNC limits it to 8 characters. `desktops.sh remote info` prints it.
|
||||
- Encryption: VNC auth has none of its own, so viewers warn about an unencrypted connection. The traffic is still encrypted by the tailnet (WireGuard), which is why it listens only there.
|
||||
- How it works: no VNC server can capture KWin on SteamOS. `krfb` needs `xdg-desktop-portal-kde`, which SteamOS lacks, and `wayvnc` is wlroots-only. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390` (never reachable from outside). `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a full-screen FreeRDP client connected to it, and serves that over VNC. Everything runs in the `dev` container. The extra hop adds some latency.
|
||||
- Security trade-off: with `REMOTE=1` the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1`, so any app inside the Frametop desktop can capture its screen or inject input. This applies to that nested session only, not the stock desktop.
|
||||
- To rotate the password, delete `~/.config/frametop-remote/` on the Frame and restart the desktop.
|
||||
- Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session, not the VR desktop.
|
||||
- Check what a viewer sees: `import -window root -display :20 /tmp/vnc.png` in the container.
|
||||
|
||||
## Limits
|
||||
|
||||
- Keyboard typing into the screens is wired (relay → ft-screens) but not yet tested with a real keyboard.
|
||||
- There's no pin to the head (HUD) yet, and no controller button to show hidden screens (a mapped mouse or keyboard button works).
|
||||
- The KWin cursor isn't drawn on the screens (KWin draws it as a host cursor, which ft-screens ignores); the 3D mouse's dot and SteamVR's laser dot show where you point.
|
||||
- With `BACKEND=gamescope`: one resolution for all screens, at most 1920 × 1080 worth of pixels; arranging borrows the pointer for a few seconds; a SteamVR update that moves the floating window's grab bar would break arranging (`LAYOUT_GRAB_OFFSET`, the helper's `grabprobe`).
|
||||
Reference in new issue
Block a user