DeeJanuz--frametop/docs/design.md

Frametop design

Why Frametop is built the way it is, and what we learned about SteamVR on the Steam Frame while building it. reference.md describes the parts and how to run them. Read the SteamVR notes here before changing how the screens, the pointer, or the input relay talk to SteamVR.

Goals

• Several desktop screens in SteamVR, each a real monitor with its own resolution and shape, placed anywhere around you.
• One physical mouse that drives a cursor through that 3D arrangement, and through the rest of SteamVR too: the dashboard, Steam, and other overlays.
• Controllers stay fully usable, and VR games aren't disturbed.
• Everything runs on the Frame itself, installed from a terminal on the headset.

Screens

Why our own compositor

SteamOS already shows a single-screen Plasma desktop in VR by running KWin nested inside gamescope. The first version of Frametop did the same with gamescope's PerWindow mode and kwin --output-count N, which gives each KWin output its own SteamVR overlay. It worked, but gamescope has two limits that ruled it out:

• It draws every window into one shared canvas of a single size, letterboxed, and never tells a window what size to be. So every screen had the same resolution and shape; portrait screens had to be rotated outputs on rolled panels.
• Its OpenVR backend allocates an upload buffer of exactly 1920 × 1080 × 4 bytes, so anything larger, such as 3440 × 1440, aborts it at start.

It also leaves the panels to SteamVR's dashboard, which places them and doesn't expose their position to other programs.

ft-screens replaces it. It's a small wlroots compositor that hosts the nested KWin. KWin's nested Wayland backend opens one window per output and resizes an output when the host configures its window, so ft-screens can give each screen any size, live. KWin needs wl_compositor v4 or later, wl_shm, wl_seat, xdg_wm_base, and zwp_linux_dmabuf_v1 v4 with feedback from its host; the rest is optional.

Frames go to SteamVR the same way gamescope sends them: OpenVR's IVRIPCResourceManagerClient imports each DMA-BUF (ImportDmabuf), and the overlay shows it as a TextureType_SharedTextureHandle texture. There's no copy and no size limit. The Frame's SteamVR supports this interface, but the OpenVR header bundled with SteamVR's samples predates it, so the build fetches a pinned header from Valve's openvr repository.

Before settling on this, we tried KWin's screencast virtual outputs. KWin 6.2.5 crashed (std::out_of_range in textureForOutput) streaming one while nested in gamescope, and its headless backend doesn't implement virtual outputs at all.

A few wlroots details worth knowing: wlr_shm_create wants DRM format codes, not wl_shm ones, and KWin asks for server-side decorations before its first commit, when setting the mode would assert, so ft-screens answers on the first commit.

Placing and sizing

Because the overlays are ours, ft-screens places them exactly with SetOverlayTransformAbsolute and draws its own controls under each screen. OpenVR has no overlay-relative transforms, so the controls are repositioned whenever their screen moves.

The curved layout chains screens edge to edge, like monitors on a desk: the middle screen (or the seam between two) straight ahead, and each neighbour hinged at the previous screen's outer edge and turned to face you. An earlier version spaced screens by angle, which assumes every screen sits on the circle; a flat 3.6 m screen's edges are much farther away than its centre, so its neighbours landed in front of it. Solving for the hinge angle needs a scan and bisection, because a simple fixed-point iteration diverges for screens nearly as wide as twice their distance.

SetOverlayCurvature takes the fraction of a full cylinder that the overlay's width covers, so the radius is width / (2π × curvature). The stated width is the arc length, and the cylinder bends toward the viewer: at a 2 m radius, a 3.57 m wide screen's corners sit about 75 cm closer to you and 23 cm further in than a flat screen's would. Controls on a curved screen are placed on that cylinder, and the bar gets the same curvature.

A resize handle has to be able to shrink a screen from any direction, so the dragged corner follows the laser along the screen's diagonal rather than taking the larger of its horizontal and vertical reach. Pushing and pulling a carried screen moves it along the line from your head, because the 3D mouse's virtual controller sits just in front of the bar, below the screen's centre, so the line from the device points mostly upward.

Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's tip component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose.

ComputeOverlayIntersection ignores SetOverlayIntersectionMask, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner.

Pinning

Pinning started as "bring the screen to your wrist", which doesn't work for big screens, because their centre is far from the edge you bring close. It became aiming: while a screen is carried, the line from the carrying device to its bar is tested against the other hand controllers. Crossing a controller's 6 cm ring arms the pin (leaving past 9 cm, so it doesn't flicker), and crossing it again disarms it. The pin happens on release, with the screen's pose at that moment, so you can arm it and then turn the screen. An earlier version pinned the moment the laser touched the wrist, which left the screen at whatever angle the carrying hand had while pointing there.

A pinned screen's alpha follows the angle between its front and the direction to your head, fully visible inside the wrist angle and fading over the last 10°.

A head pin is the same pin on the headset (device index 0): the screen's transform is relative to the headset, so SteamVR keeps it rigidly in your view with no lag from us. It skips the facing rule, since a screen on your head always faces you the way it did when pinned. There's no aiming gesture for it: the line from the carrying device can't sensibly pass through your own head, and a ring in front of your face would be in the way. So it's set from Frametop Display Settings or ft-layout, and it pins the screen where it is. Carrying a head-pinned screen re-pins it on release, like a wrist pin, so it can be adjusted in VR.

Named layouts

Named layouts are now profiles, which also hold which screens are hidden and which apps to open, with where their windows go. See profiles.md.

A profile's screen part is the custom arrangement under a name: each screen's pose relative to your head, width, curve, and pin, but not its resolution or scale, which need a desktop restart or belong to KWin. Using one copies it into the custom arrangement, so everything that applies the layout (desktop start, Meta+Shift+R, Arrange now) works unchanged, and active remembers which name it came from. Saving without a name (ft-layout capture) clears active, because the screens have been placed by hand since. Layouts are kept per screen number, so one saved with a different screen count still applies: missing screens keep their last saved place or the preset's.

Visibility and VR games

VROverlayFlags_MakeOverlaysInteractiveIfVisible keeps SteamVR's laser mouse on while an overlay with that flag is visible. Without it, the laser is off whenever the dashboard is closed: the first click on a panel only turns it on, and the laser turns off again as soon as it leaves every panel. With it, controllers work the screens normally, but the laser also takes the controllers away from a VR game.

IVRApplications::GetCurrentSceneProcessId() is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting.

Floating windows

floating-windows.md describes the feature and its parts. Drag and drop and the clipboard only work between windows of one compositor, so a floating window stays a KWin window and gets a KWin output of its own: one of the spare outputs KWin opens after the screens, shown by ft-screens as a panel cropped to the window. What follows is how KWin 6.2.5 behaves underneath that, from its source (src/backends/wayland/) and from trying it on the Frametop desktop.

KWin's nested outputs

• Disabling a nested output keeps its host window. Output::applyChanges only flips enabled, KWin stops rendering it (no more commits), and Plasma drops its desktop view. So ft-screens keeps the same toplevel, and its screen numbers stay put. Enabling the output again resumes on the same toplevel.
• Each output's host window is titled KDE Wayland Compositor WL-<n>, with - Output disabled appended while it's disabled (WaylandOutput::updateWindowTitle, on every enabledChanged). ft-screens reads the title to tell screens (WL-0 to WL-<SCREENS-1>) from spares, and to see a spare turn on and off.
• A spare resized while it's disabled comes up at the new size on its first frame, so floating a window needn't blink. Outputs with gaps between them are accepted, so ft-floatd places spares apart from the screens and from each other, within Xwayland's 32767-pixel limit.
• KWin keeps a Wayland popup inside its parent's output (XdgPopupWindow::updateRelativePlacement uses the output's placement area), and X11 apps place their menus within the monitor. That's why a floating window's output has a margin around the window: menus and dropdowns open past the window's edges, into the margin.
• KWin makes a nested output the size it's configured to times its scale, rounded (at 1.5 it lays out 1067 × 667 on a 1600 × 1000 buffer), and gives the buffer a whole buffer scale (1.2 becomes 2). A buffer whose size isn't a multiple of that is a protocol error that disconnects KWin, so ft-floatd sizes spares in multiples of it. After a scale change, ft-floatd asks for the output's size again in the new scale's terms, or the next configure would make it the old size times the scale.
• Virtual outputs don't work. createVirtualOutput makes an output window but never adds it to the backend's m_outputs, so findOutput() returns null when the pointer enters it, and the next line dereferences it (Q_ASSERT is compiled out). A click on such a panel would crash KWin. This rules out virtual outputs (stream_virtual_output) for floating windows without a patched KWin.

The pointer

• Pointer positions reach KWin only through motion events: the output's position in the layout plus the position on its window. When ft-screens stops sending motion, KWin's pointer stays put.
• KWin starts an interactive move on the press itself, before any motion. So ft-screens stops sending motion as soon as a press lands in a floating window's title bar (from the frame and client rectangles ft-floatd sends it), with no round trip, and carries the panel instead. KWin gets the release at the press point, and the window moves by nothing on its output.
• KWin's nested backend ignores the position in wl_pointer.enter, and wlroots drops a motion to the position it entered at, so the first click after crossing onto another panel landed where KWin's pointer had been. ft-screens enters one unit off.

The KWin script

The KWin side is a script (float/frametop-float.js), not a C++ effect, because a script keeps working across KWin updates and an effect would have to match the host's exact KWin build. KWin scripts can call D-Bus but can't serve it, so ft-floatd's commands come back through a long poll: the script calls NextCommand, which answers when a command is ready, or empty after 20 seconds, under KWin's 25-second D-Bus timeout. A few things about KWin's script engine:

• windowAdded reports popups as windows of their own (popupWindow true, transient true) with their geometry.
• Setting frameGeometry applies asynchronously: the app has to answer the new size first.
• A script can't read a window's maximize mode, so the script counts a window as maximized when it fills its output's maximize area.
• globalThis isn't defined. print goes to the journal unless QT_FORCE_STDERR_LOGGING=1.

The title bar's float button is Frametop's own window decoration (decoration/), written in QML for KWin's Aurorae engine, which loads it without compiling. A C++ fork of Breeze would have to match SteamOS's exact KDecoration build. A decoration can only make the window requests KWin offers it, so the button toggles keep-below, which has no visible effect on a window alone on its own output, and the script treats keep-below as the floating flag.

KWin's placement memory

KWin keeps each window's geometry, full screen, and maximized state for each layout of the outputs (its PlacementTracker, keyed by every enabled output's name and geometry). When the outputs come back to a layout it has seen, it puts every window back as it was in it. That's for plugging monitors in and out, and it does harm here. A spare output changes size after its window does, so what KWin keeps for a spare's size is the window's next size. Resizing a floating window back to an earlier size made the window and its output flip between two sizes for good (Dolphin went between 1187 and 1424 logical pixels wide, its output between 1687 and 1925). Full screen flipped the same way, and floating or docking one window could move others, even onto a spare or off one.

So the KWin script keeps where each window belongs: where ft-floatd put it, or where it went outside an output change. While KWin changes the outputs, it reports nothing to ft-floatd. Once KWin is done (screensChanged comes after its restore), it puts floating windows back, and the screens' windows too when only spares changed. KWin's resize request hasn't reached the app by then, so the app never sees it. A size asked for is held for a second, since an app can still answer an older request, and then the script takes the size the window has. If screensChanged doesn't come within 2 seconds, the script stops waiting for it.

KWin also ends an interactive move or resize whenever the outputs change. So during a resize by a floating window's edge, ft-floatd only crops the panel to the window, and resizes the output when the drag ends. The margin is the room to grow until then.

The 3D mouse

The mouse works like the pointer on the Apple Vision Pro: a small cursor floats in the room, lands on whatever panel it meets, and acts on it like a controller's laser.

A virtual controller

SteamVR's dashboard and every overlay it hosts are driven by the vrcompositor lasermouse action set: a pointer pose, left, right, and middle click, back, home, scrolling, and a system button that toggles the dashboard. So the 3D mouse is a virtual controller. The ft_pointer driver adds an invisible controller (its render model is a single transparent triangle) with its own controller type and default bindings for vrcompositor and the Steam client. Its /input/a button is bound to lasermouse_secondary/switchlaserhand, which moves the laser to it without clicking. /pose/tip didn't work for the laser, because tip is defined by a render model; /pose/raw does.

Driver poses are in SteamVR's raw tracking space, and client programs work in the standing universe, which on the Frame is about 1.6 m above raw. Mixing them up put the laser's origin 1.6 m above your head. The helper converts using the headset's pose in both spaces every frame.

Frametop's SteamVR clients (ft-pointer, ft-screens, ft-gaze) connect as a background app first and switch to an overlay app only once that works. VR_Init as an overlay app starts vrserver itself when none is running, and one started that way from the dev container never finds the headset. At a boot where the gamescope session timed out, systemd dropped steamvr.service's start job, the pointer service (ordered only After= it) started anyway, and its vrserver made every SteamVR launch fail with HmdNotFound. SteamOS's health check then kept resetting the Steam client and tried to fall back to the previous OS slot. The units also say Requisite=steamvr.service, so they don't start at all when SteamVR's start fails.

The driver starts disconnected, because holding the right-hand role while SteamVR starts leaves the Steam UI stuck on its loading icon. It connects when the mouse is used and claims the right hand. SteamVR keeps a hand role reserved for a disconnected device that still asks for it, so the driver switches its role hint between right hand (connected) and opt-out (not connected).

The cursor

Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with ComputeOverlayIntersection. On a hit, the cursor sits on that surface; otherwise it floats at POINTER_DISTANCE. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in POINTER_IGNORE are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but ComputeOverlayIntersection still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either.

OpenVR has no call to list other programs' overlays, so the helper runs vrcmd --overlays in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately.

The laser starts partway along your line of sight to the cursor rather than at your eye. SteamVR sizes its hit dot by distance from the laser's origin, and a laser from the eye still shows a beam in each eye. Starting it close to the target makes the beam and the dot tiny, while POINTER_ORIGIN_MARGIN keeps the origin in front of the small window controls, which float a few centimetres in front of their panels. The helper's own white dot is the visible cursor. In empty space it's an interactive overlay that the laser lands on, so SteamVR never draws a laser into nothing.

A few overlays need special handling:

• The dashboard's dock and the floating windows' controls are scene-graph overlays with no texture (0 × 0) and a placeholder width, so ComputeOverlayIntersection never hits them. For those the helper tests the overlay's plane within POINTER_SCENE_RADIUS of its origin.
• SteamVR's Settings page is the one page ComputeOverlayIntersection can't find. Steam's pages, Library and the rest, are drawn in valve.steam.gamepadui.main, which it hits exactly. For SteamVR Settings that overlay is hidden, and the page is drawn by the dashboard's scene-graph panel, whose shape OpenVR doesn't give out, and whose transform's plane isn't the page's surface. The laser started behind the page, so most of it took no clicks (they went to a desktop screen behind it), and the page covered the dot. On that page only, the laser now starts near the eye so SteamVR's own hit test finds the page, the dot is drawn close in front of it, and the laser-catching dot sits far behind everything, invisible, with SteamVR's hit dot hidden on it. There the beam and SteamVR's hit dot look like a controller's; everywhere else nothing changes.
• Just off a panel, the cursor stays on that panel's plane within POINTER_EDGE_REACH, so resize margins and window controls just outside the panel are reachable.
• While the left button is held, the cursor keeps the distance it had at the press and stops re-testing collisions, so dragging past a panel's edge doesn't make it jump.

Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (POINTER_FOLLOW=1, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to POINTER_FOLLOW_REACH (70 degrees) from the reference, a corner of your view included. While your head stays within POINTER_LEASH_DEG of the reference, nothing moves on its own. Once your head has been past the leash for POINTER_LEASH_DELAY (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant POINTER_LEASH_RETURN, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.

Gaze mode is experimental and off by default (POINTER_GAZE=1, the Gaze page of Frametop Input Settings, gaze/ft-gazectl on, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (gaze/ft-gazed) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (POINTER_GAZE_MOUSE_MOVE=held, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with free, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for POINTER_GAZE_HOLD (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see docs/gaze-controllers.md). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With POINTER_GAZE_DOT=moving it shows only while the mouse moves it (POINTER_GAZE_SHOW), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than POINTER_GAZE_RETAKE (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within POINTER_GAZE_NUDGE_MAX (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a right click or Meta+K stops. See gaze/README.md for the service, the calibration, and what was measured.

Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.

dashboard.laserRayWidthScale controls the beam's width, but SteamVR only applies a change from its own settings screen or at restart, so it can't be switched per device while running.

Handing the laser back and forth

The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times POINTER_CONTROLLER_PICKUP, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs POINTER_WAKE_COUNTS of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with MakeOverlaysInteractiveIfVisible keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on.

When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (power.turnOffScreensTimeout), unless something keeps it awake. An awake pointer did, and so did the helper's vrcmd runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off.

Moving floating windows

For SteamVR's own floating windows, the dashboard does the moving. A press on a window's grab bar, 7.5 cm below its bottom edge, parents the window to the pressing device with the relative transform at the press. The scroll wheel pushes it along its normal in steps of about 7 cm. The dashboard finishes the move up to 150 ms after the release and reads the device's pose again then, so the helper holds the drag pose for half a second after the button comes up. Tilting works by rotating the virtual controller around the grab point while both buttons are held.

Input relay

SteamVR opens every input device only when it starts. When a Bluetooth mouse sleeps and reconnects, it gets new device nodes, and SteamVR keeps holding the old, deleted ones, so the mouse stops working until SteamVR restarts. The relay creates permanent virtual devices through uinput before SteamVR starts and forwards the real devices' events into them. systemd keeps the virtual devices' file descriptors across relay restarts, so SteamVR never sees them disappear.

Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead.

An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's InputStealer, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's OverlayFocusChanged (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with SHARE_KEYS=1 the relay sends a grabbed keyboard's keys to @frametop_keys as datagrams (key <code> <value> <device name>). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (SO_PASSCRED) and skip any keyboard of its own that the relay grabs too.

Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (wlr_seat_keyboard_notify_enter: Assertion 'surface' failed), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (InputStealer), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing gpio-keys would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (EVIOCSKEYCODE) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed.

Frametop's keyboard opens by itself for a text field on the desktop. The apps run inside the nested KWin, so only KWin knows when a text field has focus, and the way it tells anyone is its input method protocol (zwp_input_method_v1): KWin starts one input method program and activates it whenever the focused app turns on text input. input/ft-textinput is that program, speaking the Wayland wire protocol directly so it needs nothing but Python on the host. It only reports focus. The gamescope session puts QT_IM_MODULE=xim and GTK_IM_MODULE=xim in the systemd user environment; with those, Qt and GTK apps use X input methods and never turn on Wayland text input, so the session script drops them.

The keyboard itself is ft-screens' own panel (screens/keyboard.cpp). We tried SteamVR's first (ShowKeyboardForOverlay), and on the Frame it doesn't fit a desktop. It's Steam's own panel (valve.steam.gamepadui.keyboard), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (SetKeyboardTransformAbsolute) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, like @frametop_keys), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with SetOverlayRaw when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.

The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (steamvr/globalActionSetPriority, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.

The desktop session

The session is modeled on SteamOS's steamos-nested-desktop and runs beside it. It has its own runtime directory, config (~/.config/frametop), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from dbus-run-session, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; session/keep-apps.sh moves those programs out first. And tools that need the real user bus, like podman and distrobox-host-exec, have to be pointed at it explicitly.

The VR launcher starts the session from the Steam client, and the client's environment came along: LD_LIBRARY_PATH pointing at Steam's own runtime, whose libavcodec has no H.264 decoder, so VLC in the desktop couldn't play most videos, plus the client's overlay and launch settings. The session script drops the client's variables before it starts anything. SteamOS's global Mesa settings (/usr/share/deckard/mesavars.sh) stay, and the gamescope session's Vulkan layer (ENABLE_GAMESCOPE_WSI) is only kept for the gamescope backend.

Steam, not systemd, suspends the Frame: after system_idle_suspend_ac_sec (an hour by default) without input on AC power, it logs Switching to power state: k_ESystemPowerState_Sleep and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.

Flatpak apps need XDG_DATA_DIRS to include Flatpak's exports, or Plasma opens Discover instead of launching them, so the session sources /etc/profile.d/flatpak.sh.

The private runtime directory also moves the session's document portal to $XDG_RUNTIME_DIR/frametop/doc, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, /run/user/1000/frametop/doc/ID/NAME. Inside the sandbox the portal is at /run/flatpak/doc, and /run/user/1000 is a private per-app folder (.flatpak/APP/xdg-run in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to /run/flatpak/doc in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit 69ba5e1) by handing Flatpak apps /run/flatpak/doc paths, after which the links go unused.

A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and distrobox enter starts it on demand. When a Frametop service happened to start the dev container, stopping that service stopped the container and everything in it, including the desktop's compositor. scripts/container-up.sh starts the container in a systemd scope of its own before anything enters it.

Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with pgrep -x and pkill -x. That's why the prefix is ft-.

Displays off on a stand

SteamVR decides the headset is off from its proximity sensor, which the driver reads through the DSP, and turns the displays off 5 seconds later. On a display mount that covered the sensor, that never happened: SteamVR kept the headset in use all night (no entering standby for device 0 in vrserver.txt, and XRService's user presence stayed at 1), and Steam didn't sleep either, because its idle count treats a present user as active. The battery went from 100% to 12% overnight on a 5 V, 3 A charger, with the headset drawing about 17 W.

There's no client call that puts the headset in standby. The cv driver's teststandby debug request (IVRDebug::DriverDebugRequest) only answers "Standby unknown hmd" on the Frame. But what the driver does for the displays in standby is write /sys/class/backlight/ae94000.dsi.0/brightness ("cv: Set displays off" writes 0, "Set displays on" the old value), and the video group can write that file, from the container too. So ft-powerd goes by use instead of the sensor and turns the backlight off itself. Tracking and rendering keep running. Turning the backlight off moved the battery current by only about 75 mA (0.5 W), so they're most of the load, but they're also why the displays can wake the moment the headset moves.

Movement is judged within 10-second windows. On the mount, the head pose jittered within 0.5 mm and 0.1 degrees over 20 seconds, and its position drifted 1.7 mm (0.16 degrees) in 4 minutes. Compared with a fixed reference, that drift would count as movement sooner or later and keep the displays on; within 10 seconds it never reaches the 5 mm and 0.5 degree thresholds, and anyone wearing the headset passes them now and then.

Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with dbus-send ... login1.Manager.Suspend boolean:true, and a block inhibitor does stop that (CanSuspend answers "challenge" while one is held), but it stops the power button too. system_idle_suspend_ac_sec is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with -cef-enable-debugging, SteamClient.Settings.SetSetting takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and settingsStore.clientSettings has the current values.

SteamOS updates

On the Frame, SteamVR is part of the OS image (/opt/steamvr, the deckard-steamvr-rel package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the dev container, the Bluetooth fixes are in /etc, which SteamOS keeps across updates, and nothing goes into /usr. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: IVRIPCResourceManagerClient, which is newer than the header SteamVR ships; the text vrcmd --overlays prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that ComputeOverlayIntersection can't find or the scale KWin's nested backend doesn't undo.

scripts/update-check.py, which scripts/doctor.sh runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that vrcmd's format still parses, that the eye tracker's sample timestamp is still at the offset ft-gaze reads, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (--mark-good), and after an update names what changed and what to try by hand.

Approaches we dropped

• WayVR, an existing Wayland desktop for VR. It built and connected to SteamVR on the Frame, but nothing showed in the headset. It has no bindings for the Frame's controllers, and its KDE screen capture needs xdg-desktop-portal-kde, which SteamOS doesn't ship.
• gamescope in PerWindow mode, for the reasons above. Frametop still supports it as BACKEND=gamescope. In that mode SteamVR's dashboard owns the panels, so ft-layout floats each one with vrcmd --dock-overlay and the pointer helper carries it into place with the virtual controller, hovering first and then sliding at 0.5 m/s, because the dashboard exaggerates fast movements.
• A capture pipeline from a headless KWin through KWin's screencast protocol. It's workable, but ft-screens gets the frames directly with less code.

Open questions

• A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens.
• Drawing KWin's cursor on the screens.
• Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing plasma-org.kde.plasma.desktop-appletsrc and plasmashellrc from ~/.config/frametop brings the default panels back.
• Frame pacing and GPU cost with several busy screens haven't been measured.
• Real standby on a stand, with rendering and tracking paused, not just the backlight off. SteamVR has no call for it, and its activity level follows the proximity sensor.