DeeJanuz--frametop/docs/reference.md

Frametop reference

How each part of Frametop works, where its settings live, and the commands for running parts of it by hand. For why it's built this way, see design.md.

The desktop

From the headset, open Launch a program → Desktop. The installer replaces that launcher entry with Frametop's (~/.local/share/applications/deckard-nested-desktop.desktop), and desktops.sh uninstall gives the stock single-screen desktop back.

From a terminal, on the Frame or from a PC over SSH:

desktops.sh install        # the launcher's Desktop entry starts Frametop
desktops.sh uninstall      # back to the stock SteamOS desktop
desktops.sh start | stop | restart | status | log [lines]

session/frametop-session.sh runs the desktop. It starts ft-screens in the dev container (log: /tmp/frametop-screens.log), then KWin and Plasma on the host inside it. Only one desktop runs at a time. desktops.sh start runs it in its own systemd unit, frametop-desktop. It keeps its Plasma config in ~/.config/frametop, separate from the stock desktop's.

When the VR launcher starts the desktop, it inherits the Steam client's environment. The session script drops the client's runtime from it (LD_LIBRARY_PATH, the STEAM_* settings, and the Steam overlay's Vulkan layer), so apps in the desktop use the system's libraries, including its video codecs, just as they would after a normal login.

Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in ~/.config/frametop-layout.json. The backend, remote desktop, and pointer settings are in ~/.config/frametop.conf; session/frametop.conf.example lists every key.

Restarting the desktop closes its windows. Before the unit stops, session/keep-apps.sh moves every program started in the desktop into a systemd scope of its own, so background work such as servers, tmux, and builds keeps running. An app that shuts down its own helper processes when its window closes will still lose them; run that kind of work outside the desktop, for example as a systemd user service.

ft-screens, the compositor

screens/compositor.c is a small wlroots 0.20 compositor that hosts the nested KWin, and screens/vr.cpp is its SteamVR side. screens/build.sh builds it; the installer runs that for you.

Each KWin window is one screen. ft-screens sets its size with an xdg_toplevel configure and KWin resizes the output to match, live. Frames arrive as DMA-BUFs and go to SteamVR through OpenVR's IVRIPCResourceManagerClient::ImportDmabuf, with no copy and no size limit.

Every screen is an overlay named frametop.screen.N with four controls:

• .bar moves the screen. Drag it with any laser or with the 3D mouse, whose right-drag tilts. Scrolling while you drag pushes the screen away or pulls it closer, along the line from your head.
• .curve bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
• .roll rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
• .resize, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.

The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.

To pin a screen to a wrist, carry it by its bar and sweep the laser across your other controller. A ring around that controller marks the target, and a dot shows where the laser passes. Crossing the ring arms the pin, and the ring and bar turn blue; crossing it again disarms it. When you let go while armed, the screen rides on that controller at the size, distance, and angle it had, so you can arm the pin first and then turn the screen the way you want. Grab a pinned screen's bar to adjust it; it goes back to the same wrist when you let go unless you disarm it. A pinned screen shows only while you're looking at its front, within the wrist angle, and fades out over the last 10°.

The Visibility & wrist tab of Frametop Display Settings decides when the screens show:

• Always. Meta+Shift+H, the Hide/Show Screens menu entry, or a mapped mouse button hides them.
• Only while the SteamVR dashboard is open.
• While you look at a chosen controller (the wrist gesture).
• Only after you show them with the hotkey.

In the last three modes the hotkey shows the screens anyway. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:

• During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
• Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.

Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with SHARE_KEYS=1 in ~/.config/frametop.conf, their keys also go to @frametop_keys for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.

ft-screens listens for datagrams on the abstract socket @ft_screens and replies to the sender:

place N x y z yaw pitch roll     width N metres          curve N radius|on|off
pin N|all left|right [matrix]    unpin N|all             size N w h
get N    screens    head    state    key code value
visibility always|dashboard|gesture|toggle    wrist degrees    gesture left|right degrees
hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible

Input relay

SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps and reconnects gets new device nodes, SteamVR keeps reading the dead ones, and the mouse stops working until SteamVR restarts. input/input-relay.py avoids this. It creates two virtual devices, frametop virtual mouse and frametop virtual keyboard, through /dev/uinput before SteamVR starts. It then grabs USB and Bluetooth mice and keyboards as they come and go and forwards their events, so SteamVR only ever sees the virtual devices, which never go away.

It runs as the user service frametop-input-relay.service, ordered before steamvr.service.

The relay also owns the volume keys, on every device that has them, the headset's buttons included. It changes the volume itself (wpctl, 5% a step, repeating while held), and nothing else sees a volume key, gamescope and SteamVR included: on devices with a keymap (the headset's gpio-keys, USB and Bluetooth keyboards) it remaps just the volume entries to unused codes (KEY_MACRO29, KEY_MACRO30), so the headset's click button and the other keys still work, and it grabs pmic_resin, which has only volume down. The keymaps go back when the relay stops. With --no-grab it leaves the volume keys alone.

desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
desktops.sh relay status | log | uninstall
input/input-relay.py --no-grab   # try it without taking devices from SteamVR

The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (FileDescriptorStorePreserve=yes), so SteamVR keeps the same devices.

The 3D mouse

A mouse drives SteamVR the way a controller's laser does, but shows up as a small dot anchored in the room. It snaps onto panels and works on the dashboard, Steam, overlays, and the desktop. Three pieces make it work:

• The input relay, in pointer mode (POINTER=1), sends mouse motion, clicks, and scrolling to the helper. A deliberate movement or a click wakes the pointer, and 30 seconds without mouse input releases it.
• The helper, pointer/helper/ft-pointer, runs in the dev container as frametop-pointer.service and starts with SteamVR. It keeps the cursor, tests it against every visible overlay, draws the dot, and sends the driver an exact pose.
• The driver, pointer/driver/ (ft_pointer), is loaded by SteamVR. It's an invisible virtual right-hand controller whose laser follows the cursor.

Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.

To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with META_DASHBOARD=1) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.

pointer/driver/build.sh && pointer/driver/install.sh install   # then restart SteamVR
pointer/helper/build.sh && pointer/helper/run.sh install
pointer/helper/run.sh status | log | restart
pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer

The pointer settings are in ~/.config/frametop.conf: POINTER_SENSITIVITY, POINTER_IDLE, POINTER_WAKE_COUNTS, POINTER_DISTANCE, POINTER_CURSOR_DEG, POINTER_ORIGIN_FRACTION, POINTER_ORIGIN_MARGIN, POINTER_SCENE_RADIUS, POINTER_EDGE_REACH, POINTER_LASER_WIDTH, the head follow settings POINTER_FOLLOW, POINTER_LEASH_DEG, POINTER_LEASH_DELAY, POINTER_LEASH_RETURN, and POINTER_FOLLOW_REACH, and the gaze mode settings POINTER_GAZE, POINTER_GAZE_RETAKE, POINTER_GAZE_NUDGE_MAX, POINTER_GAZE_HOLD, and POINTER_GAZE_SHOW. The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper.

Frametop Input Settings

A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the dev container and talks to the relay over its control socket, @frametop_relay. It has six pages:

• Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if META_DASHBOARD=1 is in ~/.config/frametop.conf), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
• Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
• Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through. Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (pointer/helper/vrbuttons.h, pointer/helper/actions/) and sends presses to the relay (vrbtn right/a 1), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with vrbind), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (controller_in_games), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (steamvr/globalActionSetPriority), which the page's Global input switch turns on and off. Mappings are saved as controller_buttons in ~/.config/frametop-input.json.
• Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
• Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, whether only one eye is tracked, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Reload calibration, and Forget nudges.
• Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs /etc/steamframe/bt-fixups.sh through pkexec. Pair new devices in Steam.

Device rules are saved in ~/.config/frametop-input.json. input-settings/install.sh installs the menu entry. Its launcher hands podman the real XDG_RUNTIME_DIR and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.

Frametop Display Settings and ft-layout

When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.

The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.

Frametop Display Settings has three tabs:

• Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers.
• Layout: a curve around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, or a flat wall. Both take rows, distance, gap, and height. Save current arrangement keeps the positions and sizes you set by hand instead. A preview shows the layout from above and from the front, and a switch turns auto-arrange at startup on or off.
• Visibility & wrist: the visibility, game, and controller settings described above, the wrist angle, and buttons to pin all screens to a wrist or unpin them.

layout/ft-layout does the arranging. It's a Python script that uses only the standard library and runs on the host:

layout/ft-layout apply      # arrange every screen
layout/ft-layout capture    # save the current arrangement and sizes as the layout
layout/ft-layout plan       # print the arrangement as JSON (no VR needed)
layout/ft-layout scale      # per-screen scale, positions (as the screens are around you), and taskbar screen, to KWin
layout/ft-layout toggle     # hide or show all screens
display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H shortcuts

The layout is stored relative to your head when it's applied. /tmp/frametop-layout.log has the run from the last desktop start.

Remote desktop over VNC

With REMOTE=1 in the config (desktops.sh remote on), the desktop is also served over VNC, for RealVNC Viewer or macOS Screen Sharing. desktops.sh remote info prints the address and password.

It listens on port 5900 on the Frame's Tailscale address only, not the LAN, so it needs Tailscale on the Frame (deck-tailscale). VNC authentication has no encryption of its own, so viewers warn about it, but the tailnet encrypts the traffic. The password is in ~/.config/frametop-remote/vnc-password and VNC limits it to 8 characters. To change it, delete that folder and restart the desktop.

No VNC server can capture KWin on SteamOS directly: krfb needs xdg-desktop-portal-kde, which SteamOS doesn't ship, and wayvnc only works with wlroots compositors. So session/remote-desktop.sh captures the desktop with KDE's krdpserver --plasma on 127.0.0.1:3390, and session/vnc-bridge.sh runs TigerVNC's Xvnc on display :20 with a full-screen FreeRDP client inside it and serves that. Both run in the dev container, and the extra hop adds a little latency.

With remote access on, the nested KWin runs with KWIN_WAYLAND_NO_PERMISSION_CHECKS=1, so any app in the Frametop desktop could capture its screen or inject input. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own xrdp, which starts a separate X11 session rather than showing the VR desktop.

Limits

• There's no way yet to pin a screen to your head like a HUD.
• A controller button can't show hidden screens; a mapped mouse or keyboard button can.
• KWin's cursor isn't drawn on the screens, because KWin draws it as a host cursor, which ft-screens doesn't render. The 3D mouse's dot and SteamVR's laser dot show where you're pointing.
• The old gamescope backend (BACKEND=gamescope) still works, but it gives every screen the same resolution, at most 1920×1080 pixels' worth, and arranging screens borrows the pointer for a few seconds.