mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-14 15:00:53 +02:00
hands: near input on SteamVR's own panels through the pointer helper's laser
The dashboard, the Steam menu, Steam's keyboard and other apps' overlays now
take a poke or a pinch on the content as a click, and a pinch on the border,
like a card or at a corner moves or resizes a floating one. The edge and
corner glows show there too.
OpenVR gives no transform or size for dashboard panels, so ft-screens casts a
7x7 cone of rays from the eyes past each index tip and fits each panel to the
hits' texture coordinates (steampanels.cpp). Near input treats the fitted
panels as foreign: no fist grabs, two-hand stretch, throws, flicks, pushes,
rewinds or two-finger scroll.
ft-screens can't click those panels, so it drives the helper's laser with
"touch" lines (hover, press, move, up, grab, end). A move or resize first
hovers inside the edge and presses the grab bar or resize margin only once a
floating panel's window controls show under it; otherwise the log says why.
The helper keeps its overlay list while hands are allowed and sends it to
ft-screens ("steamvr ...").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
1 parent
7dfdfd24e1
commit
9d6f5984a8
11 files changed
+941
-27
No files matched your search
+2
-1
@@ -6,7 +6,7 @@ It serves two things in Frametop:
|
||||
|
||||
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro. The hole is round, one per hand, until the tracking is good enough for a hand-shaped one (also there, built from the skeleton; see "Hand cutouts and near input" below).
|
||||
- **Rendered hands:** ft-handview draws your hands as a 3D hand model posed by the tracking, Quest-style, over everything (`screens/handview.cpp`; see "Rendered hands" below): with Room View off (a VR game, SteamVR Home, the dashboard, or the panels over the void), where ft-screens doesn't cut, and by default with Room View on too, floating over the hole.
|
||||
- **Near input on Frametop's panels:** poke a screen with your index finger to click, pinch its bar, edges or any corner to move or resize it, hold it like a card or in a fist to carry and turn it, hold it with both hands to stretch it, flick it away or throw it (ft-screens, `screens/handinput.cpp`; see "Hand cutouts and near input").
|
||||
- **Near input on Frametop's panels:** poke a screen with your index finger to click, pinch its bar, edges or any corner to move or resize it, hold it like a card or in a fist to carry and turn it, hold it with both hands to stretch it, flick it away or throw it (ft-screens, `screens/handinput.cpp`; see "Hand cutouts and near input"). SteamVR's own panels (the dashboard, the Steam menu, other apps' overlays) take pokes, pinch clicks, and moves and resizes of floating panels through the pointer helper's laser.
|
||||
- **Pinches and grips:** with `POINTER_HANDS=1`, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (`gaze/`), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.
|
||||
|
||||
Two programs, each a user service that stops when SteamVR does:
|
||||
@@ -338,6 +338,7 @@ Cost, offline (`scripts/bench-matte.sh` on the probe-moves and lit replays, a 34
|
||||
- **Buttons** (the fourth live test: "it's extremely difficult to interact with the button to make the curve curve and not"): at arm's reach the curve button is 8 mm across, 3.5 mm past the bar's end, and the bar (first in the list, with a 1 cm margin) took the six pokes the log has at it, 3-5 mm left of its centre. Now the nearest control wins, the round buttons (curve, reset, dock, close) take a fingertip or a pinch at least `HANDS_BUTTON_REACH` (0.015 m) each way from their centre, and a pinch there presses the button (`HANDS_PINCH_BUTTONS`; 0: the bar's grab zone takes it, as before). A hand at one lights it and shows a soft ring round it. The log names the control pressed.
|
||||
- **Glows at corners, edges and buttons** (`HANDS_EDGE_GLOW`, user, after the fourth live test: "almost like a cutout of the soft glow we had for highlighting, just around the edges and corners when you're within the interaction zone"): as a hand comes to a panel (in its touch zone), the same soft white halo shows only where a pinch would take it: an L round the corner it would resize from (along both edges from it), a short segment along the edge it would move the panel by, or a ring round a button; brighter as the fingers come closer, brightest while resizing from that corner or pressing that button. One overlay per hand (`frametop.guide.hand.left`, `.right`), so it shows with the cutouts off. With it the resize handle no longer follows hands (lasers and the gaze keep it); `HANDS_EDGE_GLOW=0` brings back the handle at the hand's corner and the cutout buffer's rings.
|
||||
- **Remote displays** take all of it as screens do: pokes, pinches, the card and fist, corner resize, throws and "grab that".
|
||||
- **SteamVR's own panels** (2026-10-13, user: "work on the native SteamVR overlay windows"; `screens/vr.cpp`'s "SteamVR's panels by hand", `screens/steampanels.cpp`): the dashboard, the Steam menu, Steam's keyboard and other apps' overlays take a poke or a pinch on the content as a click, a pinch on the border or like a card moves a floating one, at a corner resizes it, and the edge and corner glows show there too. ft-screens can't click them itself, so the pointer helper's laser does what the hand does ("touch" lines, the helper's "Touch"): it goes 10 cm in front of the point, aimed in, and presses there. OpenVR doesn't say where a dashboard panel is or how big, so for each hands frame ft-screens casts 7 x 7 rays from the eyes past each index tip (12 degrees each way), keeps the hits within 40 cm of the tip, and fits the panel to their texture coordinates; the overlay keys come from the helper's overlay list ("steamvr"), which it keeps while hands are allowed. A move or resize first hovers the laser just inside the edge: a floating panel then shows its window controls (`valve.steam.gamepadui.floatingfooter`) under it, and only then does the laser press its grab bar (7.5 cm under the bottom edge) or its resize margin (1 cm past the corner) and follow the hand. A docked dashboard or another app's overlay shows none: the pinch does nothing, and ft-screens' log says so ("no window controls under it"). No fist grabs, two-hand stretch, throws, flicks, pushes, rewinds or two-finger scroll there. One hand at a time: the laser is one device.
|
||||
- **Smoothing:** a grab carries the knuckles as the cameras saw them, smoothed (`include/fh_smooth.h`: One Euro in continuous time, read each frame ft-screens draws), not the point moved ahead along the hand's velocity as the cutouts use: that velocity is mostly noise times the lead. It moves the panel only once it's 1 cm from where the pinch began, then carries on from there (no jump), so a still hand leaves the panel still. The head the panel turns to face is smoothed too. `HANDS_GRAB_SMOOTH_MINCUTOFF` (1 Hz), `HANDS_GRAB_SMOOTH_BETA` (15 Hz per m/s), `HANDS_GRAB_DEADZONE` (0.01 m). On the probe-moves replay (`scripts/test-handinput.sh`, 90 Hz ticks), the carried point's per-tick second difference went from 13-17 mm rms (50 mm p99) to 2.3-2.6 mm (11 mm), its still wobble from 1.3-1.5 to 0.8-1.0 mm, and it trails the capture by 85 ms while moving (it was 10 ms ahead, by the prediction, and 18 mm off any delay).
|
||||
- With the cutouts on, a glow in the cutout buffer shows where the fingertip is on the screen (a ring that tightens as it comes to the surface, a dot while pressed); what a pinch would do where the fingers are shows as the soft white glow (above) either way; the bar and the buttons brighten under a pinch or a fingertip.
|
||||
- It takes the helper's word on which hands are bare and whether hand input is allowed ("hands"); without that word for 3 s, near input is off. The helper sends it with `POINTER_HANDS=1`. A hand typing holds back (the helper's "typing" word, `POINTER_PINCH_TYPING` above) doesn't poke, grab, move the pointer with its fingertip, or make "grab that" (below). `HANDS_NEAR` (1): `on` has the helper send it without `POINTER_HANDS` too (direct touch alone, no pinches for the pointer), 0 turns it off, `force` turns it on without the helper (testing). `ft-screens` command `handinput on|off|force|state`.
|
||||
|
||||
@@ -42,7 +42,7 @@ test-c:
|
||||
-o build/tests/relay-buttons-test
|
||||
build/tests/relay-buttons-test && echo "screens/tests/relay-buttons-test.c: ok"
|
||||
g++ -std=c++17 -O2 -Wall -Wextra -Wno-unused-parameter -Werror screens/tests/handinput_test.cpp \
|
||||
screens/handcut.cpp screens/handinput.cpp screens/palmmenu.cpp screens/grabthat.cpp -o build/tests/handinput-test
|
||||
screens/handcut.cpp screens/handinput.cpp screens/palmmenu.cpp screens/grabthat.cpp screens/steampanels.cpp -o build/tests/handinput-test
|
||||
build/tests/handinput-test && echo "screens/tests/handinput_test.cpp: ok"
|
||||
g++ -std=c++17 -O2 -Wall -Wextra -Wno-unused-parameter -Werror screens/tests/handview_test.cpp \
|
||||
screens/handmesh.cpp screens/handskin.cpp screens/handrobot.cpp screens/handcut.cpp -o build/tests/handview-test
|
||||
|
||||
@@ -436,6 +436,26 @@
|
||||
// laser starts just before the cursor point (see Looks), so a panel nearer to you doesn't
|
||||
// catch it either.
|
||||
//
|
||||
// Touch (ft-screens' hands on SteamVR's own panels; screens/vr.cpp's "SteamVR's panels by hand"):
|
||||
// ft-screens finds the dashboard's panels, the Steam menu, Steam's keyboard and other apps' overlays
|
||||
// near a hand and works out what the hand does there, but only a laser can click them, so it drives
|
||||
// ours: "touch <left|right> <verb> [x y z qw qx qy qz] [key]", the laser's origin and turn in the
|
||||
// standing universe (just in front of the point, its -z aimed in). hover: the laser there; press: the
|
||||
// trigger down, and "click <key>" to ft-screens as a mouse click on that panel sends (typing goes to
|
||||
// Steam); move: there, held; up: the trigger up; grab: there, hovering kTouchHover (a floating panel's
|
||||
// grab bar comes up) before the press (a move by the grab bar, a resize by the margin past a corner,
|
||||
// the pose then following the hand); end: the hand has left, and a pointer the touch woke goes as a
|
||||
// ray's does (POINTER_RAY_RELEASE). A touch wakes the pointer, or claims the laser back from the
|
||||
// mouse's gaze, and a press waits for the claim; a tap's up goes 40 ms after its press at the
|
||||
// least, and after an up the laser holds still kTouchDrop (the dashboard reads the pose a carried
|
||||
// panel drops at a little late). One hand at a time: the other hand's touch waits while the button
|
||||
// is down, and none starts while the mouse, a ray or a gaze pinch holds it. While a touch has the
|
||||
// laser nothing else poses it, and the "a hand touches a panel" and "hand ray gone" releases wait.
|
||||
// For it the overlay list (see OverlayList) runs while hands are allowed, and goes to ft-screens
|
||||
// whenever it changes and every 5 s: "steamvr <seq> <part> <parts> key...", in parts under 400 bytes
|
||||
// (ft-screens reads 512-byte datagrams), without ours (frametop.*), POINTER_IGNORE's, or keys with
|
||||
// spaces.
|
||||
//
|
||||
// Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
|
||||
// follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
|
||||
// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks, and "? headset" adds
|
||||
@@ -444,7 +464,8 @@
|
||||
// overlays (replies with the overlay list as JSON, see OverlayList),
|
||||
// vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
|
||||
// ctl ... and paused 0|1 (the relay, see Handoff), near <capture_ns> <L 0|1|2> <R 0|1|2> (ft-screens:
|
||||
// which hands are near its panels, for the hand ray and the gestures),
|
||||
// which hands are near its panels, for the hand ray and the gestures), touch <left|right> <verb> ...
|
||||
// (ft-screens: a hand on a SteamVR panel, see Touch),
|
||||
// and btn/scroll lines, which are forwarded to the driver unchanged. For layouts, with a
|
||||
// reply datagram to the sender's (abstract) address:
|
||||
// place <overlay> <x> <y> <z> <yaw> <pitch> [roll [grab]]: centre in the standing
|
||||
@@ -880,6 +901,9 @@ Vec3 LimitPitch(Vec3 d, double maxDeg) {
|
||||
// SteamVR Settings page (see the top): the laser starts this far from the eye, the dot is
|
||||
// drawn this far out, and the laser-catching dot sits this far out (metres).
|
||||
constexpr double SETTINGS_ORIGIN = 0.25, SETTINGS_DOT = 0.6, SETTINGS_CATCHER = 8.0;
|
||||
// A touch's grab hovers this long before it presses (a floating panel's grab bar comes up under the
|
||||
// laser), and after a release the laser holds still this long (see "Touch" at the top).
|
||||
constexpr auto kTouchHover = std::chrono::milliseconds(120), kTouchDrop = std::chrono::milliseconds(250);
|
||||
|
||||
// Whether the line of sight from `eye` along `d` crosses the SteamVR Settings page, drawn by
|
||||
// the dashboard's scene-graph panel (valve.steam.gamepadui.frame.menu.N, transform t); see
|
||||
@@ -1323,6 +1347,9 @@ int main() {
|
||||
std::string handsSent; // "hands L R allowed" as last sent to ft-screens
|
||||
Clock::time_point handsSentAt{};
|
||||
bool handsAllowed = false; // hand interaction allowed now (see "Handoff" at the top)
|
||||
unsigned steamListGen = ~0u; // the overlay list as last sent to ft-screens ("steamvr ...", see "Touch")
|
||||
long steamListSeq = 0;
|
||||
Clock::time_point steamListAt{};
|
||||
// Controllers, looked at once a second: type (ours is left out), side from the role hint, charging.
|
||||
struct DeviceInfo {
|
||||
bool controller = false, ours = false, charging = false;
|
||||
@@ -1349,6 +1376,21 @@ int main() {
|
||||
bool rayInactive = false; // a controller took the laser: our dot hides until a pinch claims
|
||||
bool rayWoke = false; // the ray woke the pointer (it goes again when the ray has gone a while)
|
||||
Clock::time_point rayGoneAt{}; // when the last ray went
|
||||
// ft-screens' touches on SteamVR's panels (see "Touch" at the top): the laser goes where it says.
|
||||
struct Touch {
|
||||
int side = -1; // the hand whose touch has the laser (-1: none)
|
||||
Vec3 origin; // where the laser is to be (standing universe)
|
||||
Basis basis{{1, 0, 0}, {0, 1, 0}, {0, 0, 1}};
|
||||
bool posed = false; // a pose came for this touch
|
||||
bool wantDown = false; // the button is to be down
|
||||
bool pressPending = false; // a press to go out (a tap whose "up" came first still clicks)
|
||||
bool down = false; // the button is down (sent)
|
||||
bool grab = false; // a move or resize: it hovers there first (kTouchHover)
|
||||
std::string key; // the panel pressed (for ft-screens' typing, as a click's)
|
||||
Clock::time_point grabAt{}, downAt{}, holdUntil{}, endAt{}, sentAt{};
|
||||
Vec3 sentOrigin;
|
||||
Basis sentBasis{};
|
||||
} touch;
|
||||
double pushOffset = 0; // push and pull: the origin moved this far along the ray (m)
|
||||
Vec3 rayOrigin, rayDir{0, 0, -1}; // the driving ray this frame (standing universe)
|
||||
// Gestures that began while ft-screens may still claim them (see "near" at the top).
|
||||
@@ -1926,6 +1968,69 @@ int main() {
|
||||
nearClaim[0] = l == 1, nearClaim[1] = r == 1; // 2: the ray hides, the gestures stay ours
|
||||
continue;
|
||||
}
|
||||
// ft-screens' touch on a SteamVR panel (see "Touch" at the top)
|
||||
char tside[8], verb[8], tkey[128] = "";
|
||||
double tp[7];
|
||||
const int tgot = std::sscanf(buf, "touch %7s %7s %lf %lf %lf %lf %lf %lf %lf %127s", tside, verb, &tp[0], &tp[1],
|
||||
&tp[2], &tp[3], &tp[4], &tp[5], &tp[6], tkey);
|
||||
if (tgot >= 2) {
|
||||
const auto t = Clock::now();
|
||||
const int k = !std::strcmp(tside, "right") ? 1 : 0;
|
||||
const char *hand = k ? "right" : "left";
|
||||
if (!std::strcmp(verb, "end")) {
|
||||
if (touch.side == k) {
|
||||
if (touch.down) SendTo(out, "ft_pointer", "btn trigger 0");
|
||||
touch.side = -1, touch.down = touch.wantDown = touch.pressPending = false;
|
||||
touch.endAt = t, rayGoneAt = t; // (a pointer it woke goes POINTER_RAY_RELEASE on)
|
||||
posed.valid = false;
|
||||
std::printf("touch %s: off SteamVR's panels\n", hand);
|
||||
std::fflush(stdout);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// the other hand holds the laser down, or the mouse, a ray or the gaze does: not now
|
||||
if ((touch.side >= 0 && touch.side != k && (touch.down || touch.wantDown || touch.pressPending)) ||
|
||||
(touch.side < 0 && (hold.src != Src::None || leftHeld || aimHeld)))
|
||||
continue;
|
||||
if (touch.side != k) {
|
||||
if (touch.side >= 0 && touch.down) SendTo(out, "ft_pointer", "btn trigger 0");
|
||||
touch = Touch{};
|
||||
touch.side = k;
|
||||
cursorDot.Show(false);
|
||||
markerDot.Show(false);
|
||||
std::printf("touch %s: on SteamVR's panels\n", hand);
|
||||
std::fflush(stdout);
|
||||
}
|
||||
handUsed = t;
|
||||
if (!active) {
|
||||
wake(t); // role, show, and the claim 300 ms on
|
||||
rayWoke = active;
|
||||
} else if (!laserOurs() && !claimPending && !claimHeld) {
|
||||
claimPending = true, claimAt = t;
|
||||
}
|
||||
if (tgot >= 9) {
|
||||
// the quaternion (w x y z) to the frame
|
||||
const double n = std::sqrt(tp[3] * tp[3] + tp[4] * tp[4] + tp[5] * tp[5] + tp[6] * tp[6]);
|
||||
if (n > 1e-6) {
|
||||
const double w = tp[3] / n, x = tp[4] / n, y = tp[5] / n, z = tp[6] / n;
|
||||
touch.basis = {{1 - 2 * (y * y + z * z), 2 * (x * y + w * z), 2 * (x * z - w * y)},
|
||||
{2 * (x * y - w * z), 1 - 2 * (x * x + z * z), 2 * (y * z + w * x)},
|
||||
{2 * (x * z + w * y), 2 * (y * z - w * x), 1 - 2 * (x * x + y * y)}};
|
||||
touch.origin = {tp[0], tp[1], tp[2]};
|
||||
touch.posed = true;
|
||||
}
|
||||
}
|
||||
if (!std::strcmp(verb, "press")) {
|
||||
touch.wantDown = touch.pressPending = true, touch.grab = false;
|
||||
touch.key = tgot >= 10 ? tkey : "-";
|
||||
} else if (!std::strcmp(verb, "grab")) {
|
||||
touch.wantDown = touch.pressPending = touch.grab = true, touch.grabAt = t;
|
||||
touch.key = "-";
|
||||
} else if (!std::strcmp(verb, "up") || !std::strcmp(verb, "hover")) {
|
||||
touch.wantDown = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (std::strcmp(buf, "typing") == 0) { // not mouse input: it never wakes the pointer
|
||||
typist.Key(secsOf(Clock::now()));
|
||||
@@ -2098,8 +2203,48 @@ int main() {
|
||||
claimHeld = false;
|
||||
}
|
||||
|
||||
// The overlay list is only needed while the pointer is awake (see OverlayList).
|
||||
overlays.SetPaused(!active || headsetOff);
|
||||
// ft-screens' touch on a SteamVR panel (see "Touch" at the top): the laser there, its button as
|
||||
// asked once the laser is ours (a press that came before the claim landed goes out after it, and a
|
||||
// tap's release 40 ms after its press). After a release the pose holds kTouchDrop, as the
|
||||
// dashboard reads it for a carried panel's drop.
|
||||
if (!active && touch.side >= 0) touch = Touch{}; // (released: the headset off, a controller took it)
|
||||
if (active && touch.side >= 0 && touch.posed) {
|
||||
const bool holdPose = tnow < touch.holdUntil;
|
||||
if (!holdPose && (tnow - touch.sentAt > std::chrono::milliseconds(100) || Length(touch.origin - touch.sentOrigin) > 0.0002 ||
|
||||
Dot(touch.basis.z, touch.sentBasis.z) < 0.999999 || Dot(touch.basis.x, touch.sentBasis.x) < 0.999999)) {
|
||||
sendPose(touch.origin, touch.basis);
|
||||
touch.sentOrigin = touch.origin, touch.sentBasis = touch.basis, touch.sentAt = tnow;
|
||||
}
|
||||
const bool laserReady = !claimPending && !claimHeld && laserOurs() && tnow - touch.sentAt < std::chrono::seconds(1);
|
||||
if (touch.pressPending && laserReady && (!touch.grab || tnow - touch.grabAt >= kTouchHover)) {
|
||||
SendTo(out, "ft_screens", "click " + touch.key); // (typing goes to Steam on a click on its panels)
|
||||
SendTo(out, "ft_pointer", "btn trigger 1");
|
||||
touch.pressPending = false, touch.down = true, touch.downAt = tnow;
|
||||
} else if (touch.down && !touch.wantDown && !touch.pressPending && tnow - touch.downAt >= std::chrono::milliseconds(40)) {
|
||||
SendTo(out, "ft_pointer", "btn trigger 0");
|
||||
touch.down = false, touch.holdUntil = tnow + kTouchDrop;
|
||||
}
|
||||
}
|
||||
|
||||
// The overlay list is only needed while the pointer is awake (see OverlayList), and while hands
|
||||
// may touch SteamVR's panels: ft-screens finds them by it (see "Touch" at the top).
|
||||
overlays.SetPaused(headsetOff || (!active && !(handsOn && handsAllowed)));
|
||||
if (handsOn && handsAllowed && !headsetOff &&
|
||||
(overlays.Generation() != steamListGen || tnow - steamListAt > std::chrono::seconds(5))) {
|
||||
steamListGen = overlays.Generation(), steamListAt = tnow;
|
||||
// "steamvr <seq> <part> <parts> key...", each part within ft-screens' 512-byte datagrams; not
|
||||
// ours, not ignored, nor keys with spaces (the reader splits on them)
|
||||
std::vector<std::string> parts(1);
|
||||
for (const std::string &key : overlays.Keys()) {
|
||||
if (key.rfind("frametop.", 0) == 0 || Ignored(ignore, key) || key.find(' ') != std::string::npos || key.size() > 200) continue;
|
||||
if (parts.back().size() + key.size() > 400) parts.emplace_back();
|
||||
parts.back() += " " + key;
|
||||
}
|
||||
for (size_t i = 0; i < parts.size(); ++i)
|
||||
SendTo(out, "ft_screens", "steamvr " + std::to_string(steamListSeq) + " " + std::to_string(i) + " " +
|
||||
std::to_string(parts.size()) + parts[i]);
|
||||
++steamListSeq;
|
||||
}
|
||||
|
||||
// Laser mode on while the pointer is awake.
|
||||
if (active != laserModeShown) {
|
||||
@@ -2431,7 +2576,7 @@ int main() {
|
||||
rayLeave(false);
|
||||
}
|
||||
// The ray woke the pointer and has been gone a while (no mouse, no gaze meanwhile): release it.
|
||||
if (active && rayWoke && raySide < 0 && hold.src == Src::None && !leftHeld && !gazeOn &&
|
||||
if (active && rayWoke && raySide < 0 && touch.side < 0 && hold.src == Src::None && !leftHeld && !gazeOn &&
|
||||
tnow - rayGoneAt > std::chrono::duration<double>(rayRelease) &&
|
||||
tnow - lastMouse > std::chrono::duration<double>(rayRelease)) {
|
||||
releasePointer("hand ray gone");
|
||||
@@ -2439,7 +2584,7 @@ int main() {
|
||||
// A hand at a panel (ft-screens' near claim: it pokes and taps there): no laser at all (user,
|
||||
// 2026-10-11: the laser from the head to where the fingertip put the cursor got in the way of
|
||||
// tapping, and held the button so pokes were refused). The mouse's next move claims it back.
|
||||
if (active && hold.src == Src::None && !leftHeld && tnow - nearAt < std::chrono::milliseconds(200) &&
|
||||
if (active && hold.src == Src::None && !leftHeld && touch.side < 0 && tnow - nearAt < std::chrono::milliseconds(200) &&
|
||||
(nearClaim[0] || nearClaim[1]) && tnow - lastMouse > std::chrono::seconds(1)) {
|
||||
releasePointer("a hand touches a panel");
|
||||
}
|
||||
@@ -3335,7 +3480,9 @@ int main() {
|
||||
if (seen != visible) visible.swap(seen), ++visibleGen;
|
||||
}
|
||||
|
||||
if (active && anchored && hmd.bPoseIsValid && tilting) {
|
||||
if (active && touch.side >= 0) {
|
||||
// (a touch on a SteamVR panel has the laser: posed above)
|
||||
} else if (active && anchored && hmd.bPoseIsValid && tilting) {
|
||||
// Rotate the device around the grab point; the grabbed panel turns with it.
|
||||
if (tiltStart) {
|
||||
pivot = lastPoint;
|
||||
|
||||
+2
-1
@@ -30,13 +30,14 @@ $cxx -c -o build/handrender.o handrender.cpp
|
||||
$cxx -c -o build/handinput.o handinput.cpp
|
||||
$cxx -c -o build/palmmenu.o palmmenu.cpp
|
||||
$cxx -c -o build/grabthat.o grabthat.cpp
|
||||
$cxx -c -o build/steampanels.o steampanels.cpp
|
||||
$cxx -c -o build/handtest.o handtest.cpp
|
||||
$cxx -c -o build/handmesh.o handmesh.cpp
|
||||
$cxx -c -o build/handskin.o handskin.cpp
|
||||
$cxx -c -o build/handrobot.o handrobot.cpp
|
||||
$cxx -c -o build/handview.o handview.cpp
|
||||
vrlibs="$(pkg-config --libs egl glesv2 gbm) $OPENVR_LIBS"
|
||||
g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o \
|
||||
g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o build/steampanels.o \
|
||||
$(pkg-config --libs wlroots-0.20 wayland-server xkbcommon pixman-1) $vrlibs
|
||||
g++ -o build/ft-handtest build/handtest.o build/handcut.o build/handrender.o $vrlibs
|
||||
g++ -o build/ft-handview build/handview.o build/handmesh.o build/handskin.o build/handrobot.o build/handcut.o build/handrender.o $vrlibs
|
||||
|
||||
+14
-9
@@ -603,8 +603,9 @@ void HandInput::EndGrab(int s, bool lost) {
|
||||
if (sd.state != State::Grab) return;
|
||||
Emit(Event::GrabEnd, s, sd.panel, 0, 0, lost);
|
||||
// let go fast, outward: thrown on along that line (Params::cardThrow)
|
||||
const bool carried = sd.zone == Zone::Card || (sd.zone == Zone::Fist && !p_.fistResizeOnly) || sd.zone == Zone::Edge ||
|
||||
sd.zone == Zone::Bar || sd.zone == Zone::Title;
|
||||
// (not SteamVR's panels: the laser lets go where it is)
|
||||
const bool carried = (sd.zone == Zone::Card || (sd.zone == Zone::Fist && !p_.fistResizeOnly) || sd.zone == Zone::Edge ||
|
||||
sd.zone == Zone::Bar || sd.zone == Zone::Title) && !foreign_.count(sd.panel);
|
||||
const int throwsBefore = stats.throws;
|
||||
if (!lost && p_.cardThrow && carried && stretch_.panel != sd.panel && haveHead_ && !sd.track.empty()) {
|
||||
// the fastest stretch outward in the last throwPeakMs before the fingers opened (a hand slows as
|
||||
@@ -743,6 +744,9 @@ void HandInput::Update(const Input &in, const std::vector<Panel> &panels) {
|
||||
g_now = in.nowNs;
|
||||
if (in.haveHead) lastHead_ = in.head, haveHead_ = true;
|
||||
events_.clear(), carries_.clear(), glows_.clear(), hovered_.clear(), corners_.clear(), nearPanels_.clear(), logs_.clear();
|
||||
foreign_.clear();
|
||||
for (const Panel &P : panels)
|
||||
if (P.foreign) foreign_.insert(P.index);
|
||||
const handcut::Hands &H = *in.hands;
|
||||
newFrame_ = H.frame() != frame_;
|
||||
frame_ = H.frame();
|
||||
@@ -903,7 +907,7 @@ void HandInput::Poke(int s, const Input &in, const std::vector<Panel> &panels, c
|
||||
const double ext = Extension(*h);
|
||||
const bool extended = ext >= p_.extended;
|
||||
// two fingers on the content: a swipe scrolls, a lift clicks (Params::twoFingerScroll)
|
||||
if (p_.twoFingerScroll && sd.state == State::Hover && !in.typing[s]) {
|
||||
if (p_.twoFingerScroll && sd.state == State::Hover && !in.typing[s] && !P->foreign) {
|
||||
const Hit m = Locate(*P, h->p[12]);
|
||||
const double together = std::hypot(h->p[8][0] - h->p[12][0], h->p[8][1] - h->p[12][1], h->p[8][2] - h->p[12][2]);
|
||||
// (both tips at about one depth: a flat hand's longer middle finger, 1 cm and more deeper, isn't two)
|
||||
@@ -1320,7 +1324,8 @@ bool HandInput::IsFist(const Hand &h) const { return Extension(h) <= p_.fistInde
|
||||
|
||||
bool HandInput::JoinsOther(int s) const {
|
||||
const Side &sd = side_[s], &o = side_[1 - s];
|
||||
return p_.joinFront > 0 && sd.state == State::Hover && sd.inZone && o.state == State::Grab && o.panel == sd.panel;
|
||||
return p_.joinFront > 0 && sd.state == State::Hover && sd.inZone && o.state == State::Grab && o.panel == sd.panel &&
|
||||
!foreign_.count(sd.panel); // (SteamVR's panels: one laser, one hand)
|
||||
}
|
||||
|
||||
void HandInput::Join(int s, const Input &in, const std::vector<Panel> &panels, bool byGrip) {
|
||||
@@ -1922,7 +1927,7 @@ void HandInput::FistPair(const Input &in, const std::vector<Panel> &panels) {
|
||||
const Panel *best = nullptr;
|
||||
double bestD = 1e9;
|
||||
for (const Panel &P : panels) {
|
||||
if (P.held) continue;
|
||||
if (P.held || P.foreign) continue;
|
||||
double sum = 0;
|
||||
bool ok = true;
|
||||
for (int s = 0; s < 2 && ok; ++s) {
|
||||
@@ -1960,7 +1965,7 @@ void HandInput::FistBegin(int s, const Input &in, const std::vector<Panel> &pane
|
||||
const handcut::Gesture &gr = byPinch ? in.hands->pinch(s) : in.hands->grip(s);
|
||||
if (!p_.fistGrab || sd.state != State::Hover || !sd.inZone || in.typing[s]) return;
|
||||
const Panel *P = Find(panels, sd.panel);
|
||||
if (!P || P->held) return;
|
||||
if (!P || P->held || P->foreign) return;
|
||||
const Hand *h = HandFor(s, in.hands->seen(), gr.handId ? gr.handId : sd.handId);
|
||||
if (!h) return;
|
||||
// the hand at the panel: its palm, knuckles or curled middle finger within the near zone, over it
|
||||
@@ -2337,7 +2342,7 @@ void HandInput::Push(int s, const Input &in, const std::vector<Panel> &panels) {
|
||||
return -(q.normal[0] * n[0] + q.normal[1] * n[1] + q.normal[2] * n[2]);
|
||||
};
|
||||
for (const Panel &P : panels) {
|
||||
if (P.held || heldByHand(P.index)) continue;
|
||||
if (P.held || P.foreign || heldByHand(P.index)) continue;
|
||||
const Hit t = Locate(P, now.palm);
|
||||
if (t.d > p_.pushDepth || t.d < -0.30 || !InFootprint(P, t, 0.02) || facing(P, now, t) < cosFacing) continue;
|
||||
// where it came from: the farthest out the palm was in the last second, at or in front of the
|
||||
@@ -2457,7 +2462,7 @@ void HandInput::Flick(int s, const Input &in, const std::vector<Panel> &panels,
|
||||
if (std::sqrt(Dot(step, step)) / ((q.t - before.t) / 1e9) > p_.flickStartSpeed) continue;
|
||||
double key = 1e9;
|
||||
for (const Panel &c : panels) {
|
||||
if (c.held || heldByHand(c.index)) continue;
|
||||
if (c.held || c.foreign || heldByHand(c.index)) continue;
|
||||
const Hit t = Locate(c, q.palm);
|
||||
if (t.d >= -p_.flickBack && t.d <= p_.flickNear && InFootprint(c, t, 0.05) && t.d < key) key = t.d, P = &c;
|
||||
}
|
||||
@@ -2520,7 +2525,7 @@ void HandInput::Glows(const Input &in, const std::vector<Panel> &panels) {
|
||||
h->Palm(palm);
|
||||
PalmOut(*h, out);
|
||||
for (const Panel &P : panels) {
|
||||
if (P.held || heldByHand(P.index)) continue;
|
||||
if (P.held || P.foreign || heldByHand(P.index)) continue;
|
||||
const Hit t = Locate(P, palm);
|
||||
if (t.d < -p_.flickBack || t.d > p_.flickNear || !InFootprint(P, t, 0.02)) continue;
|
||||
SurfaceNormal(P, t.u, n);
|
||||
|
||||
@@ -87,6 +87,11 @@ struct Panel {
|
||||
int cropX = 0, cropY = 0, cropW = 0, cropH = 0, titleH = 0;
|
||||
bool glows = false; // its cutout buffer can show glows (screens, not floating windows)
|
||||
bool held = false; // a laser is carrying it: hands don't grab it
|
||||
// One of SteamVR's own panels (vr.cpp's SteamPanels: the dashboard, the Steam menu, Steam's keyboard,
|
||||
// other apps' overlays), clicked and carried through the pointer helper's laser: pokes, pinch clicks,
|
||||
// the border's move and a corner's resize, and the glows. No fist grabs, two-hand stretch, throws,
|
||||
// flicks, pushes, two-finger scroll or let-go rewind (the laser can't do them).
|
||||
bool foreign = false;
|
||||
std::vector<Spot> controls;
|
||||
std::vector<Popup> popups;
|
||||
};
|
||||
@@ -670,6 +675,7 @@ public:
|
||||
// claims, its pinches and grips are ours: within the zone itself (or pressing, grabbing, or one
|
||||
// waiting on "grab that"), so a pinch in the margin goes to the ray or the gaze.
|
||||
bool near(int side) const { return side_[side].state != State::Idle && side_[side].state != State::Off; }
|
||||
int nearPanel(int side) const { return near(side) ? side_[side].panel : -1; }
|
||||
bool claims(int side) const {
|
||||
const Side &sd = side_[side];
|
||||
return sd.state == State::Press || sd.state == State::Grab ||
|
||||
@@ -964,6 +970,7 @@ private:
|
||||
std::map<int, double> throwAims_;
|
||||
Guide guides_[2];
|
||||
std::set<int> nearPanels_;
|
||||
std::set<int> foreign_; // this tick's SteamVR panels (Panel::foreign)
|
||||
uint64_t frame_ = 0;
|
||||
bool newFrame_ = false;
|
||||
fhsmooth::Point3 head_; // the head carried panels face, smoothed
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
// SteamVR's own panels for near hand input: see steampanels.h.
|
||||
#include "steampanels.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
|
||||
namespace steampanels {
|
||||
namespace {
|
||||
|
||||
double Det3(const double a[3][3]) {
|
||||
return a[0][0] * (a[1][1] * a[2][2] - a[1][2] * a[2][1]) - a[0][1] * (a[1][0] * a[2][2] - a[1][2] * a[2][0]) +
|
||||
a[0][2] * (a[1][0] * a[2][1] - a[1][1] * a[2][0]);
|
||||
}
|
||||
|
||||
void Cross(const double a[3], const double b[3], double out[3]) {
|
||||
out[0] = a[1] * b[2] - a[2] * b[1], out[1] = a[2] * b[0] - a[0] * b[2], out[2] = a[0] * b[1] - a[1] * b[0];
|
||||
}
|
||||
|
||||
double Len(const double a[3]) { return std::sqrt(a[0] * a[0] + a[1] * a[1] + a[2] * a[2]); }
|
||||
|
||||
Mat Mul(const Mat &a, const Mat &b) {
|
||||
Mat r{};
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 4; ++j) {
|
||||
double v = j == 3 ? a.m[i][3] : 0;
|
||||
for (int k = 0; k < 3; ++k) v += a.m[i][k] * b.m[k][j];
|
||||
r.m[i][j] = float(v);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
Mat Inverse(const Mat &a) { // rigid: R^T, -R^T t
|
||||
Mat r{};
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
for (int j = 0; j < 3; ++j) r.m[i][j] = a.m[j][i];
|
||||
r.m[i][3] = -(a.m[0][i] * a.m[0][3] + a.m[1][i] * a.m[1][3] + a.m[2][i] * a.m[2][3]);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Fit FitPanel(const std::vector<Sample> &samples, int minHits) {
|
||||
Fit f;
|
||||
f.hits = int(samples.size());
|
||||
if (f.hits < std::max(3, minHits)) return f;
|
||||
// normal equations for [1 u v] -> (x, y, z)
|
||||
double ata[3][3] = {}, atb[3][3] = {};
|
||||
double u0 = 1e9, u1 = -1e9, v0 = 1e9, v1 = -1e9;
|
||||
for (const Sample &s : samples) {
|
||||
const double row[3] = {1, s.u, s.v};
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 3; ++j) ata[i][j] += row[i] * row[j], atb[i][j] += row[i] * s.p[j];
|
||||
u0 = std::min(u0, s.u), u1 = std::max(u1, s.u), v0 = std::min(v0, s.v), v1 = std::max(v1, s.v);
|
||||
}
|
||||
// the hits spread over it both ways (a line of them fits no plane)
|
||||
if (u1 - u0 < 1e-3 || v1 - v0 < 1e-3) return f;
|
||||
const double d = Det3(ata);
|
||||
if (std::fabs(d) < 1e-12) return f;
|
||||
double x[3][3]; // x[k][j]: coefficient k (1, u, v) of coordinate j, by Cramer's rule
|
||||
for (int j = 0; j < 3; ++j)
|
||||
for (int k = 0; k < 3; ++k) {
|
||||
double t[3][3];
|
||||
for (int r = 0; r < 3; ++r)
|
||||
for (int c = 0; c < 3; ++c) t[r][c] = c == k ? atb[r][j] : ata[r][c];
|
||||
x[k][j] = Det3(t) / d;
|
||||
}
|
||||
const double *O = x[0], *U = x[1], *V = x[2];
|
||||
const double w = Len(U), h = Len(V);
|
||||
if (w < 0.03 || h < 0.03 || w > 20 || h > 20) return f;
|
||||
double n[3];
|
||||
Cross(U, V, n);
|
||||
const double nl = Len(n);
|
||||
if (nl < 1e-9) return f;
|
||||
double bx[3], bz[3], by[3];
|
||||
for (int i = 0; i < 3; ++i) bx[i] = U[i] / w, bz[i] = n[i] / nl;
|
||||
Cross(bz, bx, by);
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
f.pose.m[i][0] = float(bx[i]), f.pose.m[i][1] = float(by[i]), f.pose.m[i][2] = float(bz[i]);
|
||||
f.pose.m[i][3] = float(O[i] + U[i] * 0.5 + V[i] * 0.5);
|
||||
}
|
||||
f.width = w, f.height = h, f.ok = true;
|
||||
return f;
|
||||
}
|
||||
|
||||
std::vector<std::array<double, 3>> Cone(const double from[3], const double at[3], double halfDeg, int n) {
|
||||
std::vector<std::array<double, 3>> out;
|
||||
double f[3] = {at[0] - from[0], at[1] - from[1], at[2] - from[2]};
|
||||
const double l = Len(f);
|
||||
if (l < 1e-6 || n < 1) return out;
|
||||
for (double &c : f) c /= l;
|
||||
// two directions across it
|
||||
const double up[3] = {0, 1, 0}, side[3] = {1, 0, 0};
|
||||
double a[3], b[3];
|
||||
Cross(f, std::fabs(f[1]) < 0.9 ? up : side, a);
|
||||
const double al = Len(a);
|
||||
for (double &c : a) c /= al;
|
||||
Cross(f, a, b);
|
||||
const double t = std::tan(halfDeg * M_PI / 180);
|
||||
for (int i = 0; i < n; ++i)
|
||||
for (int j = 0; j < n; ++j) {
|
||||
const double s = n == 1 ? 0 : t * (2.0 * i / (n - 1) - 1), r = n == 1 ? 0 : t * (2.0 * j / (n - 1) - 1);
|
||||
double d[3] = {f[0] + s * a[0] + r * b[0], f[1] + s * a[1] + r * b[1], f[2] + s * a[2] + r * b[2]};
|
||||
const double dl = Len(d);
|
||||
out.push_back({d[0] / dl, d[1] / dl, d[2] / dl});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void FromPixels(double width, double height, double x, double y, double *u, double *v) {
|
||||
*u = x / kPxPerM - width / 2;
|
||||
*v = height / 2 - y / kPxPerM;
|
||||
}
|
||||
|
||||
Mat Touch(const Mat &panel, double u, double v, double back) {
|
||||
Mat m = panel;
|
||||
for (int i = 0; i < 3; ++i) m.m[i][3] = float(panel.m[i][3] + panel.m[i][0] * u + panel.m[i][1] * v + panel.m[i][2] * back);
|
||||
return m;
|
||||
}
|
||||
|
||||
void GrabPoint(double width, double height, bool resize, int sx, int sy, double barBelow, double margin, double *u,
|
||||
double *v) {
|
||||
if (resize) {
|
||||
*u = (sx >= 0 ? 1 : -1) * (width / 2 + margin);
|
||||
*v = (sy >= 0 ? 1 : -1) * (height / 2 + margin);
|
||||
} else {
|
||||
*u = 0;
|
||||
*v = -height / 2 - barBelow;
|
||||
}
|
||||
}
|
||||
|
||||
Mat Follow(const Mat &from, const Mat &to, const Mat &laser) { return Mul(to, Mul(Inverse(from), laser)); }
|
||||
|
||||
void Quat(const Mat &a, double q[4]) {
|
||||
const double m00 = a.m[0][0], m11 = a.m[1][1], m22 = a.m[2][2], trace = m00 + m11 + m22;
|
||||
if (trace > 0) {
|
||||
const double s = 0.5 / std::sqrt(trace + 1);
|
||||
q[0] = 0.25 / s, q[1] = (a.m[2][1] - a.m[1][2]) * s, q[2] = (a.m[0][2] - a.m[2][0]) * s, q[3] = (a.m[1][0] - a.m[0][1]) * s;
|
||||
} else if (m00 > m11 && m00 > m22) {
|
||||
const double s = 2 * std::sqrt(1 + m00 - m11 - m22);
|
||||
q[0] = (a.m[2][1] - a.m[1][2]) / s, q[1] = 0.25 * s, q[2] = (a.m[0][1] + a.m[1][0]) / s, q[3] = (a.m[0][2] + a.m[2][0]) / s;
|
||||
} else if (m11 > m22) {
|
||||
const double s = 2 * std::sqrt(1 + m11 - m00 - m22);
|
||||
q[0] = (a.m[0][2] - a.m[2][0]) / s, q[1] = (a.m[0][1] + a.m[1][0]) / s, q[2] = 0.25 * s, q[3] = (a.m[1][2] + a.m[2][1]) / s;
|
||||
} else {
|
||||
const double s = 2 * std::sqrt(1 + m22 - m00 - m11);
|
||||
q[0] = (a.m[1][0] - a.m[0][1]) / s, q[1] = (a.m[0][2] + a.m[2][0]) / s, q[2] = (a.m[1][2] + a.m[2][1]) / s, q[3] = 0.25 * s;
|
||||
}
|
||||
if (q[0] < 0)
|
||||
for (int i = 0; i < 4; ++i) q[i] = -q[i];
|
||||
}
|
||||
|
||||
} // namespace steampanels
|
||||
@@ -0,0 +1,67 @@
|
||||
// SteamVR's own panels for near hand input (vr.cpp's "SteamVR's panels by hand"): the dashboard, the
|
||||
// Steam menu, Steam's keyboard, other apps' overlays. OpenVR won't give out where a dashboard panel
|
||||
// is, nor its size (most report a 0x0 texture), but ComputeOverlayIntersection finds a ray's hit on
|
||||
// it and that hit's texture coordinates. So for each hands frame vr.cpp casts a small cone of rays
|
||||
// from the eyes past each hand's index tip, and FitPanel fits the panel to the hits (as
|
||||
// pointer/common/vrmath.h's ScanPanel does with a whole sphere of them). Near input then treats it as
|
||||
// one of its panels (handinput::Panel::foreign), and the pointer helper's laser does what the hand
|
||||
// does there: Touch puts it a little in front of the point, aimed into it, to hover, click and drag;
|
||||
// GrabPoint gives where a move or a resize presses (the grab bar under a floating panel, the resize
|
||||
// margin just outside a corner), and Follow keeps the laser where it was on a panel a hand carries.
|
||||
//
|
||||
// Pure math: no OpenVR calls (the tests run it, screens/tests/handinput_test.cpp).
|
||||
#pragma once
|
||||
|
||||
#include "handcut.h"
|
||||
|
||||
#include <array>
|
||||
#include <vector>
|
||||
|
||||
namespace steampanels {
|
||||
|
||||
using Mat = handcut::Mat;
|
||||
|
||||
// A ray's hit on a panel: its texture coordinates (u right, v up, 0..1) and the point in the room.
|
||||
struct Sample {
|
||||
double u = 0, v = 0;
|
||||
double p[3] = {0, 0, 0};
|
||||
};
|
||||
|
||||
// A panel fitted to hits: its centre and frame (x right, y up, z out of its front) and size, metres.
|
||||
struct Fit {
|
||||
bool ok = false;
|
||||
Mat pose{};
|
||||
double width = 0, height = 0;
|
||||
int hits = 0;
|
||||
};
|
||||
|
||||
// Least squares p = O + u U + v V over the samples; at least minHits, spread over the panel both ways,
|
||||
// and a size within 3 cm..20 m.
|
||||
Fit FitPanel(const std::vector<Sample> &samples, int minHits = 8);
|
||||
|
||||
// n x n ray directions over a square cone halfDeg each way round the line from `from` to `at`.
|
||||
std::vector<std::array<double, 3>> Cone(const double from[3], const double at[3], double halfDeg, int n);
|
||||
|
||||
// Near input's buffer pixels for a SteamVR panel (handinput::Panel::pxW, pxH), and back to the point
|
||||
// on it (u right, v up from its centre, metres).
|
||||
constexpr double kPxPerM = 1000;
|
||||
void FromPixels(double width, double height, double x, double y, double *u, double *v);
|
||||
|
||||
// The laser touching the point (u, v) on a panel: its origin `back` out in front of it, pointing in
|
||||
// (a device's -z), turned as the panel is (so a carried panel turns with it).
|
||||
Mat Touch(const Mat &panel, double u, double v, double back);
|
||||
|
||||
// Where a hand's move or resize presses the laser, on the panel's plane (u, v from its centre):
|
||||
// a move, the grab bar barBelow under the bottom edge's middle; a resize, `margin` out past corner
|
||||
// (sx, sy) both ways (SteamVR's resize margins sit just outside a floating panel).
|
||||
void GrabPoint(double width, double height, bool resize, int sx, int sy, double barBelow, double margin, double *u,
|
||||
double *v);
|
||||
|
||||
// The laser that took a panel at `from` and holds it, with the panel now at `to`: to from^-1 laser
|
||||
// (the dashboard carries a grabbed panel rigidly with the device).
|
||||
Mat Follow(const Mat &from, const Mat &to, const Mat &laser);
|
||||
|
||||
// The pose's rotation as a quaternion (w, x, y, z), for the pointer helper's "touch" lines.
|
||||
void Quat(const Mat &m, double q[4]);
|
||||
|
||||
} // namespace steampanels
|
||||
@@ -13,6 +13,7 @@
|
||||
#include "../grabthat.h"
|
||||
#include "../palmmenu.h"
|
||||
#include "../softglow.h"
|
||||
#include "../steampanels.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
@@ -5133,6 +5134,202 @@ int ScanFlags(int n, char **dirs) {
|
||||
}
|
||||
|
||||
|
||||
// SteamVR's own panels (steampanels.h, vr.cpp's "SteamVR's panels by hand"): the fit from a cone's
|
||||
// hits, and the laser's touch, grab point and carry.
|
||||
void TestSteamPanels() {
|
||||
using namespace steampanels;
|
||||
// a 0.8 x 0.5 m panel 0.6 m ahead, 1.1 m up, turned 0.4 rad: the hits on a patch of it
|
||||
const Mat P = RotY(0.4, 0.2, 1.1, -0.6);
|
||||
const double W = 0.8, H = 0.5;
|
||||
auto pointAt = [&](double u, double v, double out[3]) { // texture u, v (v up) to the room
|
||||
for (int i = 0; i < 3; ++i) out[i] = P.m[i][3] + P.m[i][0] * (u - 0.5) * W + P.m[i][1] * (v - 0.5) * H;
|
||||
};
|
||||
std::vector<Sample> s;
|
||||
for (int i = 0; i < 7; ++i)
|
||||
for (int j = 0; j < 7; ++j) {
|
||||
Sample x;
|
||||
x.u = 0.3 + 0.05 * i, x.v = 0.6 + 0.04 * j;
|
||||
pointAt(x.u, x.v, x.p);
|
||||
s.push_back(x);
|
||||
}
|
||||
const Fit f = FitPanel(s);
|
||||
CHECK(f.ok && f.hits == 49);
|
||||
NEAR(f.width, W, 1e-4), NEAR(f.height, H, 1e-4);
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 4; ++j) NEAR(f.pose.m[i][j], P.m[i][j], 1e-4);
|
||||
// ...a line of hits fits no plane; too few hits fit nothing
|
||||
std::vector<Sample> line;
|
||||
for (const Sample &x : s)
|
||||
if (std::fabs(x.v - 0.6) < 1e-9) line.push_back(x);
|
||||
CHECK(!FitPanel(line).ok && !FitPanel(std::vector<Sample>(s.begin(), s.begin() + 5)).ok);
|
||||
// ...a speck (2 cm across) isn't a panel
|
||||
std::vector<Sample> speck = s;
|
||||
for (Sample &x : speck) x.p[0] = P.m[0][3] + (x.u - 0.5) * 0.02, x.p[1] = P.m[1][3] + (x.v - 0.5) * 0.02, x.p[2] = P.m[2][3];
|
||||
CHECK(!FitPanel(speck).ok);
|
||||
|
||||
// the cone: n x n unit rays, the middle one along the line, the corners halfDeg off each way
|
||||
const double eye[3] = {0, 1.6, 0}, tip[3] = {0.1, 1.3, -0.5};
|
||||
const auto rays = Cone(eye, tip, 12, 7);
|
||||
CHECK(rays.size() == 49);
|
||||
double ax[3] = {tip[0] - eye[0], tip[1] - eye[1], tip[2] - eye[2]};
|
||||
const double al = std::sqrt(ax[0] * ax[0] + ax[1] * ax[1] + ax[2] * ax[2]);
|
||||
double most = 0;
|
||||
for (const auto &r : rays) {
|
||||
NEAR(std::sqrt(r[0] * r[0] + r[1] * r[1] + r[2] * r[2]), 1.0, 1e-9);
|
||||
most = std::max(most, std::acos(std::min(1.0, (r[0] * ax[0] + r[1] * ax[1] + r[2] * ax[2]) / al)) * 180 / M_PI);
|
||||
}
|
||||
NEAR((rays[24][0] * ax[0] + rays[24][1] * ax[1] + rays[24][2] * ax[2]) / al, 1.0, 1e-9);
|
||||
NEAR(most, std::atan(std::tan(12 * M_PI / 180) * std::sqrt(2.0)) * 180 / M_PI, 0.01);
|
||||
|
||||
// pixels (a millimetre each) to the point on it
|
||||
double u = 0, v = 0;
|
||||
FromPixels(W, H, 400, 250, &u, &v);
|
||||
NEAR(u, 0, 1e-9), NEAR(v, 0, 1e-9);
|
||||
FromPixels(W, H, 0, 0, &u, &v);
|
||||
NEAR(u, -0.4, 1e-9), NEAR(v, 0.25, 1e-9);
|
||||
// the touch: 10 cm out in front of the point, turned as the panel is (its -z aims in)
|
||||
const Mat t = Touch(P, 0.1, -0.05, 0.1);
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
NEAR(t.m[i][3], P.m[i][3] + P.m[i][0] * 0.1 - P.m[i][1] * 0.05 + P.m[i][2] * 0.1, 1e-5);
|
||||
for (int j = 0; j < 3; ++j) NEAR(t.m[i][j], P.m[i][j], 1e-6);
|
||||
}
|
||||
// where a move and a resize press
|
||||
GrabPoint(W, H, false, 0, 0, 0.075, 0.01, &u, &v);
|
||||
NEAR(u, 0, 1e-9), NEAR(v, -0.325, 1e-9);
|
||||
GrabPoint(W, H, true, 1, -1, 0.075, 0.01, &u, &v);
|
||||
NEAR(u, 0.41, 1e-9), NEAR(v, -0.26, 1e-9);
|
||||
GrabPoint(W, H, true, -1, 1, 0.075, 0.01, &u, &v);
|
||||
NEAR(u, -0.41, 1e-9), NEAR(v, 0.26, 1e-9);
|
||||
// the laser follows a carried panel: where it touched it, it still does
|
||||
const Mat Q = RotY(-0.3, -0.4, 1.3, -0.5);
|
||||
const Mat moved = Follow(P, Q, t), want = Touch(Q, 0.1, -0.05, 0.1);
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 4; ++j) NEAR(moved.m[i][j], want.m[i][j], 1e-5);
|
||||
// the quaternion: about y by 0.4, and one past 90 degrees (the trace below zero), unit with w >= 0
|
||||
double q[4];
|
||||
Quat(P, q);
|
||||
NEAR(q[0], std::cos(0.2), 1e-6), NEAR(q[1], 0, 1e-6), NEAR(q[2], std::sin(0.2), 1e-6), NEAR(q[3], 0, 1e-6);
|
||||
Quat(RotY(3.0, 0, 0, 0), q);
|
||||
NEAR(q[0], std::cos(1.5), 1e-5), NEAR(q[2], std::sin(1.5), 1e-5);
|
||||
Quat(RotY(-3.0, 0, 0, 0), q);
|
||||
CHECK(q[0] >= 0), NEAR(std::fabs(q[2]), std::sin(1.5), 1e-5), NEAR(q[0] * q[0] + q[2] * q[2], 1.0, 1e-5);
|
||||
}
|
||||
|
||||
// SteamVR's panels in near input (Panel::foreign): a poke and a pinch click, the border and corners
|
||||
// take them (vr.cpp moves and resizes them with the laser), but nothing throws, flicks, pushes or
|
||||
// rewinds them, a fist doesn't take them, and a second hand doesn't join one.
|
||||
void TestForeignPanel() {
|
||||
auto rig = [](Rig &r) {
|
||||
r.panels[0].index = 1000, r.panels[0].foreign = true, r.panels[0].controls.clear();
|
||||
r.panels[0].pxW = 1000, r.panels[0].pxH = 563;
|
||||
};
|
||||
{ // a poke presses where it touches
|
||||
Rig r;
|
||||
rig(r);
|
||||
for (int mm = 100; mm >= -15; mm -= 10) r.Frame({TipAt(0.1, 0.05, mm / 1000.0)});
|
||||
CHECK(r.Count(Event::Press) == 1);
|
||||
if (const auto p = r.Last(Event::Press)) CHECK(p->panel == 1000), NEAR(p->x, 600, 3), NEAR(p->y, 563 * (0.5 - 0.05 / 0.5625), 3);
|
||||
for (int mm = -15; mm <= 50; mm += 10) r.Frame({TipAt(0.1, 0.05, mm / 1000.0)});
|
||||
CHECK(r.Count(Event::Release) == 1);
|
||||
}
|
||||
{ // a pinch on the content clicks
|
||||
Rig r;
|
||||
rig(r);
|
||||
for (int k = 0; k < 5; ++k) r.Frame({PinchAt(0, 0, 0.01)});
|
||||
const double mid[3] = {0, 0, -0.49};
|
||||
r.synth.Pinch(1, true, r.t, mid, 1);
|
||||
r.Frame({PinchAt(0, 0, 0.01)});
|
||||
CHECK(r.Count(Event::GrabStart) == 0 && r.Count(Event::Press) == 1);
|
||||
}
|
||||
{ // the right edge takes it; carried 10 cm right it moves; let go moving fast, it stays (no throw)
|
||||
Rig r;
|
||||
rig(r);
|
||||
double x = 0.55;
|
||||
for (int k = 0; k < 5; ++k) r.Frame({PinchAt(x, 0, 0.01)});
|
||||
const double at[3] = {x, 0, -0.49};
|
||||
r.synth.Pinch(1, true, r.t, at, 1);
|
||||
r.Frame({PinchAt(x, 0, 0.01)});
|
||||
CHECK(r.Last(Event::GrabStart) && r.Last(Event::GrabStart)->zone == Zone::Edge && r.in.grabbing(1));
|
||||
for (int k = 1; k <= 20; ++k) {
|
||||
if (!r.carries.empty()) r.panels[0].pose = r.carries[0].pose;
|
||||
r.Frame({PinchAt(x += 0.005, 0, 0.01)});
|
||||
}
|
||||
CHECK(!r.carries.empty() && r.carries[0].kind == Carry::Move && r.carries[0].panel == 1000);
|
||||
for (int k = 0; k < 6; ++k) { // out at 1.5 m/s
|
||||
if (!r.carries.empty()) r.panels[0].pose = r.carries[0].pose;
|
||||
r.Frame({PinchAt(x, 0, 0.01 - 0.05 * k)});
|
||||
}
|
||||
r.synth.Pinch(1, false, r.t, at, 1);
|
||||
for (int k = 0; k < 4; ++k) {
|
||||
SynthHand o = PinchAt(x, 0, -0.3 - 0.05 * k);
|
||||
o.open = 1;
|
||||
r.Frame({o});
|
||||
}
|
||||
CHECK(r.Count(Event::GrabEnd) == 1 && r.Count(Event::Throw) == 0 && !r.in.grabbing(1));
|
||||
// ...nor does it go back (the rewind, Settle)
|
||||
for (int k = 0; k < 30; ++k) r.Frame({OpenAt(0.3, 0, -0.1)});
|
||||
CHECK(r.Count(Event::Settle) == 0 && r.Count(Event::Throw) == 0);
|
||||
}
|
||||
{ // a corner resizes
|
||||
Rig r;
|
||||
rig(r);
|
||||
for (int k = 0; k < 5; ++k) r.Frame({PinchAt(0.53, 0.30, 0.01)});
|
||||
const double at[3] = {0.53, 0.30, -0.49};
|
||||
r.synth.Pinch(1, true, r.t, at, 1);
|
||||
r.Frame({PinchAt(0.53, 0.30, 0.01)});
|
||||
const auto g = r.Last(Event::GrabStart);
|
||||
CHECK(g && g->zone == Zone::Corner && g->sx == 1 && g->sy == 1);
|
||||
}
|
||||
{ // a fist doesn't take it
|
||||
Rig r;
|
||||
rig(r);
|
||||
r.in.params().fistResizeOnly = false;
|
||||
for (int k = 0; k < 5; ++k) r.Frame({FistAt(0.0, 0, 0.05)});
|
||||
const double at[3] = {0.0, 0, -0.45};
|
||||
r.synth.Grip(1, true, r.t, at, 1);
|
||||
for (int k = 0; k < 5; ++k) r.Frame({FistAt(0.0, 0, 0.05)});
|
||||
CHECK(r.Count(Event::GrabStart) == 0 && !r.in.grabbing(1));
|
||||
}
|
||||
{ // a flick out through it and an open hand pushed into it: no throw, no throw aim
|
||||
const double a[3] = {0.05, 0, -0.44}, b[3] = {0.05, 0, -0.76};
|
||||
Rig r;
|
||||
rig(r);
|
||||
auto flick = [&](double x, double y, double z) {
|
||||
SynthHand h = OpenAt(x, y, z);
|
||||
if (z > -0.55) h.curl = h.curlIndex = 1;
|
||||
return h;
|
||||
};
|
||||
for (int k = 0; k < 8; ++k) r.Frame({flick(a[0], a[1], a[2])});
|
||||
Sweep(r, a, b, 1.5, flick);
|
||||
CHECK(r.Count(Event::Throw) == 0);
|
||||
Rig p;
|
||||
rig(p);
|
||||
const double c[3] = {0.05, 0, -0.42}, d[3] = {0.05, 0, -0.58};
|
||||
p.Frame({OpenAt(0.05, 0, -0.45)}), p.Frame({OpenAt(0.05, 0, -0.45)});
|
||||
CHECK(!p.in.throwAims().count(1000));
|
||||
for (int k = 0; k < 8; ++k) p.Frame({OpenAt(c[0], c[1], c[2])});
|
||||
Sweep(p, c, d, 0.6, [](double x, double y, double z) { return OpenAt(x, y, z); });
|
||||
CHECK(p.Count(Event::Throw) == 0);
|
||||
}
|
||||
{ // the left hand holds its left edge; the right hand's pinch on the content clicks (no join)
|
||||
Rig r;
|
||||
rig(r);
|
||||
for (int k = 0; k < 5; ++k) r.Frame({PinchAt(-0.55, 0, 0.01, false, 1)});
|
||||
const double l[3] = {-0.55, 0, -0.49};
|
||||
r.synth.Pinch(0, true, r.t, l, 1);
|
||||
r.Frame({PinchAt(-0.55, 0, 0.01, false, 1)});
|
||||
CHECK(r.in.grabbing(0));
|
||||
SynthHand p = PinchAt(0.2, 0.05, 0.01, true, 2);
|
||||
p.open = 0.6;
|
||||
for (int k = 0; k < 4; ++k) r.Frame({PinchAt(-0.55, 0, 0.01, false, 1), p});
|
||||
const double at[3] = {0.2, 0.05, -0.49};
|
||||
r.synth.Pinch(1, true, r.t, at, 2);
|
||||
p.open = 0;
|
||||
r.Frame({PinchAt(-0.55, 0, 0.01, false, 1), p});
|
||||
CHECK(r.in.stats.joins == 0 && !r.in.grabbing(1) && r.Count(Event::Press) == 1);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
@@ -5203,6 +5400,8 @@ int main(int argc, char **argv) {
|
||||
TestJoin();
|
||||
TestFlagTwoFists();
|
||||
TestSoftGlow();
|
||||
TestSteamPanels();
|
||||
TestForeignPanel();
|
||||
std::printf("%d checks, %d failed\n", g_checks, g_fail);
|
||||
return g_fail ? 1 : 0;
|
||||
}
|
||||
+342
-8
@@ -109,6 +109,7 @@
|
||||
#include "palmmenu.h"
|
||||
#include "keyboard.h"
|
||||
#include "roomview.h"
|
||||
#include "steampanels.h"
|
||||
|
||||
#include <drm_fourcc.h>
|
||||
#include <openvr.h>
|
||||
@@ -2416,6 +2417,324 @@ void WantHz(bool near) {
|
||||
close(fd);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- SteamVR's panels by hand
|
||||
|
||||
// Near input on SteamVR's own panels (user, 2026-10-13: "work on the native SteamVR overlay windows"):
|
||||
// the dashboard, the Steam menu, Steam's keyboard, other apps' overlays. A poke or a pinch on one
|
||||
// clicks it, a pinch on its border (or like a card) moves it, at a corner resizes it, and the same
|
||||
// glows show. ft-screens can't click them itself, so the pointer helper's laser (our virtual
|
||||
// controller) does what the hand does: "touch <left|right> <verb> [x y z qw qx qy qz] [key]" on
|
||||
// @ft_pointer_helper, the laser's pose in the standing universe (pointer/helper/ft-pointer.cpp's
|
||||
// "Touch"). Its origin kSteamBack in front of the point, aimed in: hover (the hand near, the laser
|
||||
// there), press (a click: its key goes to ft-screens' typing as a click there would), move (held:
|
||||
// dragging), up, grab (go there, hover, then press: a move or resize), end (the hand has left: the
|
||||
// laser goes back to whoever had it).
|
||||
// OpenVR won't say where a dashboard panel is or how big (most report a 0x0 texture), so for each hands
|
||||
// frame a cone of rays from the eyes past each index tip finds the panels near the hands, and
|
||||
// steampanels::FitPanel fits each to its hits (steampanels.h). The keys come from the helper's overlay
|
||||
// list ("steamvr <seq> <part> <parts> key...", which it keeps while hands are allowed); ours (frametop.*)
|
||||
// and POINTER_IGNORE's aren't on it, and panels the rays can't hit (SteamVR's scene-graph controls)
|
||||
// never fit. A move or a resize first hovers just inside the panel's edge: only a floating panel shows
|
||||
// its window controls then (valve.steam.gamepadui.floatingfooter, under it), and only then does the
|
||||
// laser press its grab bar (kSteamBar under the bottom edge, as the helper's layout placement does) or
|
||||
// its resize margin (kSteamMargin past the corner), and follow the hand rigidly (steampanels::Follow).
|
||||
// A docked dashboard or another app's overlay has none: the pinch does nothing, and the log says so.
|
||||
// No fist grabs, stretch, throws, flicks, pushes or two-finger scroll there (handinput's
|
||||
// Panel::foreign). Only while ft-screens runs (its desktop).
|
||||
constexpr int kSteamIndex = 1000; // near input's panel indices from here on are SteamVR's
|
||||
constexpr double kSteamBack = 0.10; // the laser starts this far in front (its small controls float a few cm out)
|
||||
constexpr double kSteamBar = 0.075; // a floating panel's grab bar under its bottom edge (LAYOUT_GRAB_OFFSET)
|
||||
constexpr double kSteamMargin = 0.01; // ...its resize margin, past a corner each way
|
||||
constexpr double kSteamCone = 12; // the rays, degrees each way round the line from the eyes to the index tip
|
||||
constexpr int kSteamRays = 7; // ...7 x 7 of them
|
||||
constexpr double kSteamReach = 0.40; // ...hitting within this of the tip
|
||||
constexpr int64_t kSteamFitMs = 500; // a panel the rays haven't found this long isn't near input's (unless a hand is at it)
|
||||
|
||||
struct SteamPanel {
|
||||
std::string key;
|
||||
vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
|
||||
int index = -1;
|
||||
bool visible = false;
|
||||
steampanels::Fit fit;
|
||||
int64_t fitMs = INT64_MIN / 2;
|
||||
};
|
||||
std::vector<SteamPanel> g_steamPanels;
|
||||
std::map<std::string, int> g_steamIndex; // a key keeps its index
|
||||
std::vector<std::string> g_steamParts; // the list as it comes, part by part
|
||||
long g_steamSeq = -1;
|
||||
int g_steamPart = 0;
|
||||
int64_t g_steamListMs = INT64_MIN / 2, g_steamLookMs = INT64_MIN / 2;
|
||||
|
||||
// What each hand is doing with the laser on a SteamVR panel.
|
||||
struct SteamTouch {
|
||||
enum class St { None, Hover, Down, GrabWait, Grab, Refused } st = St::None;
|
||||
int panel = -1;
|
||||
std::string key;
|
||||
Mat panel0{}, laser0{}; // the panel as the grab began, and the laser that holds it
|
||||
double w0 = 0, h0 = 0;
|
||||
bool resize = false;
|
||||
int sx = 1, sy = -1;
|
||||
int64_t sinceMs = 0, leftMs = 0;
|
||||
} g_steamTouch[2];
|
||||
|
||||
const SteamPanel *FindSteam(int index) {
|
||||
for (const SteamPanel &sp : g_steamPanels)
|
||||
if (sp.index == index) return &sp;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
// "steamvr <seq> <part> <parts> key key...": the helper's overlay list, in parts that fit a datagram.
|
||||
void SteamListPart(const char *cmd) {
|
||||
long seq;
|
||||
int part, parts, at = 0;
|
||||
if (std::sscanf(cmd, "steamvr %ld %d %d%n", &seq, &part, &parts, &at) < 3 || parts < 1) return;
|
||||
if (part == 0) g_steamParts.clear(), g_steamSeq = seq, g_steamPart = 0;
|
||||
if (seq != g_steamSeq || part != g_steamPart) return; // (a part lost: the next list, a few seconds on)
|
||||
++g_steamPart;
|
||||
std::istringstream words(cmd + at);
|
||||
for (std::string k; words >> k;)
|
||||
if (k.rfind("frametop.", 0) != 0) g_steamParts.push_back(k);
|
||||
if (part != parts - 1) return;
|
||||
g_steamListMs = NowMs();
|
||||
std::vector<SteamPanel> next;
|
||||
for (const std::string &k : g_steamParts) {
|
||||
SteamPanel sp;
|
||||
for (const SteamPanel &o : g_steamPanels)
|
||||
if (o.key == k) sp = o;
|
||||
if (sp.key.empty()) {
|
||||
sp.key = k;
|
||||
auto it = g_steamIndex.find(k);
|
||||
if (it == g_steamIndex.end()) it = g_steamIndex.emplace(k, kSteamIndex + int(g_steamIndex.size())).first;
|
||||
sp.index = it->second;
|
||||
}
|
||||
next.push_back(sp);
|
||||
}
|
||||
g_steamPanels.swap(next);
|
||||
g_steamLookMs = INT64_MIN / 2; // look them up now
|
||||
}
|
||||
|
||||
// For each new hands frame: the panels near the hands, fitted to the rays' hits. Their handles and
|
||||
// whether they show, four times a second.
|
||||
void ScanSteamPanels(const handinput::Input &in) {
|
||||
static uint64_t lastFrame = 0;
|
||||
const int64_t now = NowMs();
|
||||
if (now - g_steamListMs > 60000) g_steamPanels.clear(); // (the helper's list stopped: it's gone)
|
||||
if (g_hands.frame() == lastFrame || !in.haveHead || g_steamPanels.empty()) return;
|
||||
lastFrame = g_hands.frame();
|
||||
if (now - g_steamLookMs >= 250) {
|
||||
g_steamLookMs = now;
|
||||
for (SteamPanel &sp : g_steamPanels) {
|
||||
if (sp.h == vr::k_ulOverlayHandleInvalid || !sp.visible) // (Steam makes some of them again now and then)
|
||||
if (vr::VROverlay()->FindOverlay(sp.key.c_str(), &sp.h) != vr::VROverlayError_None) sp.h = vr::k_ulOverlayHandleInvalid;
|
||||
sp.visible = sp.h != vr::k_ulOverlayHandleInvalid && vr::VROverlay()->IsOverlayVisible(sp.h);
|
||||
}
|
||||
}
|
||||
bool any = false;
|
||||
for (const SteamPanel &sp : g_steamPanels) any = any || sp.visible;
|
||||
if (!any) return;
|
||||
const double eye[3] = {in.head.m[0][3], in.head.m[1][3], in.head.m[2][3]};
|
||||
std::vector<std::vector<steampanels::Sample>> hits(g_steamPanels.size());
|
||||
for (const handcut::Hand &h : g_hands.seen()) {
|
||||
if (!in.allowed[h.right ? 1 : 0]) continue;
|
||||
const double tip[3] = {h.p[8][0], h.p[8][1], h.p[8][2]};
|
||||
for (const auto &d : steampanels::Cone(eye, tip, kSteamCone, kSteamRays))
|
||||
for (size_t i = 0; i < g_steamPanels.size(); ++i) {
|
||||
const SteamPanel &sp = g_steamPanels[i];
|
||||
if (!sp.visible) continue;
|
||||
vr::VROverlayIntersectionParams_t params{};
|
||||
params.vSource = {float(eye[0]), float(eye[1]), float(eye[2])};
|
||||
params.vDirection = {float(d[0]), float(d[1]), float(d[2])};
|
||||
params.eOrigin = vr::TrackingUniverseStanding;
|
||||
vr::VROverlayIntersectionResults_t r{};
|
||||
if (!vr::VROverlay()->ComputeOverlayIntersection(sp.h, ¶ms, &r)) continue;
|
||||
const double off = std::hypot(r.vPoint.v[0] - tip[0], r.vPoint.v[1] - tip[1], r.vPoint.v[2] - tip[2]);
|
||||
if (off > kSteamReach) continue;
|
||||
steampanels::Sample q;
|
||||
q.u = r.vUVs.v[0], q.v = r.vUVs.v[1];
|
||||
for (int k = 0; k < 3; ++k) q.p[k] = r.vPoint.v[k];
|
||||
hits[i].push_back(q);
|
||||
}
|
||||
}
|
||||
for (size_t i = 0; i < g_steamPanels.size(); ++i) {
|
||||
steampanels::Fit f = steampanels::FitPanel(hits[i]);
|
||||
if (!f.ok) continue;
|
||||
// its front toward the eyes (a texture upside down fits facing away: turned about x, it's the same panel)
|
||||
const double to[3] = {eye[0] - f.pose.m[0][3], eye[1] - f.pose.m[1][3], eye[2] - f.pose.m[2][3]};
|
||||
if (to[0] * f.pose.m[0][2] + to[1] * f.pose.m[1][2] + to[2] * f.pose.m[2][2] < 0)
|
||||
for (int k = 0; k < 3; ++k) f.pose.m[k][1] = -f.pose.m[k][1], f.pose.m[k][2] = -f.pose.m[k][2];
|
||||
g_steamPanels[i].fit = f, g_steamPanels[i].fitMs = now;
|
||||
}
|
||||
}
|
||||
|
||||
// SteamVR's panels for near input this tick: those the rays found lately, and those a hand is at.
|
||||
void SteamInputPanels(std::vector<handinput::Panel> &panels) {
|
||||
const int64_t now = NowMs();
|
||||
for (const SteamPanel &sp : g_steamPanels) {
|
||||
if (!sp.visible || !sp.fit.ok) continue;
|
||||
const bool at = g_handInput.nearPanel(0) == sp.index || g_handInput.nearPanel(1) == sp.index ||
|
||||
g_steamTouch[0].panel == sp.index || g_steamTouch[1].panel == sp.index;
|
||||
if (now - sp.fitMs > kSteamFitMs && !at) continue;
|
||||
handinput::Panel P;
|
||||
P.index = sp.index, P.pose = sp.fit.pose, P.width = sp.fit.width, P.height = sp.fit.height, P.curve = 0;
|
||||
P.pxW = int(std::lround(sp.fit.width * steampanels::kPxPerM)), P.pxH = int(std::lround(sp.fit.height * steampanels::kPxPerM));
|
||||
P.foreign = true;
|
||||
panels.push_back(P);
|
||||
}
|
||||
}
|
||||
|
||||
void SendTouch(int side, const char *verb, const Mat *laser, const std::string &key = "") {
|
||||
std::string msg = std::string("touch ") + (side ? "right " : "left ") + verb;
|
||||
if (laser) {
|
||||
double q[4];
|
||||
steampanels::Quat(*laser, q);
|
||||
char b[160];
|
||||
std::snprintf(b, sizeof b, " %.4f %.4f %.4f %.5f %.5f %.5f %.5f", laser->m[0][3], laser->m[1][3], laser->m[2][3], q[0], q[1],
|
||||
q[2], q[3]);
|
||||
msg += b;
|
||||
}
|
||||
if (!key.empty() && key.size() < 100) msg += " " + key;
|
||||
SendPointer(msg);
|
||||
}
|
||||
|
||||
// A floating panel's window controls showing under it (valve.steam.gamepadui.floatingfooter): it's one
|
||||
// the dashboard lets a laser move and resize.
|
||||
bool FooterUnder(const Mat &panel, double w, double h) {
|
||||
vr::VROverlayHandle_t f = vr::k_ulOverlayHandleInvalid;
|
||||
if (vr::VROverlay()->FindOverlay("valve.steam.gamepadui.floatingfooter", &f) != vr::VROverlayError_None ||
|
||||
!vr::VROverlay()->IsOverlayVisible(f))
|
||||
return false;
|
||||
vr::ETrackingUniverseOrigin o;
|
||||
Mat m;
|
||||
if (vr::VROverlay()->GetOverlayTransformAbsolute(f, &o, &m) != vr::VROverlayError_None) return false;
|
||||
const Mat l = Mul(Inverse(panel), m);
|
||||
const double x = l.m[0][3], y = l.m[1][3], z = l.m[2][3];
|
||||
return std::fabs(x) < w / 2 + 0.10 && y < -h / 2 + 0.05 && y > -h / 2 - 0.35 && std::fabs(z) < 0.15;
|
||||
}
|
||||
|
||||
// Near input's event on a SteamVR panel: the laser does it. False: not one of theirs.
|
||||
bool SteamEvent(const handinput::Event &e) {
|
||||
if (e.panel < kSteamIndex) return false;
|
||||
const SteamPanel *sp = FindSteam(e.panel);
|
||||
if (!sp || !sp->fit.ok) return true;
|
||||
SteamTouch &t = g_steamTouch[e.side];
|
||||
const steampanels::Fit &f = sp->fit;
|
||||
const char *hand = e.side ? "right" : "left";
|
||||
double u, v;
|
||||
switch (e.type) {
|
||||
case handinput::Event::Motion:
|
||||
case handinput::Event::Press:
|
||||
case handinput::Event::Release: {
|
||||
steampanels::FromPixels(f.width, f.height, e.x, e.y, &u, &v);
|
||||
const Mat l = steampanels::Touch(f.pose, u, v, kSteamBack);
|
||||
if (t.panel != e.panel) t = SteamTouch{}, t.panel = e.panel, t.key = sp->key;
|
||||
if (e.type == handinput::Event::Press) {
|
||||
SendTouch(e.side, "press", &l, sp->key);
|
||||
t.st = SteamTouch::St::Down;
|
||||
std::printf("SteamVR panel %s: pressed (%s hand)\n", sp->key.c_str(), hand);
|
||||
} else if (e.type == handinput::Event::Release) {
|
||||
SendTouch(e.side, "up", &l);
|
||||
t.st = SteamTouch::St::Hover;
|
||||
} else {
|
||||
SendTouch(e.side, t.st == SteamTouch::St::Down ? "move" : "hover", &l);
|
||||
if (t.st == SteamTouch::St::None) t.st = SteamTouch::St::Hover;
|
||||
}
|
||||
t.leftMs = 0;
|
||||
break;
|
||||
}
|
||||
case handinput::Event::GrabStart: {
|
||||
t = SteamTouch{};
|
||||
t.panel = e.panel, t.key = sp->key, t.panel0 = f.pose, t.w0 = f.width, t.h0 = f.height;
|
||||
t.resize = e.zone == handinput::Zone::Corner || e.zone == handinput::Zone::Tab;
|
||||
t.sx = e.sx, t.sy = e.sy;
|
||||
// first a hover just inside the edge (the corner): a floating panel's window controls come up
|
||||
u = t.resize ? t.sx * std::max(0.0, f.width / 2 - 0.02) : 0;
|
||||
v = t.resize ? t.sy * std::max(0.0, f.height / 2 - 0.02) : -std::max(0.0, f.height / 2 - 0.02);
|
||||
const Mat l = steampanels::Touch(f.pose, u, v, kSteamBack);
|
||||
SendTouch(e.side, "hover", &l);
|
||||
t.st = SteamTouch::St::GrabWait, t.sinceMs = NowMs();
|
||||
break;
|
||||
}
|
||||
case handinput::Event::GrabEnd:
|
||||
if (t.panel == e.panel && t.st == SteamTouch::St::Grab) {
|
||||
SendTouch(e.side, "up", nullptr);
|
||||
std::printf("SteamVR panel %s: let go (%s hand)\n", t.key.c_str(), hand);
|
||||
}
|
||||
if (t.panel == e.panel) t.st = SteamTouch::St::Hover, t.leftMs = 0;
|
||||
break;
|
||||
default: break; // (no throws, flicks, pushes, rewinds or scrolls on them: Panel::foreign)
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// Hand input taken away (paused, off): every laser touch ends.
|
||||
void EndSteamTouches() {
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (g_steamTouch[k].st != SteamTouch::St::None) {
|
||||
if (g_steamTouch[k].st == SteamTouch::St::Down || g_steamTouch[k].st == SteamTouch::St::Grab) SendTouch(k, "up", nullptr);
|
||||
SendTouch(k, "end", nullptr);
|
||||
g_steamTouch[k] = SteamTouch{};
|
||||
}
|
||||
}
|
||||
|
||||
// Each tick: a grab's hover, then its press; carries; and a hand that has left lets the laser go.
|
||||
void UpdateSteamTouches(const std::vector<handinput::Carry> &carries) {
|
||||
if (!HandInputOn()) return EndSteamTouches();
|
||||
const int64_t now = NowMs();
|
||||
for (int k = 0; k < 2; ++k) {
|
||||
SteamTouch &t = g_steamTouch[k];
|
||||
const char *hand = k ? "right" : "left";
|
||||
if (t.st == SteamTouch::St::None) continue;
|
||||
if (t.st == SteamTouch::St::GrabWait) {
|
||||
if (g_handInput.grabPanel(k) != t.panel) {
|
||||
t.st = SteamTouch::St::Hover;
|
||||
} else if (now - t.sinceMs >= 150 && FooterUnder(t.panel0, t.w0, t.h0)) {
|
||||
double u, v;
|
||||
steampanels::GrabPoint(t.w0, t.h0, t.resize, t.sx, t.sy, kSteamBar, kSteamMargin, &u, &v);
|
||||
t.laser0 = steampanels::Touch(t.panel0, u, v, kSteamBack);
|
||||
SendTouch(k, "grab", &t.laser0);
|
||||
t.st = SteamTouch::St::Grab;
|
||||
std::printf("SteamVR panel %s: %s by the %s hand\n", t.key.c_str(), t.resize ? "resized" : "moved", hand);
|
||||
} else if (now - t.sinceMs >= 600) {
|
||||
t.st = SteamTouch::St::Refused;
|
||||
std::printf("SteamVR panel %s: no window controls under it (not a floating panel): the %s hand can't %s it\n",
|
||||
t.key.c_str(), hand, t.resize ? "resize" : "move");
|
||||
}
|
||||
}
|
||||
if (t.st == SteamTouch::St::Down && !g_handInput.pressing(k)) { // (a press that ended without its release)
|
||||
SendTouch(k, "up", nullptr);
|
||||
t.st = SteamTouch::St::Hover;
|
||||
}
|
||||
if (t.st == SteamTouch::St::Refused && g_handInput.grabPanel(k) != t.panel) t.st = SteamTouch::St::Hover;
|
||||
if (t.st == SteamTouch::St::Hover) {
|
||||
if (g_handInput.nearPanel(k) == t.panel) {
|
||||
t.leftMs = 0;
|
||||
} else if (!t.leftMs) {
|
||||
t.leftMs = now;
|
||||
} else if (now - t.leftMs >= 300) {
|
||||
SendTouch(k, "end", nullptr);
|
||||
t = SteamTouch{};
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const handinput::Carry &c : carries) {
|
||||
if (c.panel < kSteamIndex) continue;
|
||||
for (int k = 0; k < 2; ++k) {
|
||||
SteamTouch &t = g_steamTouch[k];
|
||||
if (t.panel != c.panel || t.st != SteamTouch::St::Grab) continue;
|
||||
Mat l;
|
||||
if (c.kind == handinput::Carry::Resize) {
|
||||
const double w = c.width > 0 ? c.width : t.w0, h = w * t.h0 / std::max(1e-6, t.w0);
|
||||
double u, v;
|
||||
steampanels::GrabPoint(w, h, true, t.sx, t.sy, 0, kSteamMargin, &u, &v);
|
||||
l = steampanels::Touch(c.pose, u, v, kSteamBack);
|
||||
} else {
|
||||
l = steampanels::Follow(t.panel0, c.pose, t.laser0);
|
||||
}
|
||||
SendTouch(k, "move", &l);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The palm menu (palmmenu.h): before near input, which leaves it the hand it has. Its ring is
|
||||
// a guide ("frametop.guide.palm.*"), so the pointer helper's cursor and rays pass through it.
|
||||
palmmenu::PalmMenu g_palm;
|
||||
@@ -3440,8 +3759,11 @@ void UpdateHandGuides() {
|
||||
auto hide = [&] {
|
||||
if (hg.shown) vr::VROverlay()->HideOverlay(hg.overlay), hg.shown = false;
|
||||
};
|
||||
// on one of our screens, or on one of SteamVR's panels (see "SteamVR's panels by hand")
|
||||
const auto it = g_screens.find(hg.g.panel);
|
||||
if (hg.alpha < 0.01 || hg.g.kind == handinput::Guide::None || it == g_screens.end() || !it->second.visible) {
|
||||
const SteamPanel *sp = it == g_screens.end() ? FindSteam(hg.g.panel) : nullptr;
|
||||
const bool there = it != g_screens.end() ? it->second.visible : sp && sp->visible && sp->fit.ok;
|
||||
if (hg.alpha < 0.01 || hg.g.kind == handinput::Guide::None || !there) {
|
||||
hide();
|
||||
continue;
|
||||
}
|
||||
@@ -3450,13 +3772,13 @@ void UpdateHandGuides() {
|
||||
if (vr::VROverlay()->CreateOverlay(key, "Frametop: hand guide", &hg.overlay) != vr::VROverlayError_None) continue;
|
||||
vr::VROverlay()->SetOverlaySortOrder(hg.overlay, 1);
|
||||
}
|
||||
Screen &s = it->second;
|
||||
Mat p;
|
||||
if (!ScreenPose(s, &p)) {
|
||||
Screen *s = sp ? nullptr : &it->second;
|
||||
Mat p = sp ? sp->fit.pose : Identity();
|
||||
if (s && !ScreenPose(*s, &p)) {
|
||||
hide();
|
||||
continue;
|
||||
}
|
||||
const double W = s.metres, H = s.heightMetres();
|
||||
const double W = s ? s->metres : sp->fit.width, H = s ? s->heightMetres() : sp->fit.height, curve = s ? s->curve : 0;
|
||||
double halo = std::round(softglow::Width(HeadDistance(p)) * 1000) / 1000; // (by the mm: fewer textures)
|
||||
// its size and place on the panel
|
||||
double u = hg.g.u, v = hg.g.v, wm = 0, hm = 0, arm = 0, r = 0, edge = 1e9;
|
||||
@@ -3464,7 +3786,7 @@ void UpdateHandGuides() {
|
||||
bool curved = false;
|
||||
switch (hg.g.kind) {
|
||||
case handinput::Guide::Corner:
|
||||
if (k == 2) halo = std::max(halo, HandleSize(s)); // (the laser's: as wide as the handle it presses)
|
||||
if (k == 2 && s) halo = std::max(halo, HandleSize(*s)); // (the laser's: as wide as the handle it presses)
|
||||
arm = std::clamp(0.25 * std::min(W, H), 0.03, 0.10);
|
||||
u = hg.g.sx * W / 2, v = hg.g.sy * H / 2, wm = hm = 2 * std::max(arm, halo) * 1.05, curved = true;
|
||||
break;
|
||||
@@ -3506,8 +3828,10 @@ void UpdateHandGuides() {
|
||||
hg.key = key;
|
||||
}
|
||||
vr::VROverlay()->SetOverlayWidthInMeters(hg.overlay, float(wm));
|
||||
vr::VROverlay()->SetOverlayCurvature(hg.overlay, curved && s.curve > 0 ? float(std::clamp(wm / (2 * M_PI * s.curve), 0.0, 1.0)) : 0.f);
|
||||
const Mat m = Mul(p, OnSurface(s, u, v, hg.g.kind == handinput::Guide::Touch ? 0.003 : 0)); // (the dot just over the content)
|
||||
vr::VROverlay()->SetOverlayCurvature(hg.overlay, curved && curve > 0 ? float(std::clamp(wm / (2 * M_PI * curve), 0.0, 1.0)) : 0.f);
|
||||
// (the dot just over the content; on SteamVR's panels all of it a little out, over theirs)
|
||||
const double out = hg.g.kind == handinput::Guide::Touch ? 0.003 : 0;
|
||||
const Mat m = Mul(p, s ? OnSurface(*s, u, v, out) : Translation(u, v, out + 0.003));
|
||||
vr::VROverlay()->SetOverlayTransformAbsolute(hg.overlay, vr::TrackingUniverseStanding, &m);
|
||||
if (std::fabs(hg.alpha - hg.shownAlpha) > 0.01) {
|
||||
vr::VROverlay()->SetOverlayAlpha(hg.overlay, float(hg.alpha));
|
||||
@@ -3581,6 +3905,11 @@ void UpdateHandInput(void (*handle)(const struct ft_event *, void *), void *data
|
||||
}
|
||||
for (int k = 0; k < 2; ++k) in.typing[k] = NowMs() - g_typingWord.atMs < 3000 && g_typingWord.side[k];
|
||||
in.pointerBusy = g_press.buttons && !IsHandHolder(g_press.device);
|
||||
// SteamVR's own panels near the hands (see "SteamVR's panels by hand")
|
||||
if (HandInputOn()) {
|
||||
ScanSteamPanels(in);
|
||||
SteamInputPanels(panels);
|
||||
}
|
||||
// KWin's button, let go of by someone else (ReleaseStuck: its screen hid, or paused)
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (g_handInput.pressingKwin(k) && !(g_press.buttons && g_press.device == kHandHolder[k])) g_handInput.ForgetPress(k);
|
||||
@@ -3605,6 +3934,7 @@ void UpdateHandInput(void (*handle)(const struct ft_event *, void *), void *data
|
||||
std::printf("lazy susan by hand (%s, swept %s): %s\n", e.side ? "right" : "left", e.x < 0 ? "left" : "right", reply);
|
||||
continue;
|
||||
}
|
||||
if (SteamEvent(e)) continue; // one of SteamVR's panels: the pointer helper's laser does it
|
||||
const auto it = g_screens.find(e.panel);
|
||||
if (it == g_screens.end()) continue;
|
||||
Screen &s = it->second;
|
||||
@@ -3727,6 +4057,7 @@ void UpdateHandInput(void (*handle)(const struct ft_event *, void *), void *data
|
||||
}
|
||||
SetAbsolute(s, c.pose);
|
||||
}
|
||||
UpdateSteamTouches(g_handInput.carries());
|
||||
const auto &hovered = g_handInput.hovered();
|
||||
for (auto &[i, s] : g_screens) {
|
||||
const auto h = hovered.find(i);
|
||||
@@ -4808,6 +5139,9 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (g_handsWord.hand[k]) g_handsWord.handMs[k] = NowMs();
|
||||
std::snprintf(reply, size, "ok");
|
||||
} else if (!std::strncmp(cmd, "steamvr ", 8)) { // the helper's overlay list (see "SteamVR's panels by hand")
|
||||
SteamListPart(cmd);
|
||||
std::snprintf(reply, size, "ok");
|
||||
} else if (int tl = 0, tr = 0; std::sscanf(cmd, "typing %d %d", &tl, &tr) == 2) {
|
||||
g_typingWord.side[0] = tl != 0, g_typingWord.side[1] = tr != 0, g_typingWord.atMs = NowMs();
|
||||
std::snprintf(reply, size, "ok");
|
||||
|
||||
@@ -7,4 +7,4 @@ set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
dumps=$(printf ' %q' "$@")
|
||||
"$root/scripts/frame.sh" -C screens "mkdir -p build && g++ -std=c++17 -O2 -Wall -Wextra -Wno-unused-parameter -Werror \
|
||||
tests/handinput_test.cpp handcut.cpp handinput.cpp palmmenu.cpp grabthat.cpp -o build/handinput-test && build/handinput-test${dumps:+ && build/handinput-test --replay$dumps}"
|
||||
tests/handinput_test.cpp handcut.cpp handinput.cpp palmmenu.cpp grabthat.cpp steampanels.cpp -o build/handinput-test && build/handinput-test${dumps:+ && build/handinput-test --replay$dumps}"
|
||||
Reference in new issue
Block a user