1) The host library needs to be loaded in the global namespace.
2) We need to use `RTLD_DEFAULT` instead of querying the object
directly.
We need to load the host library in the global namespace so the symbols
end up in the global symbol table. This follows how all these symbols
/usually/ get loaded. Either by linking directly to the library or how
loaders will end up loading these.
We need to use RTLD_DEFAULT to follow symbol overriding rules that tend
to occur. For example, MangoHUD will LD_PRELOAD a library that provides
GLX and EGL symbols. Which FEX's thunk libraries need to pick up this
override.
If we are querying the host library directly then we fail to pickup
these overrides, thus breaking MangoHUD and other overlays.
Due to how we use a modified ABI for these indirect functions, we don't
have a clean way to say that the host_addr lives in a side-argument.
The previous inline asm that moved the value from r11 in to a variable
worked up until you hit functions with 8 or more arguments. At that
point the compiler was generating code before our inline assembly and
using r11 as a temporary, thus destroying our value.
Then a crash would occur and it was very hard to determine why. It would
end up calling some random function (0x1 in this case) from an indirect
call.
This made it /look/ like it was calling an invalid function returned
from the loader but in reality it was a corrupt register loading bad
data.
To work around this case, we can use an inline asm register variable and
a volatile asm block that "sets" the variable. In this case GCC and
Clang both seem to extend the live range of the register from the start
of the function to the use of the variable.
This resolves the issue for now, and I tested quite a large number of
function signatures to see if it would break in the future.
Theoretically our functional testing should catch this, but we don't
currently have something that abuses all the functions like this
currently.
Fairly straightforward, just requires enabling lld in this case since
cross-compiling doesn't work well with gnu linker.
Also lld doesn't understand the linker script program header symbolic
names for read/write/execute. So we need to use the raw number there.
Works around an issue where GCC 11 generates broken `init_array` section
and also plt sections that glibc doesn't understand.
Use the fastcall ABI for 32-bit x86 to make our lives easier.
Fastcall ABI puts the first two 32-bit arguments in ECX and EDX
respectively.
Compilers are nice today and allow us to do cross-abi function calls
like this.
This avoids the need to provide a fallback definition for platform-specific
macros. The definitions are only added host-side, since only Host.h is
included in any interface files.
This changes how host trampolines for guest functions are created. Instead
of doing this purely on the guest-side, it's either the host-side that
creates them in a single step *or* a cooperative two-step initialization
process must be used. In the latter, trampolines are allocated and partially
initialized on the guest and must be finalized on the host before use.
This function must be able to handle both guest heap pointers *and* host heap
pointers, so it only forwards to the native host library for the latter.
This is because Xlibint users allocate memory using internal macros aliasing
to libc's malloc but then they free using the function XFree. For libX11,
this is not a problem since the allocation happens in a thunked API function
(and hence on the host heap), but if a function from an unthunked library
accesses Xlibint, it will allocate on the guest heap.
One notable example where this was encountered is XF86VidModeGetAllModeLines.
Interface definitions must enable this functionality by enclosing functions
that may be called through function pointers in a namespace annotated with
`fexgen::indirect_guest_calls`. The guest thunk must further link any host
function pointers to a guest-side instance of CallHostThunkFromRuntimePointer.
Previously, defining zero-initialized variables right before LOAD_LIB
could cause the compiler to put thunk definitions into bss, hence triggering
errors during assembly ("attempt to store non-zero value in section `.bss'").
Host thunk libraries are always built as part of the main project now.
Guest thunk libraries are still cross-compiled in a CMake ExternalProject,
but *additionally* there are CMake targets in the main project to make
sure IDE engines can properly handle guest source files.
Function pointer arguments given to the guest thunk library aren't
callable on the host, so they need special handling on a case-by-case
basis. Similar problems arise when the guest-side tries to consume a
pointer returned from a native host library. To automate some common
scenarios, this change adds two new annotations:
"callback_guest" indicates the callback parameter is never called on the
host and hence can be marshalled like any other argument. Accidental host
calls to the function pointer are prevented by wrapping it in an opaque
type alias.
"returns_guest_pointer" indicates that the host function returns a pointer
usable in the guest context. This applies e.g. to functions that derive
the returned pointer from input guest pointer arguments.
Previously, the host thunk would seemingly succeed to load even when
the corresponding host library could not be loaded, leading to obscure
crashes later throughout execution.
Keeping this in the commit history to go back to later.
While these are required for static-pie builds to work, with the glibc
bug we can't use those yet.
Disable for now since it is is unnecessary.
Ryan Houdek