When a fork occurs FEX needs to be incredibly careful as any thread
(that isn't forking) that holds a lock will vanish when the fork occurs.
At this point if the newly forked process tries to use these mutexes
then the process hangs indefinitely.
The three major mutexes that need to be held during a fork:
- Code Invalidation mutex
- This is the highest priority and causes us to hang frequently.
- This is highly likely to occur when one thread is loading shared
libraries and another thread is forking.
- Happens frequently with Wine and steam.
- VMA tracking mutex
- This one happens when one thread is allocating memory while a fork
occurs.
- This closely relates to the code invalidation mutex, just happens at
the syscall layer instead of the FEXCore layer.
- Happens as frequently as the code invalidation mutex.
- Allocation mutex
- This mutex is used for FEX's 64-bit Allocator, this happens when FEX
is allocating memory on one thread and a fork occurs.
- Fairly infrequent because jemalloc doesn't allocate VMA regions that
often.
While this likely doesn't hit all of the FEX mutexes, this hits the ones
that are burning fires and are happening frequently.
- FEXCore: Adds forkable mutex/locks
Necessary since we have a few locations in FEX that need to be locked
before and after a fork.
When a fork occurs the locks must be locked prior to the fork. Then
afterwards they either need to unlock or be set to default
initialization state.
- Parent
- Does an unlock
- Child
- Sets the lock to default initialization state
- This is because it pthreads does TID based ownership checking on
unique locks and refcount based waiting for shared locks.
- No way to "unlock" after fork in this case other than default
initializing.
Every time we are calling a function in `FEXCore::Allocator::` this is a
pointer indirection. Which means on x86 it is always a `call [rdi]` and
on AArch64 it is a `ldr x17, [x0]; blr x17;`.
Instead of doing this, use inline functions in the header that call the
correct allocation function directly. This function gets inlined and is
no longer an indirect call.
When compiling with jemalloc, we forward declare the jemalloc function
definitions so we don't have to pull in the entire jemalloc interface in
to the public header definitions.
This is done by consuming a single page at the end of the current sbrk
memory region. Then consuming any remaining bytes that could have
potentially ended up in it.
This ensures that glibc won't be able to return 64-bit pointers to
32-bit thunks once the remaining work is in place.
In the case of an AArch64 builder is using 16kb or 64kb pages like is
common on servers then it would fail to compile, even if the resulting
application would only ever run on 4k page hosts.
Resolve this by removing the build check and hardcoding 4kb pages for
each of our uses. We still require 4kb pages to run, so this mostly just
removes the weirdness where it is 16kb builder + 4k runner. Would have
broken some of our assumptions when running.
Adds a header only include utility folder that can be included from
everywhere.
Contains syscall helpers for older glibc and defines for older Linux
uapi headers missing some defines.
Only a partial fix for #1330, still needs preemption disabled to work.
On x86-64 hosts the Linux kernel resides in the top bit of VA which
isn't mapped in to userspace.
This means that userspace will never receive pointers living with that
top bit set unless you're running a 57bit VA host.
This results in userspace pointers never needing to do the sign
extending pointer canonicalization. But additionally some applications
actually don't understand the pointer canonicalization.
This results in bugs like: https://github.com/golang/go/issues/49405
Now if you're running on a 57bit VA host, this will end up behaving like
FEX but it seems like no one in golang land has really messed with 57bit
VA yet.
In AArch64, when configured with a 48bit VA, the userspace gets the full
48bit VA space and on EL mode switch has the full address range change
to the kernel's 48bit VA.
This means that we will /very/ likely allocate pointers in the high
48bit space since Linux currently allocates top-down.
So behave more like x86-64, hide the top 128TB of memory space from the
guest before boot.
Testing: Took the M1Max 15ms to 21ms allocate the top 128TB.
This isn't quite a 100% clean sweep of IWYU.
There are some false positives where clang fails.
Additionally there are still a few missed in the frontend side of things
that I didn't get to
When we are running a 32-bit process we end up mixing VMA Region allocators which
can cause crashes on shutdown.
This is because if a statically initialized object allocates memory, and then frees that
memory in the atexit handler. There is a chance that if it had to reallocate memory during the
VMA allocator switch, that the atexit handler will try freeing memory from the FEX VMA region allocator
AFTER it has already been deallocated.
This can't be safely worked around with atexit handlers.
So until we resolve this issue, we HAVE to leak the allocator so it can safely clean up and then let the kernel
clean up after us
Due to a disjoint mechanism inside of glibc we need to override both the glibc
publicly visible allocation functions AND the hooks.
We had broken the overriding when we changed the default visibility. So first fix that.
Then only override to FEX's allocators once we are ready for it.
Then also replace the hooks.
Additionally have a way to clear the hooks back to default.
Ryan Houdek