FEX: Allocate a VMA allocator when running on a 48-bit VA

When running on a system with a 48-bit VA, if FEX does any allocations
between us reserving the upper 128TB and the application running, then
/technically/ we are intersecting with the application's memory region
in the lower 47-bits.

This didn't typically result in any problems due to how ASLR works, but
if we did any large allocations (like #4291 wants with 128MB VMA region)
then these typically get pushed higher in the VA space.

Again not usually a problem, but if you happen to be running an
application that is using MAP_FIXED with hardcoded addresses then this
can stomp over FEX-Emu memory causing problems.

This is what happens with Wine, it reserves the upper-32MB of its 47-bit
VA space, which is /highly/ likely to stomp on FEX memory. In-fact it
likely occurs all the time, we just got lucky with whatever it was
clobbering wasn't used at the time.

On 39-bit VA systems this isn't a problem because the mmap fails
outright with a warning message from WINE.

Because we are already reserving the upper 128TB of VA space, instead
just always enable our allocator and use the regions that were reserved.
We need to be a little bit careful to ensure we don't accidentally
allocate more memory post-reservation but that just requires a small
adjustment to our unique_ptr and constructor for the 64BitAllocator.

This means /all/ FEX-Emu allocations will be in the upper 128TB VA space
when running 64-bit applications on a 48-bit VA system. Which is kind of
nice.

Fixes WINE in #4291 when the allocator stats are bumped to 128MB per
process.
This commit is contained in:
Ryan Houdek committed 2025-01-28 16:07:51 -08:00
1 parent fb2a59a67f
commit c75778abeb
5 files changed
+113 -31

No files matched your search

+12 -5
View File
@@ -112,14 +112,18 @@ void ReenableSBRKAllocations(void* Ptr) {
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
void SetupHooks() {
Alloc64 = Alloc::OSAllocator::Create64BitAllocator();
static void AssignHookOverrides() {
SetJemallocMmapHook(FEX_mmap);
SetJemallocMunmapHook(FEX_munmap);
FEXCore::Allocator::mmap = FEX_mmap;
FEXCore::Allocator::munmap = FEX_munmap;
}
void SetupHooks() {
Alloc64 = Alloc::OSAllocator::Create64BitAllocator();
AssignHookOverrides();
}
void ClearHooks() {
SetJemallocMmapHook(::mmap);
SetJemallocMunmapHook(::munmap);
@@ -300,15 +304,18 @@ fextl::vector<MemoryRegion> StealMemoryRegion(uintptr_t Begin, uintptr_t End) {
return Regions;
}
fextl::vector<MemoryRegion> Steal48BitVA() {
void Setup48BitAllocatorIfExists() {
size_t Bits = FEXCore::Allocator::DetermineVASize();
if (Bits < 48) {
return {};
return;
}
uintptr_t Begin48BitVA = 0x0'8000'0000'0000ULL;
uintptr_t End48BitVA = 0x1'0000'0000'0000ULL;
return StealMemoryRegion(Begin48BitVA, End48BitVA);
auto Regions = StealMemoryRegion(Begin48BitVA, End48BitVA);
Alloc64 = Alloc::OSAllocator::Create64BitAllocatorWithRegions(Regions);
AssignHookOverrides();
}
void ReclaimMemoryRegion(const fextl::vector<MemoryRegion>& Regions) {