Need to ensure that dirfd is AT_FDCWD and also need to check flags
correctly.
Flags were incorrectly checking mode for O_WRONLY and also we should
check for O_APPEND. Split it out to a helper function just so it is
easier to see what is going on.
Fixes the issue of proton not finding `/lib64/ld-linux-x86-64.so.2`
Fixes#2443
I found out with some profiling that this we were spending a decent
amount of time with the `openat` syscall in heavily utilized situations.
While not super common in active gameplay situations, it matters
significantly in loading screens that this is fairly optimal.
The bulk of the time is spent in the emulated files handler to ensure
that whatever path we are given, we can capture file paths that we need
to emulate. The largest contributor being the std::filesystem::canonical
function call.
A couple of optimizations in place here.
1) Do a quick hashmap check right at the start to see if we exactly fit
2) Change from `std::fs::canonical` to `realpath`
3) Switch `GetEmulatedFDPath` to not use optional so it stops building
on the stack
I'm still not super happy with the performance of `realpath` and also
not happy that we still need to use `lexically_normal` in one code path.
But short of writing a super hand-optimized `realpath` that fits our
constraints, I don't think we can do better.
Micro benchmark needs to test four different situations due to this
optimization.
1) Non-EmuFD path
2) Non-EmuFD path with dirfs
3) EmuFD path
4) EmuFD path with dirfs
And the performance improvement for each situation respectively
1) 12% performance improvement
- 213413 openat syscalls/s -> 238999 syscalls/s
2) 17% performance improvement
- 202085 openat syscalls/s -> 237309 syscalls/s
3) 17% performance improvement (/proc/cpuinfo)
- 56616 openat syscalls/s -> 66231 syscalls/s
- Includes overhead of generating temp FD and close syscall
4) 5% performance improvement (/proc/cpuinfo)
- 51080 openat syscalls/s -> 53956 syscalls/s
- Includes overhead of generating temp FD and close syscall
And for sake of comparison to the non-emulated system; My test system
can hit around 1-1.1 million openat syscalls per second in the same
microbench.
Nice little performance uplift.
Performance stats up front:
This improves pressure-vessel startup time on my test device by 10.1%
Improving the startup time from 9.71425 seconds to 8.7421 seconds.
Most filesystem based syscalls support a file descriptor version with an
*at suffix. This allows us to do these syscalls with pathnames that are
relative to the directory FD that is passed to the syscall.
This is pretty much exactly what we want when we are searching for files
inside of our rootfs. The only quirk ends up being that we are getting
passed absolute paths. This ends up being very simple to workaround by
stripping off the front '/' character. Doing this is just offsetting the
pointer passed to the syscall by one byte.
This does require having two temporary buffers of size PATH_MAX passed
to the handler since just like in the other implementation, we need to
keep the previous result around. The difference being now that we aren't
doing a bunch of std::string temporary manipulation and now we are
returning one of the passed in buffers back depending on the result.
This has been a long time coming. The C interface has been a thorn in
our side for no reason for a long time.
The purpose of this step is to remove the C interface without changing
behaviour as much as possible. This means that with this commit there
are still some bad practices but the remaining issues will be solved
with followup PRs.
Primarily, we still have a `DestroyContext(CTX)` static function which calls
the Context implementation's `DestroyContext` and does a raw C++ delete.
Follow up PR will remove that, but I didn't want to touch it yet since
it'll require checking to ensure the unique_ptr changes play nice with
our allocator hooking. Which this is already a huge PR without trying to
change behaviour.
`GetEmulatedPath` and `OpenAt` are called a /lot/ in applications.
std::filesystem::path handling here is quite heavy and costly for what
we are trying to achieve.
Remove this usage and instead use lstat, access, and readlink directly
which is a heck of a lot faster.
In particular this helps out pressure-vessel, shaving off launch times
by 1-2 seconds.
Going from ~22 seconds down to ~20 seconds.
Previously in order to enable thunks, we needed an independent
description file of which thunks to be enabled. This is nice for quickly
testing out new games by setting `FEX_THUNKCONFIG` environment variable.
For users that just want to enable thunks this is an unwieldy
indirection that doesn't make much sense at a glance.
Previously this meant you needed two files as an example:
```
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat thunks.json
{
"ThunksDB": {
"GL": 1,
"Vulkan": 1
}
}
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
{
"Config": {
"ThunkConfig":"~\/.fex-emu\/thunks.json"
}
}
```
Instead of this unwieldy redirection just support `ThunksDB` json
directly in the AppConfig.
```
ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
{
"Config": {
<...>
},
"ThunksDB": {
"GL": 1,
"Vulkan": 1
}
}
```
As can be seen this makes this significantly easier for new users
getting in to thunks. Depending on which path to enable thunks the user
is more comfortable with, they can still enable them using the
`ThunkConfig` option or embedding directly in the application
configuration.
Additionally this removes the older non-ThunksDB path to loading thunks.
All users of it have moved on to using ThunksDB.
It's common for Linux applications that use close_range to pass in ~0 as
the last FD. This was causing FEX to spin from [2, ~0U] in this loop.
This would take /forever/ to run.
Change over to an ordered map and use the map's range searching and
ranged erase to more quickly remove these elements.
Fixes a hang that occurs with first time Steam setup in
steam-linux-runtime heavy application `steam-runtime-identify-library-abi`.
Ubuntu soft links a bunch of binaries in /usr/bin to softlinks that live
in /etc/alternatives/
When hitting any of these alternative softlinks execve would fail if the
host also didn't have the same softlink paths.
Allows us to correctly follow the symlinks on execve as well which fixes
launching wine directly from the wine symlink.
Alternatively you could have launched /usr/bin/wine-stable directly.
Also fixes FEX strace again.
This adds a new `ThunksDB.json` file to the config folder.
This file lets users describe thunks in a meaningful way without
duplicating it amongst multiple configuration files.
eg:
```
{
"DB": {
"GL": {
"Library" : "libGL-guest.so",
"Depends": [
"X11"
],
"Overlay": [
"/usr/lib/x86_64-linux-gnu/libGL.so",
"/usr/lib/x86_64-linux-gnu/libGL.so.1",
"/usr/lib/x86_64-linux-gnu/libGL.so.1.2.0",
"/usr/lib/x86_64-linux-gnu/libGL.so.1.7.0",
"/lib/x86_64-linux-gnu/libGL.so",
"/lib/x86_64-linux-gnu/libGL.so.1",
"/lib/x86_64-linux-gnu/libGL.so.1.2.0",
"/lib/x86_64-linux-gnu/libGL.so.1.7.0"
]
},
"X11": {
"Library": "libX11-guest.so",
"Overlay": [
"/usr/lib/x86_64-linux-gnu/libX11.so.6",
"/lib/x86_64-linux-gnu/libX11.so.6"
]
}
}
}
```
This file lets the user describe the library with an nicer name, in this instance
`GL` instead of `libGL-guest.so`.
It also tracks depedencies, like how GL currently has a hard dependency on X11.
This allows the loader to automatically enable the dependencies if described.
The `Overlays` array is like the regular Thunks config file but now in this DB file.
With the DB file now describing the libraries, this allows us to then stick a lighter
description inside of the Thunk Config file.
```
{
"ThunksDB": {
"GL": 1
}
}
```
With this example Thunk config file (Which can be configured per application), There is
a new property of name `ThunksDB`.
All this takes is key:value pairs which describe the user friendly library name and an Integer
to state if the thunk should be enabled or not.
This allows very quick toggling of thunks directly inside of the configuration files rather than
breaking the configuration to disable it.
This isn't quite a 100% clean sweep of IWYU.
There are some false positives where clang fails.
Additionally there are still a few missed in the frontend side of things
that I didn't get to
Allows lookups with const char* and other non-allocating string types to
be performed without constructing a std::string instance.
Reduces heap usage a little.
This fixes an issue where rootfs has symlinks to other things in the rootfs so we need to track it through.
Relies on #1009 to be merged first.
Fixes any application that relies on libblas, mpv for example.
Configuration mapping was duplicated between three different tables.
Additionally default configuration values were strewn about. Making it confusing as to what the default value would end up being
Adds a new ConfigValues.inl header that defines a few things right next to each other.
Defines the enum name as usual.
Defines the JSON config option name.
Defines the Environment config option name
Defines the default value that the configuration should be
Sadly these things can't be split without breaking functionality so it
turns in to a bit of a mess.
SyscallHandler is very much something that is a Linux only construct and
shouldn't be in FEXCore itself. Lets the frontend register a
Syscallhandler with FEXCore. FEXCore itself is then aware of the current
syscall ABI and handles the ABI in an optimal fashion.
So it is not a 100% clean break otherwise we would lose performance.
The SignalDelegator then needs to move to the frontend since the
SyscallHandler requires it for signal based syscalls.
The CPU backend signal handling still needs to happen in FEXCore because
it is a very tight coupling with the CPU backend.
Once we need to support more Signal handling we can give the backends
cleaner support to select which specific OS handler to handle.
Ryan Houdek