Instead of keeping the vlaue as a string array in the MetaLayer, convert
the value to its final type once.
Improves performance in some hotpaths that were doing config based
string conversion in a relatively high frequency.
`Extract` was never set to false even if the user said not to overwrite.
Ensure that if the user decided not to overwrite an existing folder,
that it returns false so it correctly doesn't overwrite the folder.
Removes a global initializer and atexit registration
Ownership of this data has always been the frontend and the config
system, we just used these static vectors as a side-channel.
Fixes#3691
We weren't checking if the file was empty before using its `at` function
member. This was causing an early crash if the config file existed but
was empty.
Consolidates the three locations that copy and pasted the json allocator
tools and adds an empty check for all of them.
Also adds two missing checks to the ThunksDB handler that could have
resulted in the same crash if ThunksDB was an empty file.
Since we're invoking curl directly, we don't need to wrap it in `sh -c`
with this function.
Fixes an issue where curl downloads to a non-escaped path weren't
working. Now they do.
As I was poking around erofs-utils documentation, I found out that
fsck.erofs actually provides an option for extracting erofs images
without using fuse.
This finally puts the erofs handling on feature parity with squashfs.
FEXCore has no need to understand how to load these layers. Which
requires json parsing.
Move these to the frontend which is already doing the configuration
layer setup and initialization tasks anyway.
Means FEXCore itself no longer needs to link to tiny-json which can be
left to the frontend.
This basically just means that we detect ArchLinux and set a flag that
it is a rolling release, skipping doing the version check for an "exact"
match in that instance.
Switches to the new CDN which is significantly faster and has other
benefits.
In order to make sure we don't break old clients, switch to the new link
for a few months while leaving the old one operational.
The links file in the old CDN still points to the new rootfs links so
they get the performance improvement on old clients still.
Fixes#2356
In the case of the `-y` option being used, it will auto say "yes", but
when presented with the distro list this doesn't work. This happens when
used on a distro that doesn't have an exact match to what we provide.
Exposes a new option that when presented the distro list, auto select
the first option. Solving this issue when automating.
Fixes#2136
This is a fairly tricky edge case to support with FEX.
If execveat is used with AT_EMPTY_PATH then the application can pass an
FD to execve instead of a filename. This includes FDs that have been
deleted from the disk so the child process can't open it by filename
anymore.
To work around this limitation, we need to pass the FD to the new FEX
process and open it directly, similar to how binfmt_misc works with FDs.
The FD will get passed through environment variables, which the new
process will check for and then remove the variable from the
environment.
Lots of prickly edge cases to support here.
Without binfmt_misc:
- Passes the FD to FEXLoader directly.
- Requires duplicating the FD if it has O_CLOEXEC on the FD.
With binfmt_misc:
- Shebang file, pass directly to FEXLoader, just like without binfmt.
- x86 ELF Files, rely on the kernel's binfmt_misc support here.
- Unsupported ELF files, let kernel handle it through binfmt_misc
Argument handling:
- The application can pass in no arguments.
- Means our application configurations were failing to find a config
- Also various checks in the frontend were failing.
- If opened through an FD, find the symlink for that FD for the
application configuration instead.
Side note:
Fixed a performance issue in execve where when we were checking for file
format support. Either ELF or Shebang files, we were reading the /whole/
file upfront. We only need to read a header worth of ELF files, and only
257 bytes if it is potentially a shebang file. Should dramatically
reduce some application's execve times.
If the user doesn't have any of the tools necessary for handling FEX's
images then the tool would spuriously fail with `Couldn't parse rootfs definition URL.`
With zero indication as to why we removed images from the parsed json.
If the user has at least one of these tools installed then they won't
get this error message.
Main options here are `-y` and `-a`.
Passing both will allow the user to download the exact match image, use
it as-is, and set the config to use the rootfs by default.
Additional options `--distro-name` and `--distro-option` allows one to
select a particular distro version.
eg: `FEXRootFSFetcher -y -a --distro-name=ubuntu --distro-version=22.04`
* Will download the first Ubuntu 22.04 compressed in the json list
** Priority of erofs or squashfs depends on order in json and what the
host supports
Fixes a bug in `ExecAndWaitForResponse` where results > 1024 bytes would
overwrite data.
Switches from a custom format txt file to a json file.
JSON file now has a "Type" field to specify squashfs versus erofs.
JSON is now versioned so we don't need to move the file around, just
append to a new versioned segment.
Only shows erofs files if you have the bleeding edge `erofsfuse`
application.
This application was available starting with erofs-utils v1.5 which was
released on 2022-06-13, so it isn't available pretty much everywhere.
The final tail of the file reading was incorrect, so our hashing was
"correct" but it was using stale data from the previous block size read.
Noticed this while wiring up the CI rootfs fetching since the hashing is
a lot simpler there.
Now instead of reading a tail, just attempt to read the full block size
and use the resulting data size instead. Confirmed it matches expected
results now.
In the process we are going to need to update hyperlinks and hashes
anyway, change the hash to XXH3 so it is faster to run.
Fixes a regression in fa87c73b9ee60a334eace2cdc3097725cbaf5b88; curl complains "curl: option -: is unknown" when trying to fetch a RootFS without this.
Makes curl do its continue feature to give the users the best chance of
downloading a rootfs. We don't need to restart the full file transfer on
failure. Helps people with slower connections.
On failure to download, asks the user if they want to retry the download
rather than just exiting with a weird error about hash failure.
Once the image is downloaded, now changes options depending on if
squashfuse or unsquashfs works.
Prevents the user from selecting a bad option and getting unexpected
behaviour. Ideally we would do a squashfs mount test as well for
platforms that don't have working FUSE, like termux. This is harder to
get right and its for an unsupported platform, so I'm not going to
invest more time with it.
Fixes#1525Fixes#1526Fixes#1527
Location to check if curl, squashfuse, and unsquashfs are working.
unsquashfs is a bit more complex where it needs to parse the help output
to see if zstd is supported
In the case of launching without stdout/stderr then redirection could
have these constants be a redirected FD that sits in the same fd number.
Use -2 to indicate no redirection.
Use -1 to indicate closing traditional stderr/stdout
The rest will indicate if stdout and stderr should be replaced as
normal.
Making sure not to close the incoming fds if they matched the
stdout/stderr FD numbers.
tpietrus