Compare commits

..
448 Commits
Author SHA1 Message Date
Ryan Houdek fd3e988a20 Docs: Update for release FEX-2211 2022-11-02 23:25:10 -07:00
Ryan Houdek b1d98f4e58 Merge pull request #2134 from lioncash/temp
Arm64/ConversionOps: Eliminate use of temporary in Vector_FToF
2022-11-02 19:18:40 -07:00
Ryan Houdek 9e7daf61d0 Merge pull request #2133 from lioncash/inselem
IR: Handle 256-bit VInsElement
2022-11-02 18:59:18 -07:00
lioncash 6fbe25753b IR: Handle 256-bit VInsElement
Extends VInsElem to handle 256-bit vectors.
2022-11-03 01:43:42 +00:00
Ryan Houdek 03f0edc5b5 Merge pull request #2132 from lioncash/indexed
IR: Handle 256-bit LoadContextIndexed/StoreContextIndexed
2022-11-02 17:13:36 -07:00
lioncash 5536f1e835 Arm64/ConversionOps: Eliminate use of temporary in Vector_FToF
We can just use the destination register in this case.
2022-11-02 23:50:28 +00:00
lioncash 0de36706da IR.json: Expand allowed size in LoadContext and StoreContext IR ops
These can now handle 256-bit destinations
2022-11-02 16:19:05 +00:00
lioncash 17722dad6d IR: Handle 256-bit LoadContextIndexed
Extends LoadContextIndexed to handle 256-bit vectors.
2022-11-02 16:16:58 +00:00
lioncash 0371599996 IR: Handle 256-bit StoreContextIndexed
Extends StoreContextIndexed to handle 256-bit vectors.
2022-11-02 16:07:02 +00:00
Ryan Houdek 199649b30f Merge pull request #2131 from lioncash/simplify
Arm64/MemoryOps: Merge if statement into switch in ParanoidLoadMemTSO
2022-11-01 21:35:25 -07:00
lioncash 4ef35488db Arm64/MemoryOps: Merge if statement into switch in ParanoidLoadMemTSO
There's nothing preventing the OpSize == 1 case from being merged into
the switch, so we can do that to make things a little more consistent.
2022-11-02 03:45:17 +00:00
Ryan Houdek 70a91ee6ce Merge pull request #2130 from lioncash/memory
IR: Handle 256-bit StoreMem/StoreMemTSO/ParanoidStoreMemTSO
2022-11-01 20:41:34 -07:00
lioncash 418a27e47e IR: Handle 256-bit ParanoidStoreMemTSO
Extends ParanoidStoreMemTSO to handle 256-bit vectors.
2022-11-01 23:04:30 +00:00
lioncash 61c76d02cc IR: Handle 256-bit StoreMemTSO
Extends StoreMemTSO to handle 256-bit vectors.
2022-11-01 22:59:29 +00:00
lioncash d98641221d IR: Handle 256-bit StoreMem
Extends StoreMem to handle 256-bit vectors.
2022-11-01 21:39:52 +00:00
Ryan Houdek 8a14f87a44 Merge pull request #2129 from lioncash/memory
IR: Handle 256-bit LoadMem/LoadMemTSO/ParanoidLoadMemTSO
2022-11-01 14:23:19 -07:00
lioncash 02ce71734c IR: handle 256-bit ParanoidLoadTSO
Extends ParanoidLoadTSO to handle 256-bit vectors.
2022-11-01 21:02:42 +00:00
lioncash 96c2743280 IR: Handle 256-bit LoadMemTSO
Extends LoadMemTSO to handle 256-bit vectors.
2022-11-01 21:02:42 +00:00
lioncash 7bfc34b51c IR: Handle 256-bit LoadMem
Extends LoadMem to handle 256-bit vectors.
2022-11-01 21:02:38 +00:00
Ryan Houdek 40d820fd05 Merge pull request #2127 from lioncash/spill
Arm64/MemoryOps: Remove lingering unnecessary ptrue instances
2022-11-01 10:47:15 -07:00
lioncash d69287aaf7 Arm64/MemoryOps: Remove lingering unnecessary ptrue instances
Gets rid of some leftover bits from when we didn't have statically
allocated predicate registers.
2022-11-01 17:25:24 +00:00
Ryan Houdek d475b0ba9e Merge pull request #2126 from lioncash/ctx
IR: Handle 256-bit LoadContext/StoreContext
2022-11-01 10:22:26 -07:00
lioncash 1638b744b7 x86_64/MemoryOps: Ensure upper lane is cleared properly in FillRegister
Ensures that loaded values don't potentially have junk in the upper
lane. Will prevent potential wonky situations when implementing AVX
instructions.
2022-11-01 16:39:18 +00:00
lioncash 8b19894a06 IR: Handle 256-bit StoreContext
Extends StoreContext to handle 256-bit vectors.
2022-11-01 16:20:24 +00:00
Ryan Houdek d2e0dc99de Merge pull request #2125 from lioncash/unused
Interpreter/MiscOps: Remove unused StopThread() function
2022-11-01 09:13:31 -07:00
lioncash d04e40b5fd Interpreter/MiscOps: Remove unused StopThread() function
This has been unused since ff1d51c7bd

Silences a compiler warning.
2022-11-01 15:59:17 +00:00
lioncash 75d797b5cd IR: Handle 256-bit LoadContext
Extends LoadContext to handle 256-bit vectors.
2022-11-01 14:54:26 +00:00
Mai ecf4891087 Merge pull request #1668 from Sonicadvance1/wip_segment_register
Segment register index optimization
2022-11-01 02:54:44 +00:00
Ryan Houdek 0e1a418678 WIP: Segment register index optimization
Segment registers are indexed significantly more than they are changed.
Pay the cost of indexing during the set and store rather than the per
register index.

Should be a fairly significant performance improvement for 32-bit
applications. At least on hardware that doesn't have a data dependent
prefetcher.

Breaks Steam atm and isn't clean.
2022-10-31 19:42:30 -07:00
Mai 5bef13df94 Merge pull request #2124 from Sonicadvance1/gvisor_flakes
unittests/gvisor: Adds a bunch of tests to flakes
2022-10-31 21:03:38 +00:00
Ryan Houdek d8386121a8 Merge pull request #2115 from Sonicadvance1/fix_x11_thunk_recursion
Thunks/libX11: Fix recursive initialize
2022-10-31 13:41:07 -07:00
Ryan Houdek 000677abb6 Merge pull request #2078 from Sonicadvance1/fix_48bit_va_stack
Allocator: Expand stack space when stealing virtual address space
2022-10-31 13:11:20 -07:00
Ryan Houdek 64eb87e9b5 Merge pull request #2099 from Sonicadvance1/fix_infinite_loop
FEXServer: Be robust against invalid packets.
2022-10-31 13:11:13 -07:00
Ryan Houdek aa5e92bee2 Merge pull request #2083 from Sonicadvance1/fix_x87_flag_range
X87: Claim incoming float was in the range for trancendental ops
2022-10-31 13:10:42 -07:00
Ryan Houdek 0bf79dc5d6 unittests/gvisor: Adds a bunch of tests to flakes
These are getting annoying.
2022-10-31 12:53:04 -07:00
Ryan Houdek adb2171c0a Thunks/libX11: Fix recursive initialize
Fixes a crash that occurs due to `_XInitDisplayLock` due to the display
lock function being initialized to our own handler.

Once XInitThreads is called once then it becomes a no-op.

steamwebhelper was hitting this.
2022-10-31 12:38:12 -07:00
Ryan Houdek d6f8923f86 X87: Claim incoming float was in the range for trancendental ops
We don't detect the range of the long F80, so we need to set that the
source was in range to fix sin/cos/tan calculations.

If we don't set this flag to zero then glibc will do some additional
operations that causes the value to be incorrect.

Fixes the output of the test application in #2021, probably fixes some
camera orientation problems in games as well.
2022-10-31 12:36:49 -07:00
Ryan Houdek cf91ab9d5f Merge pull request #2123 from Sonicadvance1/fix_32bit_vdso
32bit: Fixes Debug build of VDSO
2022-10-31 12:17:10 -07:00
Ryan Houdek a0fb9531db FEXServer: Be robust against invalid packets.
Chrome seems to like sending us invalid packets of data sometimes. With
an invalid packet type just skip parsing the data entirely.

Fixes an infinite loop in Vampire Survivors.
2022-10-31 12:05:48 -07:00
Ryan Houdek eca9353b28 Merge pull request #2122 from Sonicadvance1/fix_rotate_right_of
OpcodeDispatcher: Fixes ROR imm OF calculation
2022-10-31 11:52:49 -07:00
Ryan Houdek a259730639 32bit: Fixes Debug build of VDSO
This was generating GOT prologues even on naked functions which was
breaking VDSO on 32-bit.

Fixes almost every 32-bit application when running with debug options.
2022-10-31 11:52:17 -07:00
Ryan Houdek 2e93d10eba OpcodeDispatcher: Fixes ROR imm OF calculation
Turns out this was calculating OF incorrectly, breaking Denuvo early in
its execution.

Changes the ROL imm OF calculation code as well to be more consistent
and not keep src1 alive longer than it needs to be.

Also adds two new unit tests to ensure this stays correct.
2022-10-31 10:28:47 -07:00
Mai 70a3ceb64e Merge pull request #2096 from Sonicadvance1/cleanup_64allocator
Utils/64BitAllocator: Minor cleanups and optimization for munmap
2022-10-31 16:47:53 +00:00
Mai b726f60afd Merge pull request #2098 from Sonicadvance1/fprem_tests
unittests/asm: Adds more extensive FPREM/FPREM1 tests
2022-10-31 16:38:42 +00:00
Mai 2fa1a64999 Merge pull request #2120 from Sonicadvance1/fix_proton_experimental_48bit
ELFCodeLoader: Fixes Proton Experimental on 48-bit VA systems
2022-10-31 16:38:07 +00:00
Ryan Houdek a42b659af9 ELFCodeLoader: Fixes Proton Experimental on 48-bit VA systems
This is a tricky situation that wine-preloader allocates the lower
32MB of stack space through fixed address mmap with MAP_FIXED.

They can't use mmap with an address hint nor MAP_FIXED_NOREPLACE because
it changes behaviour. mmap won't give you the allocation inside the
stack space even if you check `/proc/self/maps` that space isn't yet
allocated. The growable space of the stack blocks those allocations.

So the wine peeps might be SOL if they actually require this allocation
to exist.

To replicate this, allocate the application stack at the same location using an address hint.
This will give us the correct region on a 48-bit VA system, while also
letting it select a different region on a 36-bit VA system.
2022-10-28 02:30:18 -07:00
Ryan Houdek 004c3230a4 Merge pull request #2108 from Sonicadvance1/implement_thunk_disables
Thunks: Add support for disabling thunks in config
2022-10-26 23:38:55 -07:00
Ryan Houdek 2332c41510 Merge pull request #2119 from lioncash/tbl
IR: Handle 256-bit VTBL1
2022-10-26 20:44:23 -07:00
lioncash ec3039c5a2 IR: Handle 256-bit VTBL1
Extends VTBL1 to handle 256-bit vectors.
2022-10-27 00:35:45 +00:00
Ryan Houdek 639d6e6071 Merge pull request #2118 from lioncash/prfx
Arm64/VectorOps: Make use of MOVPRFX where applicable
2022-10-26 15:25:51 -07:00
lioncash cd518d4726 Arm64/VectorOps: Make use of MOVPRFX where applicable
Allows hardware to pack the move and following destructive operation
together into one constructive operation if possible.

e.g.

movprfx VTMP1.D, VectorLower.D
addp VTMP1.B, Pred, VTMP1.B, VectorUpper.B

is allowed to be merged as if it executed constructively like:

addp VTMP1.B, Pred, VectorLower.B, VectorUpper.B

if the hardware supports it. If it doesn't, then the instructions will
behave like a regular move and destructive addp operation separately.
2022-10-26 21:38:48 +00:00
Ryan Houdek b7d9c00dff Merge pull request #2117 from lioncash/ins
IR: Handle 256-bit VInsGPR
2022-10-26 13:22:35 -07:00
lioncash 4b17575f5a IR: Handle 256-bit VInsGPR
Extends VInsGPR to handle 256-bit vectors.
2022-10-26 19:45:27 +00:00
Ryan Houdek 5ba4bba138 Thunks: Add support for disabling thunks in config
Previously the config options could only have ever enabled thunks rather than
disable them.

Now sort the code so it can enable thunks, then following configs can
redisable them.  Allowing testing with global thunks enabled and
disabling problematic applications.

Also sorts the "ThunkConfigFile" config as lower priority than the
application configs. I wasn't thinking about ordering that hard for
these five configuration paths, but application configs should be higher
priority in this case.
2022-10-26 11:43:16 -07:00
Ryan Houdek b3ee5dba0f Merge pull request #2116 from lioncash/extract
IR: Handle 256-bit VExtractToGPR
2022-10-26 11:16:45 -07:00
lioncash d87ff5afa9 IR: Handle 256-bit VExtractToGPR
Extends VExtractToGPR to handle 256-bit vectors.
2022-10-26 17:43:31 +00:00
Ryan Houdek 62a24bd38f Merge pull request #2075 from Sonicadvance1/gpuvis_profiler
FEXCore: Adds support for a timeline profiler interface
2022-10-26 08:44:54 -07:00
Ryan Houdek 8d373c15b8 Merge pull request #2107 from Sonicadvance1/sort_and_upgrade_x11_thunk
Thunks/X11: Reorder and sort X11 interface by headers included.
2022-10-26 04:53:48 -07:00
Ryan Houdek 671f3e74a4 Merge pull request #2103 from Sonicadvance1/sse2_for_guest
Thunks/Guest: Enable SSE2 on thunks and set fpmath to sse
2022-10-26 04:51:35 -07:00
Ryan Houdek 7e810233d9 Merge pull request #2112 from lioncash/ftoi
IR: Handle 256-bit Vector_FToI
2022-10-26 00:07:04 -07:00
Ryan Houdek 4700dbd676 Thunks/Guest: Enable SSE2 on thunks and set fpmath to sse
Clang thunks already have these default enabled, but let's also enable
this on the GCC side.

sse2 will enable most things we care about, which matches ASIMD quite
closely.
fpmath=sse removes some x87 usage for 32-bit thunks specifically.

Should effectively be a non-functional-change
2022-10-26 00:05:16 -07:00
Ryan Houdek 74e18f4317 Merge pull request #2114 from lioncash/vec
Arm64/BranchOps: Remove unused std::vector
2022-10-25 22:02:09 -07:00
lioncash 1eea95cf18 Arm64/BranchOps: Remove unused std::vector
Removes a heap allocation for inline syscalls.
2022-10-26 04:34:57 +00:00
Ryan Houdek 7291b10727 Merge pull request #2113 from lioncash/scvtf
IR: Check for invalid conversion masks in Float_FromGPR_S
2022-10-25 21:32:43 -07:00
lioncash 6804916697 IR: Check for invalid conversion masks in Float_FromGPR_S
Previously this would silently ignore unhandled masks.
2022-10-26 03:59:25 +00:00
lioncash 819e61bf14 IR: Handle 256-bit Vector_FToI
Expands Vector_FToI to handle 256-bit vectors.
2022-10-26 03:42:23 +00:00
Ryan Houdek b8f7e4c8ec Merge pull request #2111 from lioncash/ftof
IR: Handle 256-bit Vector_FtoF
2022-10-25 20:17:26 -07:00
lioncash 17bcc0eed4 IR: Handle 256-bit Vector_FtoF
Extends Vector_FtoF to handle 256-bit vectors.
2022-10-26 02:58:06 +00:00
Ryan Houdek 13003da289 Merge pull request #2110 from lioncash/ftozs
IR: Handle 256-bit Vector_FToZS/Vector_FToS
2022-10-25 19:10:45 -07:00
lioncash 9273538955 IR: Handle 256-bit Vector_FToS
Extends Vector_FToS to handle 256-bit vectors.
2022-10-26 00:59:11 +00:00
lioncash 9750189def IR: Handle 256-bit Vector_FToZS
Extends Vector_FToZS to handle 256-bit vectors.
2022-10-26 00:53:53 +00:00
Ryan Houdek cb17ee9871 Merge pull request #2109 from lioncash/stof
IR: Handle 256-bit Vector_SToF
2022-10-25 17:14:16 -07:00
lioncash 4c3b78ba9a IR: Handle 256-bit Vector_SToF
Extends Vector_SToF to handle 256-bit vectors.
2022-10-25 23:54:41 +00:00
Ryan Houdek e00b6a401b Thunks/X11: Reorder and sort X11 interface by headers included.
Each one of these are sorted through the DefinitionExtracy.py script
running over a temporary header file for each set of includes.

eg:
```bash
$ cat test.h
 #include <X11/Xproto.h>
 #include <X11/XKBlib.h>
 #include <X11/Xlib.h>
 #include <X11/Xutil.h>
 #include <X11/Xresource.h>

 #include <X11/ImUtil.h>
$ ./Scripts/DefinitionExtract.h test.h > out.txt
```

Any custom defined types have been sorted appropriately.
A bunch of missing XKB definitions were missing and added in the
process.
I've had this stashed in my git stash for a while now, I just haven't
cleaned it up.

Fixes a bunch of thunks around X11 applications missing symbols.
2022-10-25 15:43:02 -07:00
Ryan Houdek ac0ab8a7b4 Thunks/X11: Ensure 11 headers are included with C linkage
Otherwise the compiler gets confused about some functions getting
declared with C++ linkage.
2022-10-25 15:32:40 -07:00
Ryan Houdek 0aff3941f4 Scripts/DefinitionExtract: Fixes some more function attributes
X11 has an attribute that was causing function declarations to be
missed.

These definitions exist in XLibint.h
eg:
```cpp
extern void _XEatData(
    Display*		/* dpy */,
    unsigned long	/* n */
) _X_COLD;
```

This `_X_COLD` attribute was causing these function definitions to get
missed.
2022-10-25 15:32:40 -07:00
Ryan Houdek 27b022d4d9 Merge pull request #2106 from lioncash/dup
IR: Handle 256-bit VDupElement
2022-10-25 13:49:29 -07:00
lioncash e188928742 IR: Handle 256-bit VDupElement
Extends VDupElement to handle 256-bit vectors.
2022-10-25 20:01:53 +00:00
Ryan Houdek 780e3c7fb7 Merge pull request #2105 from lioncash/unzip
IR: Handle 256-bit VUnZip/VUnZip2
2022-10-25 12:45:05 -07:00
lioncash 1b5146d3ac IR: Handle 256-bit VUnZip2
Extends VUnZip2 to handle 256-bit vectors.
2022-10-25 16:01:55 +00:00
lioncash 80cf3ca6b9 IR: Handle 256-bit VUnZip
Extends VUnZip to handle 256-bit vectors.
2022-10-25 15:49:02 +00:00
Ryan Houdek 2272b30a91 Merge pull request #2101 from Sonicadvance1/fix_thunk_versions
Thunks: Fixes missing thunk librarie so versions
2022-10-24 23:11:54 -07:00
Ryan Houdek b5fb1cb07c Merge pull request #2100 from Sonicadvance1/fix_thunk_loaded_check
ThunksDB: Fixes Thunks loaded boolean pointer check
2022-10-24 23:11:33 -07:00
Ryan Houdek 4ea34a9c22 Thunks: Fixes missing thunk librarie so versions
Some libraries were missing these version defines, which was causing
dlopen to fail.

This was causing thunks to break in pressure-vessel.
2022-10-24 20:59:54 -07:00
Ryan Houdek 48e7de9f9e ThunksDB: Fixes Thunks loaded boolean pointer check
Need to dereference the boolean to ensure we only load the thunksDB
files once.
2022-10-24 20:55:58 -07:00
Ryan Houdek 76dd2369a7 unittests/asm: Adds more extensive FPREM/FPREM1 tests
unit tests that show the difference of output between FPREM and FPREM1.
Setup as known failures on everything except for host since we don't
implement fprem correctly.

An incorrect fix to FPREM is as follows:
```diff
--- a/External/FEXCore/Source/Common/SoftFloat.h
+++ b/External/FEXCore/Source/Common/SoftFloat.h
@@ -158,6 +158,10 @@ struct X80SoftFloat {

     return Result;
 #else
+    BIGFLOAT lhs_ = lhs;
+    BIGFLOAT rhs_ = rhs;
+    BIGFLOAT Result = fmodl(lhs_, rhs_);
+    return Result;
     return extF80_rem(lhs, rhs);
 #endif
   }
```

But we shouldn't implement this fix. We should instead implement a new `extF80_mod`
function that handles the rounding differences between FPREM and FPREM1.

Fixes #2097.
Doesn't attempt to resolve #1538
2022-10-22 20:47:24 -07:00
Ryan Houdek 78e0cd6e77 Scripts: Updates testharness_runner to support runner specific known failures 2022-10-22 20:29:51 -07:00
Ryan Houdek 5514a04cb4 Utils/64BitAllocator: Minor cleanups and optimization for munmap
- Some minor cleanups in the VMARegion struct type.
- Switches over to using the FlexBitSet range scanning, based off this
implementation.
- Move memory region allocation to its own function instead of
  constructor
  - This will be used by a new constructor later for 48-bit host-side
    allocations
- Minor optimization to keep track of Munmap.
  - We were burning a bunch of time on backward scanning for free
    regions even though we never did a munmap to free anything.
  - Now only do backward scanning if a munmap occured.
  - Saves a bunch of CPU time
2022-10-21 21:07:09 -07:00
Ryan Houdek 99ca78b235 Utils/FlexBitSet: Adds range scanning functions
These were currently living in the 64BitAllocator class but can be moved
directly to the FlexBitSet.

Ideally in the future these routines can be optimized so our allocator
is faster but for now these are just moved.
2022-10-21 20:58:26 -07:00
Ryan Houdek ab45db1665 Merge pull request #2094 from lioncash/zip
IR: Handle 256-bit VZip/VZip2
2022-10-20 19:02:29 -07:00
Ryan Houdek bb38bcb67d Merge pull request #2093 from lioncash/shrn
IR: Handle 256-bit VUShrNI/VUShrNI2
2022-10-20 19:00:51 -07:00
Ryan Houdek 6cc2912542 Merge pull request #2092 from lioncash/vsqxtun
IR: Handle 256-bit VSQXTUN/VSQXTUN2
2022-10-20 18:57:24 -07:00
lioncash 340b2ca624 IR: Handle 256-bit VZip2
Extends VZip2 to handle 256-bit vectors.
2022-10-20 21:15:18 +00:00
lioncash 5baa15de03 IR: Handle 256-bit VZip
Extends VZip to handle 256-bit vectors.
2022-10-20 19:22:07 +00:00
lioncash 6ddca804d1 IR: Handle 256-bit VUShrNI2
Extends VUShrNI2 to handle 256-bit vectors.
2022-10-20 18:31:36 +00:00
lioncash f9831a85fb IR: Handle 256-bit VUShrNI
Extends VUShrNI to handle 256-bit vectors.
2022-10-20 17:51:47 +00:00
lioncash 7261033b7f IR: Handle 256-bit VSQXTUN2
Extends VSQXTUN2 to handle 256-bit vectors.
2022-10-20 17:08:00 +00:00
lioncash 3ad6866198 IR: Handle 256-bit VSQXTUN
Extends VSQXTUN to handle 256-bit vectors.
2022-10-20 16:51:53 +00:00
Ryan Houdek 1c7d4165ab Merge pull request #2091 from lioncash/vsqxtn
IR: Handle 256-bit VSQXTN/VSQXTN2
2022-10-19 20:46:53 -07:00
Ryan Houdek 3e48b1a8ac FEXCore: Adds support for a timeline profiler interface
This creates a generic interface that FEXCore can use for timeline
profiling. This allows us to create a generic interface which the
backend details are hidden so we can support multiple timeline profile
APIs.

The only API supported right now is ftrace/gpuvis. Which is extremely
lightweight of an interface with minimal overhead.

We must be careful here since in most cases will will have dozens of
FEX instances running at any given time. So a timeline profiler like
Microprofiler can have major issues since that only ever expects a
single process at a time.

Not enabled by default but just needs the `ENABLE_FEXCORE_PROFILER`
cmake option set to enable.
2022-10-19 19:56:35 -07:00
lioncash 07be100daf IR: Handle 256-bit VSQXTN2
Extends VSQXTN2 to handle 256-bit vectors.
2022-10-20 02:41:45 +00:00
lioncash de9351eefb IR: Handle 256-bit VSQXTN
Expands VSQXTN to handle 256-bit vectors.
2022-10-20 02:04:05 +00:00
Ryan Houdek 2c44b5b3a1 Allocator: Expand stack space when stealing virtual address space
If we take all of the stack space then the auto expanding stack doesn't
work and we get stuck with a small stack that breaks thunks.
2022-10-19 19:02:41 -07:00
Ryan Houdek 136f1e2fc7 Merge pull request #2090 from lioncash/sxtl
Arm64/VectorOps: Simplify SVE VSXTL/VSXTL2/VUXTL/VUXTL2 implementations
2022-10-19 17:02:17 -07:00
lioncash ad39add55f Arm64/VectorOps: Simplify VUXTL2 SVE implementation
Turns out there's an instruction that does what we need, but isn't named
similarly to UXTL2 at all.
2022-10-19 22:24:07 +00:00
lioncash f3c301e359 Arm64/VectorOps: Simplify VUXTL SVE implementation
Turns out there's an instruction that does what we need, but has a name
not similar to UXTL
2022-10-19 22:22:23 +00:00
lioncash 1c37a1b4d6 Arm64/VectorOps: Simplify VSXTL2 SVE implementation
Turns out there's a built-in instruction that does exactly what we want,
but just has a different name from SXTL2
2022-10-19 22:16:33 +00:00
lioncash 7222529904 Arm64/VectorOps: Simplify VSXTL SVE implementation
Was reading the ARM ARM and realized there's an instruction that does
exactly what we need right out of the box.
2022-10-19 22:12:22 +00:00
Ryan Houdek f26eccd00f Merge pull request #2089 from wannacu/main
Implements DAA, DAS, AAA, AAS, AAM and AAD instruction
2022-10-19 03:57:13 -07:00
wannacu 73375a76ac unittests: Adds DAA, DAS, AAA, AAS, AAM and AAD unit test 2022-10-19 13:56:21 +08:00
wannacu d4416d200e OpcodeDispatcher: Implements DAA, DAS, AAA, AAS, AAM and AAD instruction 2022-10-19 13:56:06 +08:00
Mai d1b235dd83 Merge pull request #2080 from Sonicadvance1/fix_64bit_syscall_mman
Syscalls: Fixes 64-bit mmap and munmap
2022-10-19 01:26:15 +00:00
Ryan Houdek 3ac5e0423a Merge pull request #2088 from lioncash/vsmull
IR: Handle 256-bit VSMull/VSMull2
2022-10-18 16:11:03 -07:00
Ryan Houdek fc6de5f3c0 Merge pull request #2087 from lioncash/vixl-narrow
External: Update vixl submodule
2022-10-18 16:08:31 -07:00
Mai b1e475d81d Merge pull request #2081 from Sonicadvance1/fix_rotate_flags
OpcodeDispatcher: Fixes flag calculation on ROR and ROL by immediate
2022-10-18 22:33:47 +00:00
Mai 4a09a4324f Merge pull request #2082 from Sonicadvance1/fix_c2_fprem1
OpcodeDispatcher: Fixes FPREM1 C2 flag calculation
2022-10-18 22:33:29 +00:00
lioncash 47f94327c5 IR: Amend x86_64 32->64 case for VUMull2
Realized I forgot to amend the registers used in the final multiply.
2022-10-18 16:23:16 +00:00
lioncash a009ed0b6b IR: Handle 256-bit VSMull2
Extends VSMull2 to handle 256-bit vectors.
2022-10-18 16:23:14 +00:00
lioncash 2476a686e7 IR: Handle 256-bit VSMull
Extends VSMull to handle 256-bit vectors.
2022-10-18 16:22:44 +00:00
lioncash f0db93773f unittests: Re-enable narrowing and widening tests
Now that the bug in vixl's simulator is fixed, we can enable these tests
again.
2022-10-18 15:18:14 +00:00
lioncash fabe824c8b Externals: Update vixl submodule
Includes fixes for the narrowing instructions.
2022-10-18 15:15:54 +00:00
Ryan Houdek 78a077397e Merge pull request #2085 from lioncash/vmull
IR: Handle 256-bit VUMull/VUMull2
2022-10-17 18:10:48 -07:00
lioncash 0c4b456aaa IR: Handle 256-bit VUMull2
Extends VUMull2 to handle 256-bit vectors.
2022-10-17 19:03:26 +00:00
lioncash 09185167bc OpcodeDispatcher/Vector: Amend and simplify PMULLOp
Allows PMULLOp to function correctly with the amended VPSHUFD entries.

Since this is only used to perform expanded multiplication from 32-bit
entries to 64-bit entries, we can simplify things a little bit.

All we need to do is yank the third 32-bit word down into the second
32-bit word's spot in the vector and let the VUMull/VSMull IR ops handle
it.
2022-10-17 19:03:26 +00:00
lioncash 5d78c3203c IR: Handle 256-bit VUMull
Extends VUMull to handle 256-bit vectors.

While we're at it, we can fix a typo in the VPSHUFD called for the
32->64-bit case.

To mirror UMULL, we need to replicate element 0 and 1, not 0 and 2

While we're at it, we can fix this with VSMull as well.
2022-10-17 19:03:23 +00:00
Ryan Houdek fa5322d3f9 Merge pull request #2084 from Sonicadvance1/more_auxv
ELFCodeLoader: Implement four more auxv values
2022-10-17 09:48:18 -07:00
Ryan Houdek d21aa5cac2 ELFCodeLoader: Implement four more auxv values
Implements AT_PLATFORM: Ends up being `i686` or `x86_64` depending on
ELF arch

Implements AT_HWCAP and AT_HWCAP2
AT_HWCAP is just CPUID function 01h EDX result
AT_HWCAP2 only has two defined bits in it, which we don't support
either.

Implements AT_RANDOM
Previously we were just sticking hardcoded values in to this.
Now we pass along the host's AT_RANDOM, or we generate our own if that
doesn't exist

Fixes #788
2022-10-16 19:45:19 -07:00
Ryan Houdek 102d5c57cb OpcodeDispatcher: Fixes FPREM1 C2 flag calculation
Accidentally didn't implement this for FPREM1 but it /was/ implemented
for FPREM. Fixes an infinite loop in cossin implementations.

Test code from the application returns an incorrect result, but it isn't
due to FPREM1.

```
$ `which wine` ./hello.exe
Sin 1.22460635382238E-16
Cos -1
$ FEXInterpreter `which wine` ./hello.exe
Sin 1.22460635382238E-16
Cos 0.54030230586814
```

Fixes #2021
2022-10-16 15:24:46 -07:00
Ryan Houdek ffb4de9fd9 Merge pull request #2079 from Sonicadvance1/ensure_armemitter_uses_allocator
Ensure Arm64Emitter uses FEX allocator
2022-10-15 15:37:23 -07:00
Ryan Houdek 6b3d8886e5 Merge pull request #2077 from Sonicadvance1/fix_thunks_with_lots_args
Thunks: Fixes indirect thunks with 8+ arguments
2022-10-15 15:37:09 -07:00
Ryan Houdek ddc10272a0 Merge pull request #2076 from Sonicadvance1/update_vulkan
Thunks: Update Vulkan thunk to v1.3.231
2022-10-15 15:14:30 -07:00
Ryan Houdek 0b5ef00165 Thunks: Fixes indirect thunks with 8+ arguments
Due to how we use a modified ABI for these indirect functions, we don't
have a clean way to say that the host_addr lives in a side-argument.

The previous inline asm that moved the value from r11 in to a variable
worked up until you hit functions with 8 or more arguments. At that
point the compiler was generating code before our inline assembly and
using r11 as a temporary, thus destroying our value.
Then a crash would occur and it was very hard to determine why. It would
end up calling some random function (0x1 in this case) from an indirect
call.

This made it /look/ like it was calling an invalid function returned
from the loader but in reality it was a corrupt register loading bad
data.

To work around this case, we can use an inline asm register variable and
a volatile asm block that "sets" the variable. In this case GCC and
Clang both seem to extend the live range of the register from the start
of the function to the use of the variable.

This resolves the issue for now, and I tested quite a large number of
function signatures to see if it would break in the future.

Theoretically our functional testing should catch this, but we don't
currently have something that abuses all the functions like this
currently.
2022-10-15 15:13:40 -07:00
Ryan Houdek 2a50416fc3 unittests: Ensures overloaded shifts don't result in JIT failure 2022-10-14 23:38:35 -07:00
Ryan Houdek ce514d9f83 unittests: Adds ROL and ROR CF flag calculation tests
This would have failed prior to the last commit
2022-10-14 23:37:46 -07:00
Ryan Houdek 9b77e7fd13 OpcodeDispatcher: Fixes flag calculation on ROR and ROL by immediate
These were being calculated incorrectly in the case of rotating with
values larger than 8-bit or 16-bit
2022-10-14 23:36:48 -07:00
Ryan Houdek abb44d3327 Merge pull request #2069 from wannacu/main
Flags: Refine _Bfe's shift
2022-10-14 22:53:05 -07:00
Ryan Houdek 11eaf3d48a Ensure Arm64Emitter uses FEX allocator
Otherwise we will end up allocating code buffers in the lower 32-bits,
consuming precious virtual address space.
2022-10-14 22:04:07 -07:00
Ryan Houdek 76c2cc2c3e Syscalls: Fixes 64-bit mmap and munmap
These should be using the real syscalls, not our provided allocators.

While not a problem currently since these redirect to host mmap and
munmap, it will become an issue once we have an allocator that lives
outside of x86-64 space.
2022-10-14 21:43:04 -07:00
Ryan Houdek 0e6c8bd12e Thunks: Update Vulkan thunk to v1.3.231
Only missing a few function definitions, resorted to match order of
definitions in the headers so future changes don't mix up as much
2022-10-14 01:51:35 -07:00
Ryan Houdek a9fb008317 External: Update Vulkan-Headers to v1.3.231 2022-10-14 01:50:49 -07:00
Ryan Houdek e9f3a5b3e4 Merge pull request #2074 from lioncash/vuxtl
IR: Handle 256-bit VUXTL/VUXTL2
2022-10-13 13:15:47 -07:00
Ryan Houdek ebc45dff45 Merge pull request #2070 from lioncash/vuabdl
IR: Handle 256-bit VUABDL
2022-10-13 12:17:47 -07:00
lioncash fc4a5ebfd3 IR: Handle 256-bit VUXTL2
Extends VUXTL2 to handle 256-bit vectors.
2022-10-13 19:15:05 +00:00
lioncash 1d7b688c55 IR: Handle 256-bit VUXTL
Extends VUXTL to handle 256-bit values.
2022-10-13 19:07:57 +00:00
Ryan Houdek f14a5ffbbf Merge pull request #2073 from lioncash/vsxtl
IR: Handle 256-bit VSXTL/VSXTL2
2022-10-13 11:53:12 -07:00
Ryan Houdek cada0d593c Merge pull request #2072 from lioncash/test
unittests: Amend mm register usage in H0F38/66_04.asm test
2022-10-13 11:28:27 -07:00
lioncash 0436540791 IR: Handle 256-bit VSXTL2
Extends VSXTL2 to handle 256-bit vectors.
2022-10-13 18:21:25 +00:00
lioncash a87ac86e18 IR: Handle 256-bit VSXTL
Extends VSXTL to handle 256-bit vectors.
2022-10-13 18:21:22 +00:00
lioncash 1278b23150 unittests: Amend mm register usage in H0F38/66_04.asm test
This should be using xmm2 rather than mm2.
2022-10-13 17:09:40 +00:00
lioncash 02f5ea4b9d IR: Handle 256-bit VUABDL
Extends VUABDL to handle 256-bit vectors.
2022-10-13 15:37:10 +00:00
wannacu 2e14e613d0 Flags: Refine _Bfe's shift 2022-10-13 16:41:04 +08:00
Ryan Houdek 85c2889652 Docs: Update for release FEX-2210 2022-10-13 00:46:59 -07:00
Ryan Houdek 23dd056b60 Merge pull request #2067 from lioncash/vmul
IR: Handle 256-bit VSMul/VUMul
2022-10-12 15:59:03 -07:00
lioncash 71043e372a IR: Handle 256-bit VSMul/VUMul
Extends VSMul and VUMul to handle 256-bit vectors.
2022-10-12 00:00:15 +00:00
Ryan Houdek c412d073b9 Merge pull request #2066 from lioncash/vrev64
IR: Handle 256-bit VRev64
2022-10-11 14:21:45 -07:00
lioncash 8fb03ff1b9 IR: Handle 256-bit VRev64
Extends VRev64 to handle 256-bit vectors.
2022-10-11 20:17:04 +00:00
Ryan Houdek c2b6aef6f4 Merge pull request #2065 from lioncash/shift-imm
IR: Handle 256-bit VShlI/VUShlI/VUShrI
2022-10-11 12:16:44 -07:00
lioncash 24547318c6 IR: Handle 256-bit VShlI
Extends VShlI to handle 256-bit vectors.
2022-10-11 18:19:23 +00:00
Ryan Houdek b693112c80 Merge pull request #2058 from Sonicadvance1/add_opencl_thunk_db
Add opencl thunk db
2022-10-11 11:06:20 -07:00
lioncash 48d1184066 IR: Handle 256-bit VSShrI
Extends VSShrI to handle 256-bit vectors.
2022-10-11 18:04:13 +00:00
Ryan Houdek 8da9ebc2e0 Merge pull request #2062 from wannacu/main
SMC: Fix possible deadlock
2022-10-11 10:13:02 -07:00
lioncash 5ba510474b IR: Handle 256-bit VUShrI
Extends VUShrI to handle 256-bit vectors.
2022-10-11 17:08:35 +00:00
Ryan Houdek 51214d1be1 Merge pull request #2064 from lioncash/vushls
IR: Handle 256-bit VSShrS/VUShlS/VUShrS
2022-10-11 09:59:03 -07:00
Ryan Houdek 4d6e15d7af Merge pull request #2063 from lioncash/interp-shift
Interpreter: Handle 256-bit VSShr/VUShl/VUShr
2022-10-11 09:22:11 -07:00
lioncash 4721894427 IR: Handle 256-bit VSShrS
Extends VSShrS to handle 256-bit vectors.
2022-10-11 16:21:50 +00:00
lioncash d429865b6e IR: Handle 256-bit VUShrS
Extends VUShrS to handle 256-bit vectors.
2022-10-11 16:08:09 +00:00
lioncash ca5881a72c IR: Handle 256-bit VUShlS
Extends VUShlS to handle 256-bit vectors.
2022-10-11 15:45:39 +00:00
lioncash 7151b9daff Interpreter/VectorOps: Remove lingering magic 32 constants
Makes these functions consistent with the rest that explicitly test for
256 bit width.
2022-10-11 15:02:41 +00:00
lioncash d9b5e28b22 Interpreter: Handle 256-bit VSShr
This is only implemented in the interpreter, so this is trivial.
2022-10-11 14:58:40 +00:00
lioncash aa7954a7d6 Interpreter: Handle 256-bit VUShr
This is only implemented in the interpreter, so this is trivial.
2022-10-11 14:56:19 +00:00
lioncash 802c70d1ab Interpreter: Handle 256-bit VUShl
This is only implemented in the interpreter, so this is trivial.
2022-10-11 14:55:17 +00:00
wannacu 8f905988e9 Use compatible syscall helpers 2022-10-11 16:35:17 +08:00
wannacu 478c5595ad SMC: Fix possible deadlock 2022-10-11 15:31:27 +08:00
Ryan Houdek 3977e1f29e Merge pull request #2055 from Sonicadvance1/update_description_ripping_script
Scripts: Updates DefinitionExtract
2022-10-10 10:11:16 -07:00
Ryan Houdek 2b1ef97354 Merge pull request #2060 from Sonicadvance1/clang_thunks
Thunks: Add support for building with clang
2022-10-10 09:44:58 -07:00
Mai eaddf7f1a5 Merge pull request #2061 from Sonicadvance1/fix_linker_script_depends
Thunks: Adds dependency on linker script
2022-10-10 12:17:20 -04:00
Ryan Houdek 3237de3085 Merge pull request #2056 from Sonicadvance1/guest_function_bool
Thunks/Host: Adds bool operator to fex_guest_function_ptr
2022-10-10 09:10:34 -07:00
Ryan Houdek df3d398d31 Thunks/Host: Adds bool operator to fex_guest_function_ptr
Lets us check if nullptr was passed in
2022-10-10 08:52:50 -07:00
Ryan Houdek b44b3401b7 Merge pull request #2015 from Sonicadvance1/map_regular_offset
ELFCodeLoader: Map primary ELF more like the kernel
2022-10-10 08:49:49 -07:00
Ryan Houdek c28ca0fac9 ELFCodeLoader: Map primary ELF more like the kernel
The kernel maps the primary ELF with a hint to some place *near* the
middle of the virtual address space. While the interpreter stays at the
top of the address space.

This also adds ASLR to the ELF loading, with a define for debugging and
testing purposes.

Requires both #2013 and #2014 merged first.
2022-10-10 08:38:15 -07:00
Ryan Houdek 235e2b6c2c FEXCore/Allocator: Store what the host VA is
Calling this function multiple times without this will change the
result.

Necessary so we can determine what the host VA is  from multiple
locations.
2022-10-10 08:35:40 -07:00
Mai d68b84bc27 Merge pull request #2013 from Sonicadvance1/fix_mapper
ELFCodeloader: Map once and then use MAP_FIXED to overwrite
2022-10-10 10:05:53 -04:00
Mai edca528608 Merge pull request #2039 from Sonicadvance1/fix_dynamic_non_interpreter_elfs
ELFCodeLoader: Fixes dynamic non-interpreter ELFs
2022-10-10 10:04:19 -04:00
Ryan Houdek b75e8f2abf Thunks: Add support for building with clang
Fairly straightforward, just requires enabling lld in this case since
cross-compiling doesn't work well with gnu linker.

Also lld doesn't understand the linker script program header symbolic
names for read/write/execute. So we need to use the raw number there.

Works around an issue where GCC 11 generates broken `init_array` section
and also plt sections that glibc doesn't understand.
2022-10-09 23:07:30 -07:00
Ryan Houdek ec3158e4cd Thunks: Adds dependency on linker script
Ensures the thunk is rebuilt if the linker scripts have changed.

Fixes #2054
2022-10-09 22:57:52 -07:00
Ryan Houdek 9c8c8041e0 ThunksDB: Adds OpenCL to the json
This will be used soon
2022-10-09 19:52:36 -07:00
Ryan Houdek e3adaacb51 Thunks: Adds another packed arguments template
This will be used soon
2022-10-09 19:47:37 -07:00
Ryan Houdek c49e11484f Scripts: Updates DefinitionExtract
Unused warning attribute wasn't getting ignored.
Also need to update output text to match new format
2022-10-09 19:44:49 -07:00
Ryan Houdek 7b4b9a80fa Merge pull request #2053 from lioncash/vfcmpord
IR: Handle 256-bit VFCMPORD/VFCMPUNO
2022-10-09 19:43:36 -07:00
lioncash c7ad066987 IR: Handle 256-bit VFCMPUNO
Extends VFCMPUNO to handle 256-bit vectors.
2022-10-06 15:43:01 +00:00
lioncash f4d229f1ba IR: Handle 256-bit VFCMPORD
Extends VFCMPORD to handle 256-bit vectors.
2022-10-06 15:33:30 +00:00
Ryan Houdek 25a8a00771 Merge pull request #2050 from lioncash/vfcmplt
IR: Handle 256-bit VFCMPLT/VFCMPGT/VFCMPLE
2022-10-04 14:44:54 -07:00
Ryan Houdek a67f7422b2 Merge pull request #2049 from lioncash/fcmeq
IR: Handle 256-bit VFCMPEQ/VFCMPNEQ
2022-10-04 14:43:52 -07:00
Ryan Houdek ed8150cfb6 Merge pull request #2048 from lioncash/vcmpgt
IR: Handle 256-bit VCMPGT/VCMPGTZ/VCMPLTZ
2022-10-04 14:42:53 -07:00
Ryan Houdek 462a163ba7 Merge pull request #2047 from lioncash/vcmpeq
IR: Handle 256-bit VCMPEQ/VCMPEQZ
2022-10-04 14:40:48 -07:00
Ryan Houdek 6374175a64 Merge pull request #2046 from lioncash/vbsl
IR: Handle 256-bit VBSL
2022-10-04 14:37:25 -07:00
lioncash 280b15ba2a IR: Handle 256-bit VFCMPLE
Extends VFCMPLE to handle 256-bit vectors.
2022-10-04 19:56:59 +00:00
lioncash 2a0b488e99 IR: Handle 256-bit VFCMPGT
Extends VFCMPGT to handle 256-bit vectors.
2022-10-04 19:43:32 +00:00
lioncash 3ef7c4ab51 IR: Handle 256-bit VFCMPLT
Extends VFCMPLT to handle 256-bit vectors.
2022-10-04 19:29:36 +00:00
lioncash e72d746036 IR: Handle 256-bit VFCMPNEQ
Extends VFCMPNEQ to handle 256-bit vectors.
2022-10-04 19:02:20 +00:00
lioncash 0bb4091e34 IR: Handle 256-bit VFCMPEQ
Extends VFCMPEQ to handle 256-bit vectors.
2022-10-04 18:44:12 +00:00
lioncash 6822fc595c IR: Handle 256-bit VCMPLTZ
Extends VCMPLTZ to handle 256-bit vectors.
2022-10-04 18:09:39 +00:00
lioncash a506a589dd IR: Handle 256-bit VCMPGTZ
Extends VCMPGTZ to handle 256-bit vectors.
2022-10-04 17:55:03 +00:00
lioncash 684a5977dd IR: Handle 256-bit VCMPGT
Extends VCMPGT to handle 256-bit registers.
2022-10-04 17:38:20 +00:00
lioncash ac3682e058 IR: Handle 256-bit VCMPEQZ
Extends VCMPEQZ to handle 256-bit vectors.
2022-10-04 16:57:19 +00:00
lioncash d5faf01f5a IR: Handle 256-bit VCMPEQ
Extends VCMPEQ to handle 256-bit vectors.
2022-10-04 16:33:25 +00:00
lioncash ecba1b6838 IR: Handle 256-bit VBSL
Extends VBSL to handle 256-bit vectors.
2022-10-03 18:06:27 +00:00
Ryan Houdek 8d8b029285 Merge pull request #2044 from lioncash/vumax
IR: Handle 256-bit VSMax/VUMax
2022-09-29 13:52:40 -07:00
lioncash bf6f855868 IR: Handle 256-bit VSMax
Extends VSMax to handle 256-bit vectors.
2022-09-29 20:31:19 +00:00
Ryan Houdek aa6a499329 Merge pull request #2043 from lioncash/vumin
IR: Handle 256-bit VSMin/VUMin
2022-09-29 13:19:07 -07:00
lioncash 0971650ef9 IR: Handle 256-bit VUMax
Extends VUMax to handle 256-bit vectors.
2022-09-29 20:18:42 +00:00
Ryan Houdek 64c4fdccf7 Merge pull request #2042 from lioncash/vnot
IR: Handle 256-bit VNot
2022-09-29 13:13:23 -07:00
Ryan Houdek d715ffbc8e Merge pull request #2041 from lioncash/vfneg
IR: Handle 256-bit VFNeg
2022-09-29 13:12:21 -07:00
lioncash aef801b5b5 IR: Handle 256-bit VSMin
Extends VSMin to handle 256-bit vectors.
2022-09-29 20:01:04 +00:00
lioncash 6c9e29796b IR: Handle 256-bit VUMin
Extends VUMin to handle 256-bit vectors.
2022-09-29 19:47:47 +00:00
lioncash 364bb3ac1e IR: Handle 256-bit VNot
Extends VNot to handle 256-bit vectors.
2022-09-29 14:40:46 +00:00
lioncash 428ea68507 IR: Handle 256-bit VFNeg
Extends VFNeg to handle 256-bit vectors.
2022-09-29 14:00:52 +00:00
Ryan Houdek 5cf59408a7 Merge pull request #2040 from Sonicadvance1/fix_vsyscall
VDSO: Fix vsyscall
2022-09-29 01:47:53 -07:00
Ryan Houdek 1596843015 VDSO: Fix vsyscall
The `mov ebp, ecx` was breaking vsyscall and was expected to be used
with the `syscall` instruction rather than `int 0x80`.
Remove that to fix it.

Also remove the pushes and pops around the syscall instruction, these
are unnecessary in an emulated environment, we won't clobber the
registers.

Fixes Steam execution with VDSO.
2022-09-28 17:34:24 -07:00
Ryan Houdek af6582ff5b ELFCodeLoader: Fixes dynamic non-interpreter ELFs
Specifically fixes /sbin/ldconfig.
Fixes 8df7c2d84f
Fixes Steam launching

I failed to test ELF files that are dynamic with no interpreter here, so
EntryPoint ended up being set to zero, which results in an instant
crash.

Ensure these values are set correctly after the primary ELF and
interpreter are loaded so starting RIP is correct.
2022-09-28 16:00:02 -07:00
Ryan Houdek 1799d4c675 Merge pull request #2038 from lioncash/vneg
IR: Handle 256-bit VNeg
2022-09-28 13:16:55 -07:00
Ryan Houdek 808e1c0330 Merge pull request #2029 from lioncash/interp
Interpreter: Use constant for AVX register size where applicable
2022-09-28 13:16:14 -07:00
Ryan Houdek dacd96cab5 Merge pull request #2037 from lioncash/vfrsqrt
IR: Handle 256-bit VFRSqrt
2022-09-28 13:15:50 -07:00
Ryan Houdek ca4d3bf64d Merge pull request #2036 from lioncash/vfsqrt
IR: Handle 256-bit VFSqrt
2022-09-28 13:14:25 -07:00
Ryan Houdek ea38b043c1 Merge pull request #2035 from lioncash/vfrecp
IR: Handle 256-bit VFRecp
2022-09-28 13:12:57 -07:00
Ryan Houdek a39746df2e Merge pull request #2034 from lioncash/vfmax
IR: Handle 256-bit VFMax
2022-09-28 13:11:04 -07:00
Ryan Houdek 2367a8e50b Merge pull request #2033 from lioncash/vfmin
IR: Handle 256-bit VFMin
2022-09-28 13:10:17 -07:00
Ryan Houdek cb121d7f17 Merge pull request #2032 from lioncash/vaddp
IR: Handle 256-bit VAddP
2022-09-28 13:08:40 -07:00
Ryan Houdek 412793c21d Merge pull request #2030 from lioncash/interp-mov
Interpreter: Handle 256-bit VMov
2022-09-28 13:05:16 -07:00
lioncash ce2286c48b IR: Handle 256-bit VNeg
Extends VNeg to handle 256-bit vectors.
2022-09-28 19:25:33 +00:00
lioncash d5694d6de0 IR: Handle 256-bit VFRSqrt
Extends VFRSqrt to handle 256-bit vectors.
2022-09-28 19:11:01 +00:00
lioncash 121218aa8a IR: Handle 256-bit VFSqrt
Extends VFSqrt to handle 256-bit vectors.
2022-09-28 18:46:43 +00:00
lioncash 6bb53fa758 IR: Handle 256-bit VFRecp
Extends VFRecp to handle 256-bit vectors.
2022-09-28 18:27:34 +00:00
lioncash 89aa0c5471 IR: Handle 256-bit VFMax
Extends VFMax to handle 256-bit vectors.
2022-09-28 17:40:16 +00:00
lioncash 53fcbf6afa IR: Handle 256-bit VFMin
Extends VFMin to handle 256-bit vectors.
2022-09-28 16:56:22 +00:00
lioncash 2f5643ae6b Arm64/VectorOps: Amend half-precision case in VFAddP
Noticed that I forgot to change the Zn register over to VTMP1.
This would have been caught by a vixl internal assert anyway.

Also make the behavior equal with VAddP, where we clear and only copy
over the exact amount of bytes instead of the whole register.
2022-09-28 15:10:49 +00:00
lioncash d162ac8b3d IR: Handle 256-bit VAddP
Extends VAddP to handle 256-bit vectors.
2022-09-28 15:01:31 +00:00
lioncash c9a704fbde Interpreter: Handle 256-bit VMov
Now we'll properly handle the move.

Also put an assert in place to catch any over-sized values.
2022-09-28 13:47:23 +00:00
lioncash 9d5a822a3a Interpreter: Use constant for AVX register size where applicable
Makes the previously implemented ops a little more self-documenting,
and, if we ever actually need to change this, there's a nice constant
that can be looked up instead of magic 32 values.
2022-09-28 13:38:03 +00:00
Ryan Houdek 50eba4066a Merge pull request #2028 from lioncash/vfdiv
IR: Handle 256-bit VFDiv
2022-09-27 13:39:02 -07:00
lioncash 6116ae5330 IR: Handle 256-bit VFDiv
Extends VFDiv to handle 256-bit vectors.
2022-09-27 20:20:31 +00:00
Ryan Houdek 447226576f Merge pull request #2027 from lioncash/vfmul
IR: Handle 256-bit VFMul
2022-09-27 13:18:38 -07:00
Ryan Houdek 3f8b872f17 Merge pull request #2026 from lioncash/vfsub
IR: Handle 256-bit VFSub
2022-09-27 13:17:00 -07:00
Ryan Houdek e573ddc2db Merge pull request #2025 from lioncash/vfaddp
IR: Handle 256-bit VFAddP
2022-09-27 12:56:58 -07:00
lioncash fe9aa681f0 IR: Handle 256-bit VFMul
Extends VFMul to handle 256-bit vectors.
2022-09-27 19:54:57 +00:00
lioncash 1b2f2c1559 IR: Handle 256-bit VFSub
Extends VFSub to be able to handle 256-bit vectors.
2022-09-27 19:33:46 +00:00
lioncash 84c75a86c3 IR: Handle 256-bit VFAddP
Extends VFAddP to be able to handle 256-bit vectors.
2022-09-27 19:03:44 +00:00
Ryan Houdek eedbde6f15 Merge pull request #2024 from lioncash/vfadd
IR: Handle 256-bit VFAdd
2022-09-27 09:53:25 -07:00
Ryan Houdek 4e441e5a08 Merge pull request #2023 from lioncash/vpopcnt
IR: Handle 256-bit VPopcount
2022-09-27 09:51:27 -07:00
Ryan Houdek 3e287a36c2 Merge pull request #2022 from lioncash/vabs
IR: Handle 256-bit VAbs
2022-09-27 09:48:26 -07:00
lioncash eadc477695 IR: Handle 256-bit VFAdd 2022-09-27 16:07:42 +00:00
lioncash 59aa324678 IR: Handle 256-bit VPopcount
Extends VPopcount to be able to handle 256-bit width vectors
2022-09-27 15:19:24 +00:00
lioncash 219bce1467 IR: Handle 256-bit VAbs
Extends VAbs to be able to handle 256-bit width vectors.
2022-09-27 13:56:38 +00:00
Ryan Houdek 46bde401bd Merge pull request #2019 from Sonicadvance1/remove_mov
IR: Removes Mov IR op
2022-09-26 19:49:52 -07:00
Ryan Houdek 01beac4956 Merge pull request #2018 from Sonicadvance1/remove_vextractelement
IR: Removes VExtractElement
2022-09-26 19:49:46 -07:00
Ryan Houdek fcd981e6b7 Merge pull request #2017 from Sonicadvance1/remove_vbitcast
IR: Removes unnecessary VBitcast IR op
2022-09-26 19:44:12 -07:00
Ryan Houdek 825833cfcc IR: Removes Mov IR op
This is unused and shouldn't ever be used.
2022-09-26 16:04:27 -07:00
Ryan Houdek 4a4c49bf68 IR: Removes VExtractElement
This is a duplicate of VDupElement since AArch64 doesn't support an
element extract plus zero of the rest of the register.

Removes and replaces its uses with VDupElement.
2022-09-26 15:55:41 -07:00
Ryan Houdek 763cea423a IR: Removes unnecessary VBitcast IR op
This instruction was purely a move that did format reinterpretation.
This was necessary with LLVM when we had implicit IR op sizes.

Now that all vector ops are explicitly sized, this is not only
redundant, but also completely unnecessary sicne we don't support LLVM
anymore.

Remove the op, which technically is a very minor optimization for the
two instructions that still used it.
2022-09-26 15:42:24 -07:00
Ryan Houdek 0fee355ff5 Merge pull request #2016 from lioncash/pred
Arm64/VectorOps: Make use of static predicate registers
2022-09-26 14:48:26 -07:00
Ryan Houdek 6f6f3c9dc5 Merge pull request #2012 from Sonicadvance1/32bit_vdso
32-bit VDSO support
2022-09-26 14:46:27 -07:00
Ryan Houdek 25e5d88ab2 VDSO Emulation: Wires up support for 32-bit VDSO 2022-09-26 14:35:38 -07:00
Ryan Houdek 87013340bb Thunks: Adds support for building 32-bit. Only VDSO for now. 2022-09-26 14:35:38 -07:00
Ryan Houdek 47c075ccc9 Thunks/VDSO: Add 32-bit linker script 2022-09-26 14:35:38 -07:00
Ryan Houdek b1a32d4ccf Thunks: Ensure fexthunks functions are hidden visible by default 2022-09-26 14:35:38 -07:00
Ryan Houdek 7af6a8dbdf Thunks/VDSO: Extend to support clock_gettime64 2022-09-26 14:35:38 -07:00
Ryan Houdek 383e99e4ef unittests: Extend VDSO test for gettime64 2022-09-26 14:35:38 -07:00
Ryan Houdek 8c7cfc4d11 Config: Adds support for unique 32-bit GuestThunk path 2022-09-26 14:01:48 -07:00
lioncash 496ee730c8 Arm64/VectorOps: Make use of static predicate registers
Since PR #2003, we now set up some predicate registers within
FillStaticRegs. We can now make use of those instead of manually setting
up predicate registers inside the IR opcodes.
2022-09-26 13:20:01 +00:00
Ryan Houdek 71f7ff5101 Merge pull request #2014 from Sonicadvance1/map_interp_first
ELFCodeLoader: Map interpreter first
2022-09-26 02:42:39 -07:00
Ryan Houdek 1ea00f68a2 Merge pull request #2010 from Sonicadvance1/add_support_for_32_bit_thunk_op
Thunks: Implement the Thunk IR op for 32-bit mode
2022-09-26 02:33:47 -07:00
Ryan Houdek 8df7c2d84f ELFCodeLoader: Map interpreter first
This more closely matches behaviour of the kernel.
Provides an example in source to ensure we don't break it in the future.
2022-09-25 18:07:31 -07:00
Ryan Houdek 46557a7a1f ELFCodeloader: Map once and then use MAP_FIXED to overwrite
Instead of mapping to find a range, unmapping, and then submapping
inside of it.

Map once, then mmap with MAP_FIXED to overwrite the mapping.
This fixes an issue where if you enabled ASAN then it would stick
additional mappings inbetween where we want to map. Thus breaking asan.
2022-09-25 17:50:16 -07:00
Ryan Houdek d8c2a8271f Merge pull request #2009 from Sonicadvance1/libvulkan_fix_print
Thunks/libvulkan: Fixes print for 32-bit
2022-09-25 13:33:00 -07:00
Ryan Houdek cc4c705fc0 Merge pull request #2008 from Sonicadvance1/disable_32_bit_x11
ThunkLibs: X11/Xext: Removes two functions that don't exist on 32-bit
2022-09-25 13:32:54 -07:00
Ryan Houdek 22f249fcf6 Thunks: Implement the Thunk IR op for 32-bit mode
Use the fastcall ABI for 32-bit x86 to make our lives easier.
Fastcall ABI puts the first two 32-bit arguments in ECX and EDX
respectively.

Compilers are nice today and allow us to do cross-abi function calls
like this.
2022-09-25 13:16:30 -07:00
Mai c262362a03 Merge pull request #2011 from Sonicadvance1/add_missing_flake
FEXLinuxTests: Adds missing pthread_cancel flake status
2022-09-24 20:53:22 -04:00
Ryan Houdek 212df9aa7b FEXLinuxTests: Adds missing pthread_cancel flake status
Missed the 32-bit version of this test
2022-09-24 10:05:36 -07:00
Ryan Houdek 691e39ec76 Thunks/libvulkan: Fixes print for 32-bit
Value passed in to this print will be 32-bit or 64-bit depending on
arch.

Noticed this while tinkering around and is easy enough to solve today.
2022-09-24 09:56:50 -07:00
Ryan Houdek 107cae2975 ThunkLibs: X11/Xext: Removes two functions that don't exist on 32-bit
_XData32 and _XRead32 don't exist as real functions in 32-bit versions
of these libraries, these end up just being defines that redirect to the
non-suffixed versions of the functions.

Noticed this while tinkering around and is easy enough to solve today.
2022-09-24 09:51:40 -07:00
Ryan Houdek 6742e0c376 Merge pull request #2003 from lioncash/svespill
JITs: Handle spilling/filling 256-bit vectors
2022-09-23 17:27:53 -07:00
Ryan Houdek 8f70137b1a Merge pull request #1981 from neobrain/feature_flt_catch2
FEXLinuxTests: Migrate to Catch2
2022-09-23 17:27:45 -07:00
lioncash 707db51b1b Arm64Emitter: Amend comment for GPR temporaries
Only x3 can be used across spill boundaries.
2022-09-24 00:12:01 +00:00
lioncash 5b5fa1aa29 x86_64/JIT: Handle pushing and popping 256-bit values 2022-09-24 00:11:56 +00:00
Ryan Houdek 2b9cc9666a Merge pull request #2006 from Sonicadvance1/remove_splat
IR: Removes SplatVector{2,4}
2022-09-23 14:42:44 -07:00
Ryan Houdek 82eba22292 Merge pull request #2007 from Sonicadvance1/remove_insscalar
IR: Removes VInsScalarElement
2022-09-23 14:42:36 -07:00
Ryan Houdek 5c84e8f23c IR: Removes VInsScalarElement
This IR op duplicates what VInsElement does.
2022-09-22 17:57:20 -07:00
Ryan Houdek 081b61677a IR: Removes SplatVector{2,4}
These IR ops are redundant and mostly unused.
VDupElement does exactly what these operations were already doing and
more closely matches what the hardware wants.
2022-09-22 17:49:28 -07:00
lioncash 9c54814b98 Arm64Emitter: Handle spilling 256-bit dynamic regs 2022-09-22 12:40:54 +00:00
lioncash 35d7b855ed Arm64Dispatcher: Increment code buffer size
vixl hits an assertion in CodeBuffer's Emit() function since there's no
space for any more instructions with the changes made to handle SVE.
2022-09-22 12:40:54 +00:00
lioncash 0b8799274c Arm64Emitter: Handle filling/spilling 256-bit static FPRs
Drops in handling for spilling/filling FPRs using SVE for supporting
AVX.

Also alters the dispatcher and JIT a little to avoid accidentally clobbering
TMP4 (x3 as of this commit)
2022-09-22 12:40:54 +00:00
lioncash ace2b737d8 Arm64Emitter: Initialize fixed predicate register values in FillStaticRegs
Allows us to have values set up in a way that we don't need to
constantly set up predicates in IR ops.
2022-09-22 12:40:54 +00:00
Mai ad85268524 Merge pull request #2005 from Sonicadvance1/fix_sve_vectorimm
Arm64: Fixes SVE VectorImm
2022-09-22 08:38:10 -04:00
Mai 832a320e22 Merge pull request #2004 from Sonicadvance1/update_vixl3
Update vixl external
2022-09-22 08:33:39 -04:00
Ryan Houdek 83763df6fd Arm64: Fixes SVE VectorImm
SVE DUP instruction does sign extension on the incoming immediate, while
ASIMD MOVI does zero extension.

If the immediate doesn't fit then move in to a GPR first and then DUP
from GPR.
2022-09-22 01:16:58 -07:00
Ryan Houdek bee868e9ba Update vixl external 2022-09-22 01:10:25 -07:00
Tony Wasserka c41de81694 FEXLinuxTests: Drop support for now unused "args:" annotations 2022-09-22 10:03:38 +02:00
Tony Wasserka 41aaeb1ff0 FEXLinuxTests: Migrate signal tests to Catch2 2022-09-22 10:03:37 +02:00
Tony Wasserka 16be2792ab FEXLinuxTests: Migrate FD test to Catch2 2022-09-22 10:03:36 +02:00
Tony Wasserka 6610bb355c FEXLinuxTests: Migrate VDSO test to Catch2 2022-09-22 10:03:35 +02:00
Tony Wasserka 4312fd7291 FEXLinuxTests: Migrate SMC tests to Catch2 2022-09-22 10:03:33 +02:00
Tony Wasserka 89a225a96d FEXLinuxTests: Enable use of Catch2 in tests 2022-09-22 10:03:31 +02:00
Ryan Houdek a590977639 Merge pull request #1984 from Sonicadvance1/functional_thunk_ci
Thunks: Adds functional thunk testing to CI
2022-09-20 11:14:03 -07:00
Ryan Houdek 0d0d116bde Merge pull request #2002 from lioncash/slots
JITs: Expand max spill slot size to 32 bytes
2022-09-19 15:22:25 -07:00
lioncash 341bdb5a54 JITs: Handle 32 byte spills and fills
Puts in the plumbing necessary to handle spilling and filling 256-bit
data.
2022-09-19 22:00:20 +00:00
lioncash 2f3dbfb289 JITs: Expand max spill slot size to 32 bytes
This will be necessary to handle spilling 256-bit vectors.
2022-09-19 19:51:32 +00:00
Ryan Houdek 169cfbbeed Merge pull request #2001 from lioncash/defmove
Arm64: Centralize location for register defines
2022-09-19 11:45:24 -07:00
Ryan Houdek f97a4afd8f Merge pull request #2000 from Sonicadvance1/fix_struct_verifier_ubuntu_20_04
CI: Fixes struct verifier on Ubuntu 20.04
2022-09-19 11:44:38 -07:00
lioncash 868e4a6d81 Arm64: Centralize location for register defines
Gets rid of a few repeated definitions and allows the emitter itself to
make use of these defines without causing a circular dependency on the
JIT.
2022-09-19 17:44:21 +00:00
Ryan Houdek 6f48f7d3ac CI: Fixes struct verifier on Ubuntu 20.04
Older clang fails to pull in these include paths when cross compiling.
Add them manually.
2022-09-19 01:27:54 -07:00
Tony Wasserka 1ed3ecb409 Merge pull request #1999 from neobrain/feature_toolchain_32bit
CMake: Add toolchain file for 32-bit cross-compiler
2022-09-19 09:30:57 +02:00
Tony Wasserka 2cb455b9d4 CMake: Add toolchain file for 32-bit cross-compiler 2022-09-19 09:19:22 +02:00
Ryan Houdek d4b5bf0f78 Merge pull request #1998 from Sonicadvance1/fix_struct_verifier
StructVerifier: Fixes CI failure
2022-09-18 19:26:21 -07:00
Ryan Houdek 69013772c1 StructVerifier: Fixes CI failure
The x86 runner had unattended-upgrades accidentally still enabled. It
upgraded a bunch of development packages which broke CI.

Fix the struct verifier so it works with the new packages.
Sadly python3-clang doesn't support all the new CursorKind types so we
need to self-define some of them for now.

Once this tool gets converted over to C++ it will be a non-issue.
2022-09-16 19:45:13 -07:00
Ryan Houdek 3448c83431 Merge pull request #1997 from neobrain/refactor_flt_unified_cmake
FEXLinuxTests: Build 32-bit and 64-bit test variants separately
2022-09-16 17:14:24 -07:00
Tony Wasserka e4c84542ea FEXLinuxTests: Build 32-bit and 64-bit test variants separately
This allows to use different toolchain files for each and it reduces
build system repetition in test target setup.

The "tests-32" directories has been integrated into the "tests" one. Tests
that should only run on 32-bit are detected by their filename ending with
".32.cpp" now.
2022-09-16 11:24:18 +02:00
Ryan Houdek acddc0323b Merge pull request #1995 from Sonicadvance1/disable_bad_test
unittests: Disable gvisor pselect test
2022-09-15 22:06:37 -07:00
Ryan Houdek c4285f0d30 unittests: Disable gvisor pselect test
This has a badly coded test that can hang forever. Our timeout kills it
at 5 minutes, which causes it to not even fall down the flake path.

Just disable it outright because of the bad test.
2022-09-15 16:29:02 -07:00
Ryan Houdek 977d6dd247 Merge pull request #1993 from lioncash/vuravg
VectorOps: Handle 256-bit VURAvg
2022-09-15 15:09:46 -07:00
lioncash 2bb27fffb7 VectorOps: Handle 256-bit VURAvg 2022-09-15 19:38:31 +00:00
Ryan Houdek 0261ed353d Merge pull request #1992 from lioncash/uminv
VectorOps: Handle 256-bit VUMinV
2022-09-15 12:05:27 -07:00
Ryan Houdek 96fecfd7c5 Merge pull request #1989 from Sonicadvance1/ci_flakes
CI: Adds support for flakes
2022-09-15 12:05:08 -07:00
Ryan Houdek 9fac1b8105 CI: Adds support for flakes
If a test is marked as a flake then it will be tried five times before
giving up.

Works around the problem of needing to babysit CI once a PR is pushed.
As long as we have all the flake tests marked.
2022-09-15 11:44:56 -07:00
lioncash 95fbcd7b9a VectorOps: Handle 256-bit VUMinV 2022-09-15 18:36:46 +00:00
Ryan Houdek 6adf227611 Merge pull request #1990 from Sonicadvance1/uninstall
cmake: Adds uninstall target
2022-09-15 11:33:27 -07:00
Ryan Houdek b5cb429243 Merge pull request #1991 from lioncash/bits
Interpreter: Handle 256-bit VAnd/VBic/VOr/VXor
2022-09-15 11:23:49 -07:00
Ryan Houdek 26ba8079a3 cmake: Adds uninstall target
Following guidance from cmake's FAQ:
https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#can-i-do-make-uninstall-with-cmake

Due to some of the special handling that we do with installs, we need to
do additional uninstall handling that the install manifest doesn't cover.

Specifically we need to add additional uninstall targets for:
- FEXInterpreter
- binfmt_misc
- guest_thunks (Doing its own uninstall target, so passthrough)

While it isn't generally advised to install and uninstall through source
systems, this is something that users want to do all the time.
This has been asked for a couple of times now.

Fixes #1592
2022-09-15 11:22:24 -07:00
lioncash f999d30bc5 Interpreter: Handle 256-bit VAnd 2022-09-15 16:18:14 +00:00
lioncash ecb1cc4ed4 Interpreter: Handle 256-bit VBic 2022-09-15 16:18:14 +00:00
lioncash 5622bcae16 Interpreter: Handle 256-bit VOr 2022-09-15 16:18:14 +00:00
lioncash f4539ee289 Interpreter: Handle 256-bit VXor 2022-09-15 16:18:10 +00:00
Ryan Houdek d2138694b4 Merge pull request #1986 from neobrain/refactor_flt_cmake_cleanup
FEXLinuxTests: Use the build system instead of setting up compile flags via source-code annotations
2022-09-15 01:14:08 -07:00
Tony Wasserka 1767e21273 FEXLinuxTests: Use the build system instead of setting up compile flags via source-code annotations
The intent of these annotations was presumably to make it easier to adjust
build settings on a per-test basis, but doing this in the build system is
actually much cleaner.
2022-09-15 08:55:18 +02:00
Ryan Houdek 8f9d799342 Merge pull request #1988 from Sonicadvance1/fexserver_wait_old_kernel
FEXServer: Fix waiting on kernel version older than 5.3
2022-09-14 17:07:41 -07:00
Ryan Houdek a3b0b246f4 FEXServer: Fix waiting on kernel version older than 5.3
pidfd_open was added in kernel 5.3 so older kernel devices weren't able
to use `FEXServer -w`. If the syscall doesn't give us an FD to the
process, then use a pipe instead.

Since we are only polling for the FD to hangup this works for us.
2022-09-14 16:04:26 -07:00
Ryan Houdek dee85f14fe Merge pull request #1987 from Sonicadvance1/fix_fhu_syscalls
FHU: Convert to a interface target
2022-09-14 15:55:04 -07:00
Ryan Houdek 27309114be FHU: Convert to a interface target
Noticed recently that `FEXServer -w` was broken and couldn't understand
why. Turns out that FHU syscall handling was /always/ falling down the
`#else` path in the handlers since cmake `add_definitions` follows
folder scoping rules.

This means it was always returning -1, which was causing FEXServer's
pidfd_open usage to always receive -1, which meant the sendmsg with FD
was always failing, which meant the `FEXServer -w` would forever wait
for a message that was never sent.

Converting the utility over to a target not only fixes definition
scoping problems, but also makes the other paths actually work.

This found some compiling bugs and instead lets us define SYS_pidfd_open
if it doesn't exist. Letting the kernel return the ENOSYS if it doesn't
exist on that platform.

Main thing, fixes FEXServer -w hanging forever.
2022-09-14 14:58:23 -07:00
Ryan Houdek 704afed97b Merge pull request #1985 from neobrain/refactor_thunkgen_fmt
Thunks/gen: Use fmt for writing formatted output
2022-09-14 13:52:15 -07:00
Ryan Houdek 121f0a2c6c CI: FetchRootFS More robust rootfs fetching
Permissions mean we need to delete the folder before extracting.
On error make sure to delete the image file as well to ensure it reruns
everything.
2022-09-14 13:18:24 -07:00
Ryan Houdek 1c580ec92c Thunks: Adds functional thunk testing to CI
This is the bare minimum, it only tests glxinfo and vulkaninfo with and
without thunks. Nothing more special than that. Already found the .1 bug
with libvulkan host library loading.
2022-09-14 12:48:13 -07:00
Tony Wasserka ab8fc721a0 Thunks/gen: Use fmt for writing formatted output 2022-09-14 11:56:15 +02:00
Ryan Houdek 790447115c CI: Set CMAKE_INSTALL_PREFIX
This will be used in the next commit
2022-09-13 17:21:03 -07:00
Ryan Houdek 80abeac28a Thunks: Fixes a missing version number on libvulkan
Fixes an issue with loading libvulkan without development packages.
2022-09-13 17:20:06 -07:00
Ryan Houdek 54915f87ce Merge pull request #1978 from neobrain/refactor_astvisitor_to_frontendaction
Move thunk generator logic from ASTVisitor to ASTFrontendAction
2022-09-13 11:26:36 -07:00
Ryan Houdek f34f1309a7 Merge pull request #1983 from lioncash/vsqadd
VectorOps: Extend VSQAdd/VSQSub/VUQAdd/VUQSub
2022-09-13 11:26:11 -07:00
Ryan Houdek 0ad52b7d19 Merge pull request #1982 from lioncash/vadd
VectorOps: Extend VAdd/VSub
2022-09-13 11:24:39 -07:00
lioncash 809f60df06 VectorOps: Handle 256-bit VSQSub 2022-09-13 16:43:42 +00:00
lioncash cbdcd8253c VectorOps: Handle 256-bit VSQAdd 2022-09-13 16:20:37 +00:00
lioncash 7ac2cd7cc8 VectorOps: Handle 256-bit VUQSub 2022-09-13 16:03:30 +00:00
lioncash cf1bb1348c VectorOps: Handle 256-bit VUQAdd 2022-09-13 16:03:27 +00:00
lioncash b60a26ff9e VectorOps: Handle 256-bit VSub 2022-09-13 15:40:45 +00:00
lioncash b805c07342 VectorOps: Handle 256-bit VAdd 2022-09-13 15:40:06 +00:00
Mai 8d69f539ac Merge pull request #1979 from Sonicadvance1/fix_thunkconfig
FEXConfig: Ensure APP_CONFIG_NAME isn't stored in json
2022-09-13 11:38:24 -04:00
Ryan Houdek c8c0054f67 FEXConfig: Ensure APP_CONFIG_NAME isn't stored in json
Also in FEXLoader make sure to use `EraseSet` for these runtime options.

Fixes a bug where the config was being set to nothing, breaking the
ThunksDB configuration option.
2022-09-13 00:08:20 -07:00
Tony Wasserka 1085385bbe Thunks/gen: Move logic from ASTVisitor to ASTFrontendAction
ASTVisitor is great for iterating over AST nodes by type, but most of our
analysis is based on symbol names. For this task, a lookup in DeclContexts
after parsing is complete is better suited.
2022-09-12 18:52:33 +02:00
Tony Wasserka 56460b220c Thunks/gen: Move definition of GenerateThunkLibsAction into gen.cpp 2022-09-12 18:52:33 +02:00
Mai a583ebe590 Merge pull request #1977 from Sonicadvance1/extend_arch_check
CMake: Extend AArch64 check to include arm64
2022-09-09 22:36:30 -04:00
Ryan Houdek ce8175a800 CMake: Extend AArch64 check to include arm64
This has been seen in some build environments, forgot to commit this a
while ago.
2022-09-08 15:34:35 -07:00
Mai c987e1ef44 Merge pull request #1974 from Sonicadvance1/update_release_process
Docs: Update Release docs
2022-09-08 11:48:41 -04:00
Mai b36ec152d2 Merge pull request #1976 from Sonicadvance1/support_simulator
Add support for the vixl simulator
2022-09-08 11:47:46 -04:00
Ryan Houdek 44c62e703e github: Adds vixl simulator CI 2022-09-07 20:08:50 -07:00
Ryan Houdek 0f59c1d5e3 Add support for the vixl simulator
This will allow CI to test ARM features before we have any hardware that
supports it.
2022-09-07 19:54:07 -07:00
Ryan Houdek 5739f0b459 Update vixl 2022-09-07 19:10:13 -07:00
Ryan Houdek 4145fabfb6 Docs: Update Release docs
Reorder PPA building to be after the github tag. PPA takes a while to
run, so good to get it out of the way up front so it can be handled in
the background while doing the rest of the release.

Also update the link which was renamed.
2022-09-05 11:08:44 -07:00
Ryan Houdek ae34b1e521 Docs: Update for release FEX-2209 2022-09-05 10:32:07 -07:00
Ryan Houdek c17da25617 Merge pull request #1973 from neobrain/refactor_single_thunkgen_output
Thunks: Consolidate all generated code to one file per library per platform
2022-09-05 10:20:54 -07:00
Tony Wasserka cc8ef16240 Thunks/gen: Consolidate all generated code to one file per library per platform 2022-09-05 15:03:49 +02:00
Tony Wasserka 30fac81c41 Thunks/gen: Remove unused symtable generation code 2022-09-05 15:03:49 +02:00
Ryan Houdek bbcca80b60 Merge pull request #1971 from Sonicadvance1/termux_namespace_collision
Syscalls: Use underscored shm syscall names
2022-09-05 02:55:29 -07:00
Ryan Houdek 56ae3716bb Syscalls: Use underscored shm syscall names
Removes us needing to carry a patch downstream in the termux package
repo.
2022-09-05 02:39:42 -07:00
Ryan Houdek 2fba3a4a9a LinuxAllocator: Uppercase function names
Not only does this match our internal naming convention. This avoids the
shm symbol namespace collision as well.
2022-09-05 02:39:42 -07:00
Ryan Houdek 8ba0312d40 Syscalls: Prefix _ to shm syscall names to avoid namespace conflict
Termux uses defines for these, so our token pasting fails, but we also
still want to use their define so we can fall down their emulation
library whenever possible.

Prefix an underscore to be able to use both our number definitions and
their defines in the same file.
2022-09-05 02:39:42 -07:00
Ryan Houdek 53623ffa72 Merge pull request #1941 from Sonicadvance1/vdso_implementation
Thunks: Adds VDSO thunk library
2022-09-04 23:02:49 -07:00
Mai d6199687f4 Merge pull request #1970 from Sonicadvance1/fix_fresh_runner
Github: Fix fresh runner rootfs checkout
2022-09-02 21:49:22 -04:00
Ryan Houdek 7cb413dde6 Github: Fix fresh runner rootfs checkout
build folder doesn't exist on a freshly started runner.
Executing the rootfs fetch script doesn't care what the working
directory is. Doesn't need to be in `build/` which doesn't exist on a
fresh runner and will fail chdir.
2022-09-02 15:15:21 -07:00
Ryan Houdek c5a7fc1e2a Resolve most VDSO comments 2022-09-02 15:13:18 -07:00
Ryan Houdek 0869b0aa29 FEXLinuxTests: Adds a VDSO test
Ensures VDSO is working as correctly as it can.
2022-09-02 13:31:36 -07:00
Ryan Houdek 4e81f847f2 ELFCodeLoader: Load VDSO thunk at application startup
This ensures it will always be available to the application as long as
the library is installed.
2022-09-02 13:31:36 -07:00
Ryan Houdek 8fa27d2d9f Thunks: Adds VDSO thunk library
VDSO is heavily abused by Proton games to the point it is showing up as
CPU time.
Implement a guest-facing only thunk library using the hardcoded VDSO
interface in Thunks.

If available this will always be loaded on application load and set the
auxv value to support it.

This requires a bit of special treatment as our first user of linker
scripts since the format of the ELF must be careful crafted to not break
applications trying to parse it.

This library exposes a handful of symbols:
- clock_gettime
- clock_getres
- gettimeofday
- time
- getcpu
- All previous with `__vdso_` prefix
- LINUX_2.6

All of these symbols get routed directly to the host architecture VDSO
interface if they exist.
AArch64 doesn't have getcpu or time VDSO.

In a microbench, VDSO improved bench times substantially
x86-64 host: 3.612s -> 1.369s - 2.63x speed
AArch64 host: 3.821s -> 2.284s - 1.67x speed
  - AArch64 isn't as improved due to missing VDSO symbols

This is also our first /always/ enabled thunk as long as the file exists
2022-09-02 13:31:36 -07:00
Ryan Houdek e5a8a29efe Thunks: Add support for lds linker script on Guest libraries
This is going to be necessary in the next commit
2022-09-02 13:31:36 -07:00
Ryan Houdek f967f53176 Thunks: Adds VDSO specific thunks
x86-64 has five symbols within VDSO that we need to emulate.

Pass this through either glibc or host vdso if the symbol exists.
AArch64 doesn't have the time or getcpu vdso interface, so fall down
glibc instead.
2022-09-02 13:31:36 -07:00
Ryan Houdek 31fefaae0d Merge pull request #1969 from Sonicadvance1/fexrootfsfetcher_fix_crash
FEXRootFSFetcher: Fix crash if curl fails to download rootfs definition file
2022-09-02 11:07:03 -07:00
Ryan Houdek 7f9edbf39e Merge pull request #1968 from Sonicadvance1/new_domain
New domain.
2022-09-02 10:57:35 -07:00
Ryan Houdek 760b9c8e7f FEXRootFSFetcher: Fix crash if curl fails to download rootfs definition file
This can happen when the link goes down or internet blip.
2022-09-02 10:53:51 -07:00
Ryan Houdek cc7fb008fc New domain.
Needed to fix FEXRootFSFetcher from #1967
2022-09-02 10:43:07 -07:00
Ryan Houdek 98dbfbe654 Merge pull request #1949 from lioncash/interp-op
InterpreterOps: Extend SSAData size to accomodate 256-bit operations
2022-09-02 09:54:58 -07:00
lioncash 6444726614 InterpreterOps: Use designated initializer for IR op data
Same behavior, but keeps everything all initialized at the point of
declaration, rather than after the fact.
2022-09-02 12:40:25 -04:00
lioncash 0a562fbb10 InterpreterOps: Extend SSAData to handle 256-bit vectors 2022-09-02 12:40:14 -04:00
Ryan Houdek 7d8950de40 Merge pull request #1966 from lioncash/bool
Arm64/JIT: Rename CanUseSVE to HostSupportsSVE
2022-09-01 13:13:00 -07:00
lioncash 3fdde0c90b Arm64/JIT: Rename CanUseSVE to HostSupportsSVE
This is a much more descriptive name.

Spawned off of discussion in #1944
2022-09-01 13:22:41 -04:00
Ryan Houdek e776f4cd4e Merge pull request #1948 from lioncash/svebit
VectorOps: Extend VAnd/VBic/VOr/VXor
2022-08-30 17:57:51 -07:00
Ryan Houdek e7d7dd13d7 Merge pull request #1945 from lioncash/vectormov
VectorOps: Extend VMov
2022-08-30 17:57:00 -07:00
Ryan Houdek d5c83a2e45 Merge pull request #1950 from lioncash/x86run
HostRunner: Handle upper YMM lanes in sigsegv handler
2022-08-30 17:55:38 -07:00
Ryan Houdek 37ccb13917 Merge pull request #1946 from lioncash/x86dep
x86_64/JIT: Resolve lingering fmt deprecation warning
2022-08-30 17:54:57 -07:00
Ryan Houdek 8439cf410f Merge pull request #1944 from lioncash/sveimm
VectorOps: Extend VectorImm
2022-08-30 17:54:32 -07:00
lioncash 614d9448f7 HostRunner: Handle upper YMM lanes in sigsegv handler
Now we properly copy out the upper lanes instead of ignoring them.
2022-08-25 14:13:19 -04:00
lioncash 70efdbba5d VectorOps: Handle 256-bit VectorImm
Extends VectorImm to be capable of using SVE to handle 256-bit length
vectors.
2022-08-24 13:28:48 -04:00
Stefanos Kornilios Mitsis Poiitidis 12fee91ebb Merge pull request #1947 from neobrain/fix_tmpnam_warning
unittests/ThunkLibs: Fix warning about "dangerous" use of tmpnam
2022-08-24 20:13:04 +03:00
lioncash bcb7e20619 VectorOps: Handle 256-bit VXor 2022-08-24 12:52:05 -04:00
lioncash afc5e8a140 VectorOps: Handle 256-bit VOr 2022-08-24 12:52:05 -04:00
lioncash 4be6626c89 VectorOps: Handle 256-bit VBic 2022-08-24 12:52:02 -04:00
lioncash 5f2b6d629b VectorOps: Handle 256-bit VAnd 2022-08-24 12:43:51 -04:00
Tony Wasserka 1b8f5f08f0 unittests/ThunkLibs: Fix warning about "dangerous" use of tmpnam
tmpnam is considered insecure since it's vulnerable to TOCTOU issues.
This is not an issue for these tests, but replacing tmpnam is not any
more complicated than silencing the warning.
2022-08-24 18:15:51 +02:00
lioncash 79674c697a x86_64/JIT: Resolve lingering fmt deprecation warning
Just a log that was missed during the previous fmt deprecation cleanup.
2022-08-24 11:49:02 -04:00
lioncash 416d8c1df6 VectorOps: Handle 256-bit VMov
Kind of sucky that SVE doesn't have a convenient way to manipulate
predicate registers with immediates or anything to make this nicer (that
I know of).

Having to use a temp to clear the upper part of the vector reliably is
bleh.
2022-08-24 11:35:34 -04:00
Ryan Houdek d03b6a9382 Merge pull request #1942 from lioncash/zero
VectorOps: Extend VectorZero
2022-08-23 20:00:39 -07:00
lioncash df22e0c796 x86_64/JITClass: Add ToYMM helper
Will be used in subsequent changes to handle 256-bit operations in the
x86-64 backend
2022-08-23 12:58:25 -04:00
lioncash 79a3bd75cc VectorOps: Handle 256-bit VectorZero 2022-08-23 12:57:59 -04:00
Stefanos Kornilios Mitsis Poiitidis e6acdcc583 Merge pull request #1940 from neobrain/refactor_1868_cleanups
Thunks: Minor cleanups for signature-based function pointer thunking
2022-08-23 01:08:51 +03:00
Tony Wasserka cd78984228 Thunks: Define _M_X86_64/_M_ARM_64 when invoking thunkgen
This avoids the need to provide a fallback definition for platform-specific
macros. The definitions are only added host-side, since only Host.h is
included in any interface files.
2022-08-22 18:11:28 +02:00
Tony Wasserka 2f007c5f0a Thunks: Remove unused parameters of exports initializer 2022-08-22 18:11:28 +02:00
Tony Wasserka ae64a1e30c Thunks: Replace compiler-specific attributes with FEX_DEFAULT_VISIBILITY 2022-08-22 18:11:28 +02:00
Stefanos Kornilios Mitsis Poiitidis 097184c3e0 Merge pull request #1926 from Sonicadvance1/no_irloader_on_no_tests
IRLoader/TestHarnessLoader: Don't build if not building tests
2022-08-21 09:50:44 +03:00
Stefanos Kornilios Mitsis Poiitidis 84a95adae1 Merge pull request #1931 from Sonicadvance1/support_thunksdb_in_config
Thunks: Support direct thunk config in configuration files
2022-08-21 09:50:08 +03:00
Stefanos Kornilios Mitsis Poiitidis 123b6728e4 Merge pull request #1932 from Sonicadvance1/fix_allocator_perf_hit
64BitAllocator: Fixes a significant state tracking perf problem
2022-08-21 09:47:54 +03:00
Ryan Houdek 69b4fc98ed 64BitAllocator: Fixes a significant state tracking perf problem
Due to how the 64-bit allocator previously worked, it was never subjected to memory
regions larger than 64GB to be tracked. With the change in
PR #1885, this has changed to have regions that will hit sizes larger
than 170TB on some platforms.

Better yet, even with smaller regions it still had a performance issue,
it just wasn't as visible.

First problem: We used MemSet instead of MemClear for the live page
clearing. This caused pages to be claimed as "always in use".

This would cause us to always scan the entire region on allocation, find
that it didn't work and allocate a fresh region on every slab
allocation.
jemalloc saving us here since it allocates slabs from the OS fairly
aggressively.

Second problem: We used MemSet (now changed to MemClear) to "clear" the
state tracking for pages.

This causes ~600MB of memory to be used purely for state tracking.
This was physically backed since we were writing to every bit of
tracking for handling 256TB of VA.
This had a fault dance with the kernel for every new page being hit
here.
Instead of clearing the the bits with a memset, clear it with madvise so
it doesn't consume physical pages at all.

This means we use significantly less physical memory for 32-bit
applications.

With this change, pressure-vessel startup time goes from 24 seconds down
to 17 seconds. 70% of the original startup time.
But really the main savings here comes from the memory reduction that
PR #1885 ballooned, but has been an unseen problem before.

Before that PR we were burning 2MB of physical memory per region for no
reason.
After that PR we were burning up to 600MB of physical memory per region
for no reason. Changing a bit depending on how large the region ended up
being.

This now ends up being 2 pages starting out and grows as more pages are
are used. A significant improvement.
2022-08-20 15:32:40 -07:00
Ryan Houdek 35cf7703b1 Thunks: Support direct thunk config in configuration files
Previously in order to enable thunks, we needed an independent
description file of which thunks to be enabled. This is nice for quickly
testing out new games by setting `FEX_THUNKCONFIG` environment variable.

For users that just want to enable thunks this is an unwieldy
indirection that doesn't make much sense at a glance.

Previously this meant you needed two files as an example:
```
  ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat thunks.json
  {
    "ThunksDB": {
      "GL": 1,
      "Vulkan": 1
    }
  }
  ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
  {
    "Config": {
      "ThunkConfig":"~\/.fex-emu\/thunks.json"
    }
  }
```

Instead of this unwieldy redirection just support `ThunksDB` json
directly in the AppConfig.

```
  ryanh@ubuntu-linux-20-04-desktop:~/.fex-emu$ cat AppConfig/EnderLiliesSteam-Linux-Shipping.json
  {
    "Config": {
      <...>
    },
    "ThunksDB": {
      "GL": 1,
      "Vulkan": 1
    }
  }
```

As can be seen this makes this significantly easier for new users
getting in to thunks. Depending on which path to enable thunks the user
is more comfortable with, they can still enable them using the
`ThunkConfig` option or embedding directly in the application
configuration.

Additionally this removes the older non-ThunksDB path to loading thunks.
All users of it have moved on to using ThunksDB.
2022-08-20 10:02:13 -07:00
Ryan Houdek 8c1137543b Config: Adds APP_CONFIG_NAME meta config option
We have separate configurations for the Application path versus the
application name we are using as a configuration choice.

Example 1: FEXBash "wine Crysis64.exe"

Previous APP_FILENAME will contain `/usr/bin/wine`, which is still used
elsewhere.

This new APP_CONFIG_NAME will contain `Crysis64.exe`

Example 2: FEXBash glxgears

Previous APP_FILENAME will contain `/usr/bin/glxgears`
APP_CONFIG_NAME will contain `glxgears`

We didn't have this exposed any other way before.
2022-08-20 09:58:23 -07:00
Ryan Houdek b8e66e56a0 Config: Remove log message about Config file existing without Config json object 2022-08-20 09:58:04 -07:00
Ryan Houdek fbb008e510 Merge pull request #1929 from Sonicadvance1/support_variadic_struct_packing
Thunks/X11: Support Variadic stack packing
2022-08-20 06:31:22 -07:00
Ryan Houdek 04678f8404 Merge pull request #1885 from FEX-Emu/skmp/simpler-memory-stealing
Allocator: Simplify StealMemory, make it less chatty with kernel space
2022-08-20 06:20:36 -07:00
Stefanos Kornilios Mitsis Poiitidis 1b5aaf1fb8 Allocator: Simplify StealMemory, make it less chatty with kernel space 2022-08-20 13:10:50 +03:00
Ryan Houdek d8e4873f43 Thunks/X11: Support Variadic stack packing
Found an issue with wine + DXVK + thunks where these were passing in
more than 7 arguments and crashing.

Create some assembly to support any size of variadic stack packing.
Only implemented for AArch64 for now.
2022-08-19 21:52:19 -07:00
Ryan Houdek 998a3d8353 Merge pull request #1928 from Sonicadvance1/more_x11_thunks
Thunks/X11: Adds missing XLibint functions
2022-08-19 17:08:40 -07:00
Ryan Houdek 336dedbed5 Merge pull request #1927 from Sonicadvance1/non_fatal_get_fdpath
FDUtils: Don't make unknown get_fdpath fatal
2022-08-19 17:08:32 -07:00
Ryan Houdek 955595be8a FDUtils: Don't make unknown get_fdpath fatal
Encountered this while running wine things.
Non-fatal so don't explode
2022-08-19 02:28:19 -07:00
Ryan Houdek 576bd4f69a Thunks/X11: Adds missing XLibint functions
Some of these were required to get thunking to work with Proton and
DXVK.
2022-08-19 00:03:54 -07:00
Ryan Houdek cd6915917b IRLoader/TestHarnessLoader: Don't build if not building tests
If we're not building tests then just don't build the IRLoader since it
won't get used.
2022-08-17 17:04:13 -07:00
Ryan Houdek 0adbe31112 Merge pull request #1919 from Sonicadvance1/termux_shm_library
Termux: Add android-shmem library
2022-08-15 11:14:27 -07:00
Ryan Houdek d5138f509c Merge pull request #1917 from Sonicadvance1/fix_compile_without_jemalloc
Thunks: Fix compile without jemalloc
2022-08-15 11:14:17 -07:00
Ryan Houdek 1fe6fc3feb Merge pull request #1920 from Sonicadvance1/tests_from_host_features
unittests: Support skipping unit tests based on host feature support
2022-08-15 11:07:02 -07:00
Ryan Houdek c03a7fd482 Merge pull request #1924 from 1ace/fix-git-abbrev
cmake: fix incorrect assumption about the value of git's core.abbrev
2022-08-15 10:44:59 -07:00
Eric Engestrom 31c47f05f4 cmake: fix incorrect assumption about the value of git's core.abbrev
While the default value of `git config core.abbrev` is `7` (used to
truncate the commit hash in several git commands) and most people don't
configure anything else, I happen to prefer when commits hashes stay
valid for a while, so I changed that value to `20`. Because of this,
FEX fails to build in External/FEXCore/Source/Interface/Core/CPUID.cpp:961

Let's be explicit in the git command about what minimum length of commit
hash we expect.
2022-08-15 16:25:27 +01:00
Ryan Houdek edad24479b unittests: Support skipping unit tests based on host feature support
For these unit tests we no longer need to put them in the disabled tests
file. Instead it will be skipped if the host doesn't support the feature
required.
2022-08-14 20:04:26 -07:00
Ryan Houdek bba58732e0 HostFeatures: Add new supported features flag 2022-08-14 19:56:14 -07:00
Ryan Houdek 8d1cc6cf51 Termux: Add android-shmem library
Otherwise we fail at linking.
2022-08-14 18:23:53 -07:00
Ryan Houdek f03d0be5ef Thunks: Fix compile without jemalloc
This only fixes the compile error. Thunks aren't expected to be used
without jemalloc enabled.
2022-08-13 11:29:11 -07:00
Ryan Houdek a2f4f494a9 Merge pull request #1916 from Sonicadvance1/server_socket_path_override
FEXServer: Support socket path override
2022-08-13 07:00:59 -07:00
Ryan Houdek 9de25c200c Merge pull request #1915 from Sonicadvance1/emulate_64bit_getdents
Linux: Emulate classic getdents syscall for x64 and x32
2022-08-13 07:00:52 -07:00
Ryan Houdek fe1f00aadb FEXServer: Support socket path override
This is necessary for the fexserver to function correctly when chrooting
in to our rootfs and doing things.

Requires independent rootfs script modifications which will come with
the next rootfs update.

Problem comes down to a chroot supporting multiple users, where our
typical use case is only one user. Bind the server file to a single
server for the entire chroot session regardless of users, solving this
problem inside the chroot.

Fixes apt-get inside of chroot, which runs as user _apt.
2022-08-12 22:52:30 -07:00
Ryan Houdek a847fac4ca Linux: Emulate classic getdents syscall for x64 and x32
Arm64 doesn't have the classic getdents syscall, only getdents64.
Old glibc versions (like 2.17) don't support getdents64

This fixes 64-bit ls with a centos 7 rootfs. Likely also fixes some other very
old applications.

Packing differences between 32-bit and 64-bit getdents means we need to
template this between the two types, otherwise it's quite similar.

No known applications rely on the 32-bit getdents but worked with test
applications.
2022-08-12 18:21:49 -07:00
Ryan Houdek 0496506fb7 Linux: Define linux_dirent types for x64
These don't match the compatibility defines and they aren't part of the
public interface.
2022-08-12 16:00:06 -07:00
Ryan Houdek e544591c9c Merge pull request #1901 from Sonicadvance1/build_thunks
CI: Build Thunks
2022-08-12 14:47:54 -07:00
Ryan Houdek e5237d1149 Github: Enable Thunkgen tests 2022-08-12 14:31:18 -07:00
Ryan Houdek b9c848c5e9 Thunks: Xext version check define function prototypes
Nothing major here
2022-08-12 14:31:18 -07:00
Ryan Houdek b64e61a793 Thunks: Check for X11 version
There is no define for this so we must generate our own.
Declare a type ourselves if the library is too old
2022-08-12 14:31:18 -07:00
Ryan Houdek 249e19bf2a Thunks: Asound fix building with alsa 1.2.2 2022-08-09 06:18:43 -07:00
Ryan Houdek baf52ed286 CI: Try building thunks 2022-08-09 05:30:53 -07:00
333 changed files with 17240 additions and 7896 deletions

No files matched your search

+46 -3
View File
@@ -40,7 +40,6 @@ jobs:
# Use a bash shell so we can use the same syntax for environment variable
# access regardless of the host operating system
shell: bash
working-directory: ${{runner.workspace}}/build
run: $GITHUB_WORKSPACE/Scripts/CI_FetchRootFS.py
- name : submodule checkout
@@ -65,7 +64,7 @@ jobs:
# Note the current convention is to use the -S and -B options here to specify source
# and build directories, but this is only available with CMake 3.13 and higher.
# The CMake binaries on the Github Actions machines are (as of this writing) 3.12
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True -DENABLE_INTERPRETER=True -DBUILD_FEX_LINUX_TESTS=True
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True -DENABLE_INTERPRETER=True -DBUILD_FEX_LINUX_TESTS=True -DBUILD_THUNKS=True -DCMAKE_INSTALL_PREFIX=${{runner.workspace}}/build/install
- name: Build
working-directory: ${{runner.workspace}}/build
@@ -178,6 +177,51 @@ jobs:
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_FEXLinuxTests.log || true
- name: Thunkgen tests
working-directory: ${{runner.workspace}}/build
shell: bash
run: cmake --build . --config $BUILD_TYPE --target thunkgen_tests
- name: Thunkgen Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_ThunkgenTests.log || true
- name: Install
if: matrix.arch[1] == 'x64'
working-directory: ${{runner.workspace}}/build
shell: bash
run: cmake --build . --config $BUILD_TYPE --target install
- name: Test GL No-Thunks
if: matrix.arch[1] == 'x64'
working-directory: ${{runner.workspace}}/build
shell: bash
env:
DISPLAY: ":0"
run: cmake --build . --config $BUILD_TYPE --target thunk_functional_tests_nothunks
- name: No thunks Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_NoThunkResults.log || true
- name: Test GL Thunks
if: matrix.arch[1] == 'x64'
working-directory: ${{runner.workspace}}/build
shell: bash
env:
DISPLAY: ":0"
run: cmake --build . --config $BUILD_TYPE --target thunk_functional_tests_thunks
- name: Thunks Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_ThunkResults.log || true
- name: Truncate test results
if: ${{ always() }}
shell: bash
@@ -197,4 +241,3 @@ jobs:
name: Results-${{ env.runner_name }}
path: ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log
retention-days: 3
+118
View File
@@ -0,0 +1,118 @@
name: Vixl Simulator run
on:
push:
branches:
- main
pull_request:
branches:
- main
env:
# Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.)
BUILD_TYPE: Release
CC: clang
CXX: clang++
jobs:
build:
runs-on: ${{ matrix.arch }}
strategy:
matrix:
# Only the x86-64 runner is fast enough to run this
arch: [[self-hosted, x64], [self-hosted, ARMv8.4]]
fail-fast: false
steps:
- uses: actions/checkout@v2
- name: Set runner label
run: echo "runner_label=${{ matrix.arch[1] }}" >> $GITHUB_ENV
- name: Set rootfs paths
run: |
echo "FEX_ROOTFS_MOUNT=/mnt/AutoNFS/rootfs/" >> $GITHUB_ENV
echo "FEX_ROOTFS_PATH=$HOME/Rootfs/" >> $GITHUB_ENV
echo "FEX_ROOTFS=$HOME/Rootfs/" >> $GITHUB_ENV
echo "ROOTFS=$HOME/Rootfs/" >> $GITHUB_ENV
- name: Update RootFS cache
# Use a bash shell so we can use the same syntax for environment variable
# access regardless of the host operating system
shell: bash
run: $GITHUB_WORKSPACE/Scripts/CI_FetchRootFS.py
- name : submodule checkout
# Need to update submodules
run: |
git submodule sync --recursive
git submodule update --init --depth 1
- name: Clean Build Environment
run: rm -Rf ${{runner.workspace}}/build
- name: Create Build Environment
# Some projects don't allow in-source building, so create a separate build directory
# We'll use this as our working directory for all subsequent commands
run: cmake -E make_directory ${{runner.workspace}}/build
- name: Configure CMake
# Use a bash shell so we can use the same syntax for environment variable
# access regardless of the host operating system
shell: bash
working-directory: ${{runner.workspace}}/build
# Note the current convention is to use the -S and -B options here to specify source
# and build directories, but this is only available with CMake 3.13 and higher.
# The CMake binaries on the Github Actions machines are (as of this writing) 3.12
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_VIXL_SIMULATOR=True -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True
- name: Build
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the build. You can specify a specific target with "--target <NAME>"
run: cmake --build . --config $BUILD_TYPE
- name: ASM Tests
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the unit tests
run: cmake --build . --config $BUILD_TYPE --target asm_tests
- name: ASM Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_ASM.log || true
- name: IR Tests
working-directory: ${{runner.workspace}}/build
shell: bash
# Execute the unit tests
run: cmake --build . --config $BUILD_TYPE --target ir_tests
- name: IR Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_IR.log || true
- name: Truncate test results
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
# Cap out the log files at 20M in case something crash spins and dumps fault text
# ASM tests get quite close to 10MB
run: truncate --size=<20M ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log || true
- name: Set runner name
if: ${{ always() }}
run: echo "runner_name=$(hostname)" >> $GITHUB_ENV
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v2'
with:
name: Results-${{ env.runner_name }}
path: ${{runner.workspace}}/build/Testing/Temporary/LastTest_*.log
retention-days: 3
+5
View File
@@ -0,0 +1,5 @@
{
"ThunksDB": {
"GL": 1
}
}
+5
View File
@@ -0,0 +1,5 @@
{
"ThunksDB": {
"Vulkan": 1
}
}
+21
View File
@@ -0,0 +1,21 @@
if(NOT EXISTS "@CMAKE_BINARY_DIR@/install_manifest.txt")
message(FATAL_ERROR "Cannot find install manifest: @CMAKE_BINARY_DIR@/install_manifest.txt")
endif()
file(READ "@CMAKE_BINARY_DIR@/install_manifest.txt" files)
string(REGEX REPLACE "\n" ";" files "${files}")
foreach(file ${files})
message(STATUS "Uninstalling $ENV{DESTDIR}${file}")
if(IS_SYMLINK "$ENV{DESTDIR}${file}" OR EXISTS "$ENV{DESTDIR}${file}")
exec_program(
"@CMAKE_COMMAND@" ARGS "-E remove \"$ENV{DESTDIR}${file}\""
OUTPUT_VARIABLE rm_out
RETURN_VALUE rm_retval
)
if(NOT "${rm_retval}" STREQUAL 0)
message(FATAL_ERROR "Problem when removing $ENV{DESTDIR}${file}")
endif()
else(IS_SYMLINK "$ENV{DESTDIR}${file}" OR EXISTS "$ENV{DESTDIR}${file}")
message(STATUS "File $ENV{DESTDIR}${file} does not exist.")
endif()
endforeach()
+73 -7
View File
@@ -7,6 +7,7 @@ CHECK_INCLUDE_FILES ("gdb/jit-reader.h" HAVE_GDB_JIT_READER_H)
option(BUILD_TESTS "Build unit tests to ensure sanity" TRUE)
option(BUILD_FEX_LINUX_TESTS "Build FEXLinuxTests, requires x86 compiler" FALSE)
option(BUILD_THUNKS "Build thunks" FALSE)
option(BUILD_CLANG_THUNKS "Build thunks with clang" FALSE)
option(ENABLE_CLANG_FORMAT "Run clang format over the source" FALSE)
option(ENABLE_IWYU "Enables include what you use program" FALSE)
option(ENABLE_LTO "Enable LTO with compilation" TRUE)
@@ -27,10 +28,36 @@ option(ENABLE_LIBCXX "Enables LLVM libc++" FALSE)
option(ENABLE_INTERPRETER "Enables FEX's Interpreter" FALSE)
option(ENABLE_CCACHE "Enables ccache for compile caching" TRUE)
option(ENABLE_TERMUX_BUILD "Forces building for Termux on a non-Termux build machine" FALSE)
option(ENABLE_VIXL_SIMULATOR "Forces the FEX JIT to use the VIXL simulator" FALSE)
option(ENABLE_FEXCORE_PROFILER "Enables use of the FEXCore timeline profiling capabilities" FALSE)
set (FEXCORE_PROFILER_BACKEND "gpuvis" CACHE STRING "Set which backend you want to use for the FEXCore profiler")
set (X86_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain_x86.cmake" CACHE FILEPATH "Toolchain file for the x86 (cross-)compiler")
set (X86_32_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain_x86_32.cmake" CACHE FILEPATH "Toolchain file for the (cross-)compiler targeting i686")
set (X86_64_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain_x86_64.cmake" CACHE FILEPATH "Toolchain file for the (cross-)compiler targeting x86_64")
set (DATA_DIRECTORY "${CMAKE_INSTALL_PREFIX}/share/fex-emu" CACHE PATH "global data directory")
if (ENABLE_FEXCORE_PROFILER)
add_definitions(-DENABLE_FEXCORE_PROFILER=1)
string(TOUPPER "${FEXCORE_PROFILER_BACKEND}" FEXCORE_PROFILER_BACKEND)
if (FEXCORE_PROFILER_BACKEND STREQUAL "GPUVIS")
add_definitions(-DFEXCORE_PROFILER_BACKEND=1)
else()
message(FATAL_ERROR "Unknown FEXCore profiler backend ${FEXCORE_PROFILER_BACKEND}")
endif()
endif()
# uninstall target
if(NOT TARGET uninstall)
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/CMakeFiles/cmake_uninstall.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/cmake_uninstall.cmake"
IMMEDIATE @ONLY)
add_custom_target(uninstall
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/cmake_uninstall.cmake)
endif()
# These options are meant for package management
set (TUNE_CPU "native" CACHE STRING "Override the CPU the build is tuned for")
set (TUNE_ARCH "generic" CACHE STRING "Override the Arch the build is tuned for")
@@ -84,10 +111,9 @@ if (CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
set(_M_X86_64 1)
add_definitions(-D_M_X86_64=1)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcx16")
set (X86_TOOLCHAIN_FILE "")
endif()
if (CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64")
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^aarch64|^arm64|^armv8\.*")
set(_M_ARM_64 1)
add_definitions(-D_M_ARM_64=1)
endif()
@@ -363,8 +389,6 @@ if (BUILD_TESTS)
endif()
add_subdirectory(FEXHeaderUtils/)
include_directories(FEXHeaderUtils/)
add_subdirectory(External/FEXCore)
# Binfmt_misc files must be installed prior to Source/ installs
@@ -403,8 +427,28 @@ if (BUILD_THUNKS)
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ThunkLibs/GuestLibs"
BINARY_DIR "Guest"
CMAKE_ARGS
"-DBITNESS=64"
"-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
"-DCMAKE_TOOLCHAIN_FILE:FILEPATH=${X86_TOOLCHAIN_FILE}"
"-DENABLE_CLANG_THUNKS=${ENABLE_CLANG_THUNKS}"
"-DCMAKE_TOOLCHAIN_FILE:FILEPATH=${X86_64_TOOLCHAIN_FILE}"
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DSTRUCT_VERIFIER=${CMAKE_SOURCE_DIR}/Scripts/StructPackVerifier.py"
"-DFEX_PROJECT_SOURCE_DIR=${FEX_PROJECT_SOURCE_DIR}"
"-DGENERATOR_EXE=$<TARGET_FILE:thunkgen>"
INSTALL_COMMAND ""
BUILD_ALWAYS ON
DEPENDS thunkgen
)
ExternalProject_Add(guest-libs-32
PREFIX guest-libs-32
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ThunkLibs/GuestLibs"
BINARY_DIR "Guest_32"
CMAKE_ARGS
"-DBITNESS=32"
"-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
"-DENABLE_CLANG_THUNKS=${ENABLE_CLANG_THUNKS}"
"-DCMAKE_TOOLCHAIN_FILE:FILEPATH=${X86_32_TOOLCHAIN_FILE}"
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DSTRUCT_VERIFIER=${CMAKE_SOURCE_DIR}/Scripts/StructPackVerifier.py"
"-DFEX_PROJECT_SOURCE_DIR=${FEX_PROJECT_SOURCE_DIR}"
@@ -422,6 +466,28 @@ if (BUILD_THUNKS)
)"
DEPENDS guest-libs
)
install(
CODE "MESSAGE(\"-- Installing: guest-libs-32\")"
CODE "
EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} --build . --target install
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/Guest_32
)"
DEPENDS guest-libs-32
)
add_custom_target(uninstall_guest-libs
COMMAND ${CMAKE_COMMAND} "--build" "." "--target" "uninstall"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/Guest
)
add_custom_target(uninstall_guest-libs-32
COMMAND ${CMAKE_COMMAND} "--build" "." "--target" "uninstall"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/Guest_32
)
add_dependencies(uninstall uninstall_guest-libs)
add_dependencies(uninstall uninstall_guest-libs-32)
endif()
set(FEX_VERSION_MAJOR "0")
@@ -476,7 +542,7 @@ endif()
set (CPACK_GENERATOR "DEB")
set (CPACK_PACKAGE_NAME fex-emu)
set (CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${GIT_DESCRIBE_STRING}_${CMAKE_SYSTEM_PROCESSOR}")
set (CPACK_PACKAGE_CONTACT "FEX-Emu Maintainers <team@fex-emu.org>")
set (CPACK_PACKAGE_CONTACT "FEX-Emu Maintainers <team@fex-emu.com>")
set (CPACK_PACKAGE_VERSION_MAJOR "${FEX_VERSION_MAJOR}")
set (CPACK_PACKAGE_VERSION_MINOR "${FEX_VERSION_MINOR}")
set (CPACK_PACKAGE_VERSION_PATCH "${FEX_VERSION_PATCH}")
+1 -1
View File
@@ -55,7 +55,7 @@ further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at team@fex-emu.org. All
reported by contacting the project team at team@fex-emu.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
+8
View File
@@ -165,6 +165,14 @@
"@PREFIX_LIB@/x86_64-linux-gnu/libXfixes.so.3.1.0"
]
},
"OpenCL": {
"Library" : "libOpenCL-guest.so",
"Overlay": [
"@PREFIX_LIB@/x86_64-linux-gnu/libOpenCL.so",
"@PREFIX_LIB@/x86_64-linux-gnu/libOpenCL.so.1",
"@PREFIX_LIB@/x86_64-linux-gnu/libOpenCL.so.1.0.0"
]
},
"":{}
}
}
+12 -6
View File
@@ -9,12 +9,19 @@ if (CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcx16")
endif()
if (CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64")
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^aarch64|^arm64|^armv8\.*")
set(_M_ARM_64 1)
endif()
set(ENABLE_JIT_X86_64 ${_M_X86_64} CACHE BOOL "Enable the x86_64 JIT")
set(ENABLE_JIT_ARM64 ${_M_ARM_64} CACHE BOOL "Enable the ARM64 JIT")
if (ENABLE_VIXL_SIMULATOR)
# If the vixl simulator is enabled then we are using the ARM64 JIT
option(ENABLE_JIT_X86_64 "Enable the x86_64 JIT" FALSE)
option(ENABLE_JIT_ARM64 "Enable the ARM64 JIT" TRUE)
else()
option(ENABLE_JIT_X86_64 "Enable the x86_64 JIT" ${_M_X86_64})
option(ENABLE_JIT_ARM64 "Enable the ARM64 JIT" ${_M_ARM_64})
endif()
option(ENABLE_CLANG_FORMAT "Run clang format over the source" FALSE)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
@@ -27,7 +34,6 @@ set(CMAKE_INCLUDE_CURRENT_DIR ON)
include(CheckCXXCompilerFlag)
include(CheckIncludeFileCXX)
if (EXISTS ${CMAKE_CURRENT_DIR}/External/vixl/)
# Useful to have for freestanding libFEXCore
add_subdirectory(External/vixl/)
@@ -46,14 +52,14 @@ if (OVERRIDE_VERSION STREQUAL "detect")
if (GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
COMMAND ${GIT_EXECUTABLE} rev-parse --short=7 HEAD
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_SHORT_HASH
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe
COMMAND ${GIT_EXECUTABLE} describe --abbrev=7
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_DESCRIBE_STRING
ERROR_QUIET
+10 -4
View File
@@ -143,6 +143,7 @@ set (SRCS
Utils/NetStream.cpp
Utils/Telemetry.cpp
Utils/Threads.cpp
Utils/Profiler.cpp
)
if (ENABLE_INTERPRETER)
@@ -177,6 +178,11 @@ if (_M_ARM_64)
list(APPEND DEFINES -D_M_ARM_64=1)
endif()
if (ENABLE_VIXL_SIMULATOR)
# We can run the simulator on both x86-64 or AArch64 hosts
list(APPEND DEFINES -DVIXL_SIMULATOR=1 -DVIXL_INCLUDE_SIMULATOR_AARCH64=1)
endif()
if (ENABLE_JIT_X86_64)
list(APPEND SRCS
Interface/Core/JIT/x86_64/JIT.cpp
@@ -213,7 +219,7 @@ if (ENABLE_JIT_ARM64)
)
endif()
set (LIBS vixl dl xxhash tiny-json)
set (LIBS fmt::fmt vixl dl xxhash tiny-json FEXHeaderUtils)
if (ENABLE_JEMALLOC)
list (APPEND LIBS FEX_jemalloc)
endif()
@@ -359,14 +365,14 @@ endfunction()
# Build FEXCore_Config static library
add_library(FEXCore_Base STATIC ${FEXCORE_BASE_SRCS})
target_link_libraries(FEXCore_Base fmt::fmt tiny-json)
target_link_libraries(FEXCore_Base ${LIBS})
AddDefaultOptionsToTarget(FEXCore_Base)
function(AddObject Name Type)
add_library(${Name} ${Type} ${SRCS})
add_dependencies(${Name} IR_INC)
target_link_libraries(${Name} FEXCore_Base ${LIBS})
target_link_libraries(${Name} FEXCore_Base)
AddDefaultOptionsToTarget(${Name})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
@@ -374,7 +380,7 @@ endfunction()
function(AddLibrary Name Type)
add_library(${Name} ${Type} $<TARGET_OBJECTS:${PROJECT_NAME}_object>)
target_link_libraries(${Name} FEXCore_Base ${LIBS})
target_link_libraries(${Name} FEXCore_Base)
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
AddDefaultOptionsToTarget(${Name})
+1 -1
View File
@@ -82,7 +82,7 @@ namespace JSON {
json_t const* ConfigList = json_getProperty(json, "Config");
if (!ConfigList) {
LogMan::Msg::EFmt("Couldn't get config list");
// This is a non-error if the configuration file exists but no Config section
return;
}
+25
View File
@@ -90,6 +90,13 @@
"Folder to find the guest-side thunking libraries."
]
},
"ThunkGuestLibs32": {
"Type": "str",
"Default": "@CMAKE_INSTALL_PREFIX@/share/fex-emu/GuestThunks_32/",
"Desc": [
"Folder to find the 32-bit guest-side thunking libraries."
]
},
"ThunkConfig": {
"Type": "str",
"Default": "",
@@ -351,6 +358,13 @@
"Desc": [
"Loads an AOT IR cache for the loaded executable."
]
},
"ServerSocketPath": {
"Type": "str",
"Default": "",
"Desc": [
"Override for a FEXServer socket path. Only useful for chroots."
]
}
}
},
@@ -368,6 +382,17 @@
"Type": "str",
"Default": ""
},
"APP_CONFIG_NAME": {
"Type": "str",
"Default": "",
"Desc": [
"This is the application config name that has been loaded.",
"This differs from APP_FILENAME in two ways",
"Where APP_FILENAME always points to the executable path that FEX-Emu is executing.",
"This matches what is used to load the AppLayer configuration name.",
"When running through a compatibility layer like wine, this will only be the exe name, instead of wine full path."
]
},
"IS64BIT_MODE": {
"Type": "bool",
"Default": "false"
+4
View File
@@ -202,6 +202,10 @@ namespace FEXCore::Context {
return CTX->AddCustomIREntrypoint(Entrypoint, Handler, Creator, Data);
}
void AppendThunkDefinitions(FEXCore::Context::Context *CTX, std::vector<FEXCore::IR::ThunkDefinition> const& Definitions) {
CTX->AppendThunkDefinitions(Definitions);
}
namespace Debug {
void CompileRIP(FEXCore::Context::Context *CTX, uint64_t RIP) {
CTX->CompileRIP(CTX->ParentThread, RIP);
+4 -1
View File
@@ -15,6 +15,7 @@
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include <FEXCore/Utils/Event.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <stdint.h>
@@ -189,7 +190,7 @@ namespace FEXCore::Context {
static void ThreadRemoveCodeEntryFromJit(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP) {
auto Thread = Frame->Thread;
LogMan::Throw::AFmt(Thread->ThreadManager.GetTID() == gettid(), "Must be called from owning thread {}, not {}", Thread->ThreadManager.GetTID(), gettid());
LogMan::Throw::AFmt(Thread->ThreadManager.GetTID() == FHU::Syscalls::gettid(), "Must be called from owning thread {}, not {}", Thread->ThreadManager.GetTID(), FHU::Syscalls::gettid());
FHU::ScopedSignalMaskWithUniqueLock lk(Thread->CTX->CodeInvalidationMutex);
@@ -322,6 +323,8 @@ namespace FEXCore::Context {
IRCaptureCache.SetAOTIRRenamer(CacheRenamer);
}
void AppendThunkDefinitions(std::vector<FEXCore::IR::ThunkDefinition> const& Definitions);
FEXCore::Utils::PooledAllocatorMMap OpDispatcherAllocator;
FEXCore::Utils::PooledAllocatorMMap FrontendAllocator;
@@ -17,23 +17,35 @@
#include <utility>
namespace FEXCore::CPU {
#define STATE x28
// We want vixl to not allocate a default buffer. Jit and dispatcher will manually create one.
Arm64Emitter::Arm64Emitter(FEXCore::Context::Context *ctx, size_t size)
: vixl::aarch64::Assembler(size, vixl::aarch64::PositionDependentCode)
: vixl::aarch64::Assembler(size ? (byte*)FEXCore::Allocator::mmap(nullptr, size, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0) : reinterpret_cast<byte*>(~0ULL),
size,
vixl::aarch64::PositionDependentCode)
, EmitterCTX {ctx} {
CPU.SetUp();
#ifdef VIXL_SIMULATOR
auto Features = vixl::CPUFeatures::All();
#else
auto Features = vixl::CPUFeatures::InferFromOS();
if (ctx->HostFeatures.SupportsAtomics) {
// Hypervisor can hide this on the c630?
Features.Combine(vixl::CPUFeatures::Feature::kLORegions);
}
#endif
SetCPUFeatures(Features);
}
Arm64Emitter::~Arm64Emitter() {
auto CodeBuffer = GetBuffer();
if (CodeBuffer->GetCapacity()) {
FEXCore::Allocator::munmap(CodeBuffer->GetStartAddress<void*>(), CodeBuffer->GetCapacity());
}
}
void Arm64Emitter::LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant, bool NOPPad) {
bool Is64Bit = Reg.IsX();
int Segments = Is64Bit ? 4 : 2;
@@ -214,7 +226,8 @@ void Arm64Emitter::SpillStaticRegs(bool FPRs, uint32_t GPRSpillMask, uint32_t FP
const auto Reg = SRAFPR[i];
if (((1U << Reg.GetCode()) & FPRSpillMask) != 0) {
str(Reg.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm.avx.data[i][0])));
mov(TMP4, offsetof(Core::CpuStateFrame, State.xmm.avx.data[i][0]));
st1b(Reg.Z().VnB(), PRED_TMP_32B, SVEMemOperand(STATE, TMP4));
}
}
} else {
@@ -257,11 +270,19 @@ void Arm64Emitter::FillStaticRegs(bool FPRs, uint32_t GPRFillMask, uint32_t FPRF
if (FPRs) {
if (EmitterCTX->HostFeatures.SupportsAVX) {
// Set up predicate registers.
// We don't bother spilling these in SpillStaticRegs,
// since all that matters is we restore them on a fill.
// It's not a concern if they get trounced by something else.
ptrue(PRED_TMP_16B.VnB(), SVE_VL16);
ptrue(PRED_TMP_32B.VnB(), SVE_VL32);
for (size_t i = 0; i < SRAFPR.size(); i++) {
const auto Reg = SRAFPR[i];
if (((1U << Reg.GetCode()) & FPRFillMask) != 0) {
ldr(Reg.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm.avx.data[i][0])));
mov(TMP4, offsetof(Core::CpuStateFrame, State.xmm.avx.data[i][0]));
ld1b(Reg.Z().VnB(), PRED_TMP_32B.Zeroing(), SVEMemOperand(STATE, TMP4));
}
}
} else {
@@ -286,20 +307,31 @@ void Arm64Emitter::FillStaticRegs(bool FPRs, uint32_t GPRFillMask, uint32_t FPRF
}
void Arm64Emitter::PushDynamicRegsAndLR() {
uint64_t SPOffset = AlignUp((RA64.size() + 1) * 8 + RAFPR.size() * 16, 16);
const auto CanUseSVE = EmitterCTX->HostFeatures.SupportsAVX;
const auto GPRSize = (RA64.size() + 1) * Core::CPUState::GPR_REG_SIZE;
const auto FPRRegSize = CanUseSVE ? Core::CPUState::XMM_AVX_REG_SIZE
: Core::CPUState::XMM_SSE_REG_SIZE;
const auto FPRSize = RAFPR.size() * FPRRegSize;
const uint64_t SPOffset = AlignUp(GPRSize + FPRSize, 16);
sub(sp, sp, SPOffset);
int i = 0;
for (auto RA : RAFPR)
{
str(RA.Q(), MemOperand(sp, i * 8));
i+=2;
if (CanUseSVE) {
for (const auto& RA : RAFPR) {
mov(TMP4, i * 8);
st1b(RA.Z().VnB(), PRED_TMP_32B, SVEMemOperand(sp, TMP4));
i += 4;
}
} else {
for (const auto& RA : RAFPR) {
str(RA.Q(), MemOperand(sp, i * 8));
i += 2;
}
}
#if 0 // All GPRs should be caller saved
for (auto RA : RA64)
{
for (const auto& RA : RA64) {
str(RA, MemOperand(sp, i * 8));
i++;
}
@@ -309,18 +341,29 @@ void Arm64Emitter::PushDynamicRegsAndLR() {
}
void Arm64Emitter::PopDynamicRegsAndLR() {
uint64_t SPOffset = AlignUp((RA64.size() + 1) * 8 + RAFPR.size() * 16, 16);
const auto CanUseSVE = EmitterCTX->HostFeatures.SupportsAVX;
const auto GPRSize = (RA64.size() + 1) * Core::CPUState::GPR_REG_SIZE;
const auto FPRRegSize = CanUseSVE ? Core::CPUState::XMM_AVX_REG_SIZE
: Core::CPUState::XMM_SSE_REG_SIZE;
const auto FPRSize = RAFPR.size() * FPRRegSize;
const uint64_t SPOffset = AlignUp(GPRSize + FPRSize, 16);
int i = 0;
for (auto RA : RAFPR)
{
ldr(RA.Q(), MemOperand(sp, i * 8));
i+=2;
if (CanUseSVE) {
for (const auto& RA : RAFPR) {
mov(TMP4, i * 8);
ld1b(RA.Z().VnB(), PRED_TMP_32B.Zeroing(), SVEMemOperand(sp, TMP4));
i += 4;
}
} else {
for (const auto& RA : RAFPR) {
ldr(RA.Q(), MemOperand(sp, i * 8));
i += 2;
}
}
#if 0 // All GPRs should be caller saved
for (auto RA : RA64)
{
for (const auto& RA : RA64) {
ldr(RA, MemOperand(sp, i * 8));
i++;
}
@@ -8,6 +8,10 @@
#include <aarch64/cpu-aarch64.h>
#include <aarch64/operands-aarch64.h>
#include <platform-vixl.h>
#ifdef VIXL_SIMULATOR
#include <aarch64/simulator-aarch64.h>
#include <aarch64/simulator-constants-aarch64.h>
#endif
#include <FEXCore/Config/Config.h>
@@ -58,15 +62,41 @@ const std::array<aarch64::VRegister, 12> RAFPR = {
v8, v9, v10, v11, v12, v13, v14, v15
};
// Contains the address to the currently available CPU state
#define STATE x28
// GPR temporaries. Only x3 can be used across spill boundaries
// so if these ever need to change, be very careful about that.
#define TMP1 x0
#define TMP2 x1
#define TMP3 x2
#define TMP4 x3
// Vector temporaries
#define VTMP1 v1
#define VTMP2 v2
#define VTMP3 v3
// Predicate register temporaries (used when AVX support is enabled)
// PRED_TMP_16B indicates a predicate register that indicates the first 16 bytes set to 1.
// PRED_TMP_32B indicates a predicate register that indicates the first 32 bytes set to 1.
#define PRED_TMP_16B p6
#define PRED_TMP_32B p7
// This class contains common emitter utility functions that can
// be used by both Arm64 JIT and ARM64 Dispatcher
class Arm64Emitter : public vixl::aarch64::Assembler {
protected:
Arm64Emitter(FEXCore::Context::Context *ctx, size_t size);
~Arm64Emitter();
FEXCore::Context::Context *EmitterCTX;
vixl::aarch64::CPU CPU;
void LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant, bool NOPPad = false);
// NOTE: These functions WILL clobber the register TMP4 if AVX support is enabled
// and FPRs are being spilled or filled. If only GPRs are spilled/filled, then
// TMP4 is left alone.
void SpillStaticRegs(bool FPRs = true, uint32_t GPRSpillMask = ~0U, uint32_t FPRSpillMask = ~0U);
void FillStaticRegs(bool FPRs = true, uint32_t GPRFillMask = ~0U, uint32_t FPRFillMask = ~0U);
@@ -83,6 +113,71 @@ protected:
void PopCalleeSavedRegisters();
void Align16B();
#ifdef VIXL_SIMULATOR
// Generates a vixl simulator runtime call.
//
// This matches behaviour of vixl's macro assembler, but we need to reimplement it since we aren't using the macro assembler.
// This isn't too complex with how vixl emits this.
//
// Emit:
// 1) hlt(kRuntimeCallOpcode)
// 2) Simulator wrapper handler
// 3) Function to call
// 4) Style of the function call (Call versus tail-call)
template<typename R, typename... P>
void GenerateRuntimeCall(R (*Function)(P...)) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
uintptr_t FunctionAddress = reinterpret_cast<uintptr_t>(Function);
hlt(kRuntimeCallOpcode);
// Simulator wrapper address pointer.
dc(SimulatorWrapperAddress);
// Runtime function address to call
dc(FunctionAddress);
// Call type
dc32(kCallRuntime);
}
template<typename R, typename... P>
void GenerateIndirectRuntimeCall(vixl::aarch64::Register Reg) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(Simulator::RuntimeCallStructHelper<R, P...>::Wrapper));
hlt(kIndirectRuntimeCallOpcode);
// Simulator wrapper address pointer.
dc(SimulatorWrapperAddress);
// Register that contains the function to call
dc(Reg.GetCode());
// Call type
dc32(kCallRuntime);
}
template<>
void GenerateIndirectRuntimeCall<float, __uint128_t>(vixl::aarch64::Register Reg) {
uintptr_t SimulatorWrapperAddress = reinterpret_cast<uintptr_t>(
&(Simulator::RuntimeCallStructHelper<float, __uint128_t>::Wrapper));
hlt(kIndirectRuntimeCallOpcode);
// Simulator wrapper address pointer.
dc(SimulatorWrapperAddress);
// Register that contains the function to call
dc(Reg.GetCode());
// Call type
dc32(kCallRuntime);
}
#endif
FEX_CONFIG_OPT(StaticRegisterAllocation, SRA);
};
+24 -13
View File
@@ -44,6 +44,7 @@ $end_info$
#include <FEXCore/Utils/Event.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Threads.h>
#include <FEXCore/Utils/Profiler.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <FEXHeaderUtils/TodoDefines.h>
@@ -221,7 +222,7 @@ namespace FEXCore::Context {
#if (_M_X86_64 && JIT_X86_64)
FEXCore::CPU::InitializeX86JITSignalHandlers(this);
BackendFeatures = FEXCore::CPU::GetX86JITBackendFeatures();
#elif (_M_ARM_64 && JIT_ARM64)
#elif (_M_ARM_64 && JIT_ARM64) || defined(VIXL_SIMULATOR)
FEXCore::CPU::InitializeArm64JITSignalHandlers(this);
BackendFeatures = FEXCore::CPU::GetArm64JITBackendFeatures();
#else
@@ -238,16 +239,16 @@ namespace FEXCore::Context {
DispatcherConfig.StaticRegisterAllocation = Config.StaticRegisterAllocation && BackendFeatures.SupportsStaticRegisterAllocation;
#if (_M_X86_64)
Dispatcher = FEXCore::CPU::Dispatcher::CreateX86(this, DispatcherConfig);
#elif (_M_ARM_64)
#if JIT_ARM64
Dispatcher = FEXCore::CPU::Dispatcher::CreateArm64(this, DispatcherConfig);
#elif JIT_X86_64
Dispatcher = FEXCore::CPU::Dispatcher::CreateX86(this, DispatcherConfig);
#else
ERROR_AND_DIE_FMT("FEXCore has been compiled with an unknown target");
#endif
// Initialize common signal handlers
auto PauseHandler = [](FEXCore::Core::InternalThreadState *Thread, int Signal, void *info, void *ucontext) -> bool {
return Thread->CTX->Dispatcher->HandleSignalPause(Thread, Signal, info, ucontext);
};
@@ -573,7 +574,7 @@ namespace FEXCore::Context {
#if (_M_X86_64 && JIT_X86_64)
Thread->CPUBackend = FEXCore::CPU::CreateX86JITCore(this, Thread);
#elif (_M_ARM_64 && JIT_ARM64)
#elif (_M_ARM_64 && JIT_ARM64) || defined(VIXL_SIMULATOR)
Thread->CPUBackend = FEXCore::CPU::CreateArm64JITCore(this, Thread);
#else
ERROR_AND_DIE_FMT("FEXCore has been compiled without a viable JIT core");
@@ -674,6 +675,8 @@ namespace FEXCore::Context {
}
void Context::ClearCodeCache(FEXCore::Core::InternalThreadState *Thread) {
FEXCORE_PROFILE_INSTANT("ClearCodeCache");
{
// Ensure the Code Object Serialization service has fully serialized this thread's data before clearing the cache
// Use the thread's object cache ref counter for this
@@ -740,7 +743,9 @@ namespace FEXCore::Context {
}
}
Context::GenerateIRResult Context::GenerateIR(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP, bool ExtendedDebugInfo) {
Context::GenerateIRResult Context::GenerateIR(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP, bool ExtendedDebugInfo) {
FEXCORE_PROFILE_SCOPED("GenerateIR");
Thread->OpDispatcher->ReownOrClaimBuffer();
Thread->OpDispatcher->ResetWorkingList();
@@ -749,7 +754,7 @@ namespace FEXCore::Context {
std::shared_lock lk(CustomIRMutex);
auto Handler = CustomIRHandlers.find(GuestRIP);
if (Handler != CustomIRHandlers.end()) {
TotalInstructions = 1;
@@ -872,7 +877,7 @@ namespace FEXCore::Context {
}
}
}
Thread->OpDispatcher->Finalize();
Thread->FrontendDecoder->DelayedDisownBuffer();
@@ -1011,6 +1016,7 @@ namespace FEXCore::Context {
}
uintptr_t Context::CompileBlock(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP) {
FEXCORE_PROFILE_SCOPED("CompileBlock");
auto Thread = Frame->Thread;
// Invalidate might take a unique lock on this, to guarantee that during invalidation no code gets compiled
@@ -1182,7 +1188,7 @@ namespace FEXCore::Context {
static void InvalidateGuestCodeRangeInternal(FEXCore::Context::Context *CTX, uint64_t Start, uint64_t Length) {
std::lock_guard lk(CTX->ThreadCreationMutex);
for (auto &Thread : CTX->Threads) {
InvalidateGuestThreadCodeRange(Thread, Start, Length);
}
@@ -1190,7 +1196,7 @@ namespace FEXCore::Context {
void InvalidateGuestCodeRange(FEXCore::Context::Context *CTX, uint64_t Start, uint64_t Length) {
FHU::ScopedSignalMaskWithUniqueLock CodeInvalidationLock(CTX->CodeInvalidationMutex);
InvalidateGuestCodeRangeInternal(CTX, Start, Length);
}
@@ -1233,9 +1239,9 @@ namespace FEXCore::Context {
Thread->LookupCache->AddBlockLink(GuestDestination, HostLink, delinker);
}
void Context::ThreadRemoveCodeEntry(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) {
void Context::ThreadRemoveCodeEntry(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) {
LogMan::Throw::AFmt(Thread->CTX->CodeInvalidationMutex.try_lock() == false, "CodeInvalidationMutex needs to be unique_locked here");
std::lock_guard<std::recursive_mutex> lk(Thread->LookupCache->WriteLock);
Thread->DebugStore.erase(GuestRIP);
@@ -1332,6 +1338,11 @@ namespace FEXCore::Context {
}
}
void Context::AppendThunkDefinitions(std::vector<FEXCore::IR::ThunkDefinition> const& Definitions) {
ThunkHandler->AppendThunkDefinitions(Definitions);
}
void ConfigureAOTGen(FEXCore::Core::InternalThreadState *Thread, std::set<uint64_t> *ExternalBranches, uint64_t SectionMaxAddress) {
Thread->FrontendDecoder->SetExternalBranches(ExternalBranches);
Thread->FrontendDecoder->SetSectionMaxAddress(SectionMaxAddress);
@@ -38,11 +38,20 @@ namespace FEXCore::CPU {
using namespace vixl;
using namespace vixl::aarch64;
static constexpr size_t MAX_DISPATCHER_CODE_SIZE = 4096;
#define STATE x28
constexpr size_t MAX_DISPATCHER_CODE_SIZE = 8192;
Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const DispatcherConfig &config)
: FEXCore::CPU::Dispatcher(ctx, config), Arm64Emitter(ctx, MAX_DISPATCHER_CODE_SIZE) {
: FEXCore::CPU::Dispatcher(ctx, config), Arm64Emitter(ctx, MAX_DISPATCHER_CODE_SIZE)
#ifdef VIXL_SIMULATOR
, Simulator {&Decoder}
#endif
{
#ifdef VIXL_SIMULATOR
// Hardcode a 256-bit vector width if we are running in the simulator.
Simulator.SetVectorLengthInBits(256);
#endif
SetAllowAssembler(true);
DispatchPtr = GetCursorAddress<AsmDispatch>();
@@ -178,7 +187,12 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
ret();
}
#ifdef VIXL_SIMULATOR
// VIXL simulator can't run syscalls.
constexpr bool SignalSafeCompile = false;
#else
constexpr bool SignalSafeCompile = true;
#endif
{
ExitFunctionLinkerAddress = GetCursorAddress<uint64_t>();
if (config.StaticRegisterAllocation)
@@ -206,8 +220,12 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
mov(x0, STATE);
mov(x1, lr);
ldr(x3, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionLink));
blr(x3);
ldr(x2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionLink));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uintptr_t, void *, void *>(x2);
#else
blr(x2);
#endif
if (SignalSafeCompile) {
// Now restore the signal mask
@@ -266,8 +284,11 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
ldr(x3, &l_CompileBlock);
// X2 contains our guest RIP
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<void, void *, uint64_t, void *>(x3);
#else
blr(x3); // { CTX, Frame, RIP}
#endif
if (SignalSafeCompile) {
// Now restore the signal mask
@@ -349,7 +370,11 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
ldr(x0, &l_CTX);
mov(x1, STATE);
ldr(x2, &l_Sleep);
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<void, void *, void *>(x2);
#else
blr(x2);
#endif
PauseReturnInstruction = GetCursorAddress<uint64_t>();
// Fault to start running again
@@ -412,11 +437,14 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
LUDIVHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR();
SpillStaticRegs();
ldr(x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LUDIV));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(x3);
#else
blr(x3);
#endif
FillStaticRegs();
// Result is now in x0
@@ -431,11 +459,14 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
LDIVHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR();
SpillStaticRegs();
ldr(x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LDIV));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(x3);
#else
blr(x3);
#endif
FillStaticRegs();
// Result is now in x0
@@ -450,11 +481,14 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
LUREMHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR();
SpillStaticRegs();
ldr(x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LUREM));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(x3);
#else
blr(x3);
#endif
FillStaticRegs();
// Result is now in x0
@@ -469,11 +503,14 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
LREMHandlerAddress = GetCursorAddress<uint64_t>();
PushDynamicRegsAndLR();
SpillStaticRegs();
ldr(x3, STATE_PTR(CpuStateFrame, Pointers.AArch64.LREM));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t>(x3);
#else
blr(x3);
#endif
FillStaticRegs();
// Result is now in x0
@@ -504,13 +541,27 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, const Dispatche
}
}
#ifdef VIXL_SIMULATOR
void Arm64Dispatcher::ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) {
Simulator.WriteXRegister(0, reinterpret_cast<int64_t>(Frame));
Simulator.RunFrom(reinterpret_cast<Instruction const*>(DispatchPtr));
}
void Arm64Dispatcher::ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) {
Simulator.WriteXRegister(0, reinterpret_cast<int64_t>(Frame));
Simulator.WriteXRegister(1, RIP);
Simulator.RunFrom(reinterpret_cast<Instruction const*>(CallbackPtr));
}
#endif
// Used by GenerateGDBPauseCheck, GenerateInterpreterTrampoline, destination buffer is set before use
static thread_local vixl::aarch64::Assembler emit((uint8_t*)&emit, 1);
size_t Arm64Dispatcher::GenerateGDBPauseCheck(uint8_t *CodeBuffer, uint64_t GuestRIP) {
*emit.GetBuffer() = vixl::CodeBuffer(CodeBuffer, MaxGDBPauseCheckSize);
vixl::CodeBufferCheckScope scope(&emit, MaxGDBPauseCheckSize, vixl::CodeBufferCheckScope::kDontReserveBufferSpace, vixl::CodeBufferCheckScope::kNoAssert);
aarch64::Label RunBlock;
@@ -546,7 +597,7 @@ size_t Arm64Dispatcher::GenerateGDBPauseCheck(uint8_t *CodeBuffer, uint64_t Gues
size_t Arm64Dispatcher::GenerateInterpreterTrampoline(uint8_t *CodeBuffer) {
LOGMAN_THROW_AA_FMT(!config.StaticRegisterAllocation, "GenerateInterpreterTrampoline dispatcher does not support SRA");
*emit.GetBuffer() = vixl::CodeBuffer(CodeBuffer, MaxInterpreterTrampolineSize);
vixl::CodeBufferCheckScope scope(&emit, MaxInterpreterTrampolineSize, vixl::CodeBufferCheckScope::kDontReserveBufferSpace, vixl::CodeBufferCheckScope::kNoAssert);
@@ -594,7 +645,7 @@ void Arm64Dispatcher::SpillSRA(FEXCore::Core::InternalThreadState *Thread, void
}
void Arm64Dispatcher::InitThreadPointers(FEXCore::Core::InternalThreadState *Thread) {
// Setup dispatcher specific pointers that need to be accessed from JIT code
// Setup dispatcher specific pointers that need to be accessed from JIT code
{
auto &Common = Thread->CurrentFrame->Pointers.Common;
@@ -3,6 +3,10 @@
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#ifdef VIXL_SIMULATOR
#include <aarch64/simulator-aarch64.h>
#endif
namespace FEXCore::Context {
struct Context;
}
@@ -20,6 +24,11 @@ class Arm64Dispatcher final : public Dispatcher, public Arm64Emitter {
size_t GenerateGDBPauseCheck(uint8_t *CodeBuffer, uint64_t GuestRIP) override;
size_t GenerateInterpreterTrampoline(uint8_t *CodeBuffer) override;
#ifdef VIXL_SIMULATOR
void ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) override;
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) override;
#endif
protected:
void SpillSRA(FEXCore::Core::InternalThreadState *Thread, void *ucontext, uint32_t IgnoreMask) override;
@@ -29,6 +38,11 @@ class Arm64Dispatcher final : public Dispatcher, public Arm64Emitter {
uint64_t LDIVHandlerAddress{};
uint64_t LUREMHandlerAddress{};
uint64_t LREMHandlerAddress{};
#ifdef VIXL_SIMULATOR
vixl::aarch64::Decoder Decoder;
vixl::aarch64::Simulator Simulator;
#endif
};
}
@@ -212,12 +212,20 @@ void Dispatcher::RestoreThreadState(FEXCore::Core::InternalThreadState *Thread,
Frame->State.flags[9] = 1;
Frame->State.rip = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP];
Frame->State.cs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS];
Frame->State.ds = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS];
Frame->State.es = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES];
Frame->State.fs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS];
Frame->State.gs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_GS];
Frame->State.ss = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_SS];
Frame->State.cs_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS];
Frame->State.ds_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS];
Frame->State.es_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES];
Frame->State.fs_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS];
Frame->State.gs_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_GS];
Frame->State.ss_idx = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_SS];
Frame->State.cs_cached = Frame->State.gdt[Frame->State.cs_idx >> 3].base;
Frame->State.ds_cached = Frame->State.gdt[Frame->State.ds_idx >> 3].base;
Frame->State.es_cached = Frame->State.gdt[Frame->State.es_idx >> 3].base;
Frame->State.fs_cached = Frame->State.gdt[Frame->State.fs_idx >> 3].base;
Frame->State.gs_cached = Frame->State.gdt[Frame->State.gs_idx >> 3].base;
Frame->State.ss_cached = Frame->State.gdt[Frame->State.ss_idx >> 3].base;
#define COPY_REG(x) \
Frame->State.gregs[X86State::REG_##x] = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_##x];
COPY_REG(RDI);
@@ -565,10 +573,13 @@ bool Dispatcher::HandleGuestSignal(FEXCore::Core::InternalThreadState *Thread, i
auto *xstate = reinterpret_cast<x86::xstate*>(FPStateLocation);
SetXStateInfo(xstate, IsAVXEnabled);
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_GS] = Frame->State.gs;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS] = Frame->State.fs;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES] = Frame->State.es;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS] = Frame->State.ds;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS] = Frame->State.cs_idx;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS] = Frame->State.ds_idx;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES] = Frame->State.es_idx;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS] = Frame->State.fs_idx;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_GS] = Frame->State.gs_idx;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_SS] = Frame->State.ss_idx;
if (ContextBackup->FaultToTopAndGeneratedException) {
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_TRAPNO] = Frame->SynchronousFaultData.TrapNo;
guest_siginfo->si_code = Frame->SynchronousFaultData.si_code;
@@ -581,10 +592,8 @@ bool Dispatcher::HandleGuestSignal(FEXCore::Core::InternalThreadState *Thread, i
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ERR] = ConvertSignalToError(Signal, HostSigInfo);
}
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP] = Frame->State.rip;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS] = Frame->State.cs;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EFL] = 0;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_UESP] = 0;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_SS] = Frame->State.ss;
#define COPY_REG(x) \
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_##x] = Frame->State.gregs[X86State::REG_##x];
@@ -32,7 +32,7 @@ struct DispatcherConfig {
class Dispatcher {
public:
virtual ~Dispatcher() = default;
/**
* @name Dispatch Helper functions
* @{ */
@@ -75,12 +75,12 @@ public:
static std::unique_ptr<Dispatcher> CreateX86(FEXCore::Context::Context *CTX, const DispatcherConfig &Config);
static std::unique_ptr<Dispatcher> CreateArm64(FEXCore::Context::Context *CTX, const DispatcherConfig &Config);
void ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) {
virtual void ExecuteDispatch(FEXCore::Core::CpuStateFrame *Frame) {
DispatchPtr(Frame);
}
void ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) {
virtual void ExecuteJITCallback(FEXCore::Core::CpuStateFrame *Frame, uint64_t RIP) {
CallbackPtr(Frame, RIP);
}
+2 -1
View File
@@ -18,6 +18,7 @@ $end_info$
#include <FEXCore/HLE/SyscallHandler.h>
#include <FEXCore/Utils/Allocator.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <FEXCore/Utils/Telemetry.h>
#include <FEXHeaderUtils/TypeDefines.h>
#include <set>
@@ -1132,6 +1133,7 @@ const uint8_t *Decoder::AdjustAddrForSpecialRegion(uint8_t const* _InstStream, u
}
void Decoder::DecodeInstructionsAtEntry(uint8_t const* _InstStream, uint64_t PC, std::function<void(uint64_t BlockEntry, uint64_t Start, uint64_t Length)> AddContainedCodePage) {
FEXCORE_PROFILE_SCOPED("DecodeInstructions");
Blocks.clear();
BlocksToDecode.clear();
HasBlocks.clear();
@@ -1166,7 +1168,6 @@ void Decoder::DecodeInstructionsAtEntry(uint8_t const* _InstStream, uint64_t PC,
std::set<uint64_t> CodePages = { CurrentCodePage };
AddContainedCodePage(PC, CurrentCodePage, FHU::FEX_PAGE_SIZE);
while (!BlocksToDecode.empty()) {
auto BlockDecodeIt = BlocksToDecode.begin();
+55 -28
View File
@@ -1,7 +1,7 @@
#include "Interface/Core/CPUID.h"
#include <FEXCore/Core/HostFeatures.h>
#ifdef _M_ARM_64
#if defined(_M_ARM_64) || defined(VIXL_SIMULATOR)
#include "aarch64/assembler-aarch64.h"
#include "aarch64/cpu-aarch64.h"
#include "aarch64/disasm-aarch64.h"
@@ -50,8 +50,12 @@ static uint32_t GetDCZID() {
HostFeatures::HostFeatures() {
#ifdef _M_ARM_64
#if defined(_M_ARM_64) || defined(VIXL_SIMULATOR)
#ifdef VIXL_SIMULATOR
auto Features = vixl::CPUFeatures::All();
#else
auto Features = vixl::CPUFeatures::InferFromOS();
#endif
SupportsAES = Features.Has(vixl::CPUFeatures::Feature::kAES);
SupportsCRC = Features.Has(vixl::CPUFeatures::Feature::kCRC32);
SupportsAtomics = Features.Has(vixl::CPUFeatures::Feature::kAtomics);
@@ -62,9 +66,24 @@ HostFeatures::HostFeatures() {
SupportsRCPC = Features.Has(vixl::CPUFeatures::Feature::kRCpc);
SupportsTSOImm9 = Features.Has(vixl::CPUFeatures::Feature::kRCpcImm);
Supports3DNow = true;
SupportsSSE4A = true;
#ifdef VIXL_SIMULATOR
// Hardcode enable SVE with 256-bit wide registers.
SupportsAVX = true;
#else
SupportsAVX = Features.Has(vixl::CPUFeatures::Feature::kSVE2) &&
vixl::aarch64::CPU::ReadSVEVectorLengthInBits() >= 256;
#endif
SupportsSHA = true;
SupportsBMI1 = true;
SupportsBMI2 = true;
if (!SupportsAtomics) {
WARN_ONCE_FMT("Host CPU doesn't support atomics. Expect bad performance");
}
#ifdef _M_ARM_64
// We need to get the CPU's cache line size
// We expect sane targets that have correct cacheline sizes across clusters
uint64_t CTR;
@@ -74,32 +93,6 @@ HostFeatures::HostFeatures() {
DCacheLineSize = 4 << ((CTR >> 16) & 0xF);
ICacheLineSize = 4 << (CTR & 0xF);
if (!SupportsAtomics) {
WARN_ONCE_FMT("Host CPU doesn't support atomics. Expect bad performance");
}
#endif
#ifdef _M_X86_64
Xbyak::util::Cpu Features{};
SupportsAES = Features.has(Xbyak::util::Cpu::tAESNI);
SupportsCRC = Features.has(Xbyak::util::Cpu::tSSE42);
SupportsRAND = Features.has(Xbyak::util::Cpu::tRDRAND) && Features.has(Xbyak::util::Cpu::tRDSEED);
SupportsRCPC = true;
SupportsTSOImm9 = true;
SupportsAVX = true;
// xbyak doesn't know how to check for CLZero
uint32_t eax, ebx, ecx, edx;
// First ensure we support a new enough extended CPUID function range
__cpuid(0x8000'0000, eax, ebx, ecx, edx);
if (eax >= 0x8000'0008U) {
// CLZero defined in 8000_00008_EBX[bit 0]
__cpuid(0x8000'0008, eax, ebx, ecx, edx);
SupportsCLZERO = ebx & 1;
}
SupportsFlushInputsToZero = true;
SupportsFloatExceptions = true;
#else
// Test if this CPU supports float exception trapping by attempting to enable
// On unsupported these bits are architecturally defined as RAZ/WI
constexpr uint32_t ExceptionEnableTraps =
@@ -120,6 +113,39 @@ HostFeatures::HostFeatures() {
SetFPCR(OriginalFPCR);
#endif
#endif
#if defined(_M_X86_64) && !defined(VIXL_SIMULATOR)
Xbyak::util::Cpu Features{};
SupportsAES = Features.has(Xbyak::util::Cpu::tAESNI);
SupportsCRC = Features.has(Xbyak::util::Cpu::tSSE42);
SupportsRAND = Features.has(Xbyak::util::Cpu::tRDRAND) && Features.has(Xbyak::util::Cpu::tRDSEED);
SupportsRCPC = true;
SupportsTSOImm9 = true;
Supports3DNow = Features.has(Xbyak::util::Cpu::t3DN) && Features.has(Xbyak::util::Cpu::tE3DN);
SupportsSSE4A = Features.has(Xbyak::util::Cpu::tSSE4a);
SupportsAVX = true;
SupportsSHA = Features.has(Xbyak::util::Cpu::tSHA);
SupportsBMI1 = Features.has(Xbyak::util::Cpu::tBMI1);
SupportsBMI2 = Features.has(Xbyak::util::Cpu::tBMI2);
// xbyak doesn't know how to check for CLZero
uint32_t eax, ebx, ecx, edx;
// First ensure we support a new enough extended CPUID function range
__cpuid(0x8000'0000, eax, ebx, ecx, edx);
if (eax >= 0x8000'0008U) {
// CLZero defined in 8000_00008_EBX[bit 0]
__cpuid(0x8000'0008, eax, ebx, ecx, edx);
SupportsCLZERO = ebx & 1;
}
SupportsFlushInputsToZero = true;
SupportsFloatExceptions = true;
#endif
#ifdef VIXL_SIMULATOR
// simulator doesn't support dc(ZVA)
SupportsCLZERO = false;
#else
// Check if we can support cacheline clears
uint32_t DCZID = GetDCZID();
if ((DCZID & DCZID_DZP_MASK) == 0) {
@@ -129,5 +155,6 @@ HostFeatures::HostFeatures() {
// This means we can use the instruction
SupportsCLZERO = DCZID_Bytes == CPUIDEmu::CACHELINE_SIZE;
}
#endif
}
}
+35 -17
View File
@@ -894,33 +894,51 @@ DEF_OP(Select) {
}
DEF_OP(VExtractToGPR) {
auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const auto OpSize = IROp->Size;
constexpr auto AVXRegSize = Core::CPUState::XMM_AVX_REG_SIZE;
constexpr auto SSERegSize = Core::CPUState::XMM_SSE_REG_SIZE;
constexpr auto SSEBitSize = SSERegSize * 8;
const auto ElementSize = Op->Header.ElementSize;
const auto ElementSizeBits = ElementSize * 8;
const auto Shift = ElementSizeBits * Op->Index;
const uint32_t SourceSize = GetOpSize(Data->CurrentIR, Op->Vector);
LOGMAN_THROW_AA_FMT(IROp->Size <= 16, "OpSize is too large for VExtractToGPR: {}", IROp->Size);
LOGMAN_THROW_AA_FMT(OpSize <= AVXRegSize,
"OpSize is too large for VExtractToGPR: {}", OpSize);
if (SourceSize == 16) {
__uint128_t SourceMask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
uint64_t Shift = Op->Header.ElementSize * Op->Index * 8;
if (Op->Header.ElementSize == 8)
if (SourceSize >= SSERegSize) {
__uint128_t SourceMask = (1ULL << ElementSizeBits) - 1;
if (ElementSize == 8) {
SourceMask = ~0ULL;
}
__uint128_t Src = *GetSrc<__uint128_t*>(Data->SSAData, Op->Vector);
Src >>= Shift;
Src &= SourceMask;
memcpy(GDP, &Src, Op->Header.ElementSize);
const auto Src = *GetSrc<InterpVector256*>(Data->SSAData, Op->Vector);
const auto GetResult = [&] {
if (Shift >= SSEBitSize) {
const auto NormalizedShift = Shift - SSEBitSize;
return (Src.Upper >> NormalizedShift) & SourceMask;
} else {
return (Src.Lower >> Shift) & SourceMask;
}
};
const auto Result = GetResult();
memcpy(GDP, &Result, ElementSize);
}
else {
uint64_t SourceMask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
uint64_t Shift = Op->Header.ElementSize * Op->Index * 8;
if (Op->Header.ElementSize == 8)
uint64_t SourceMask = (1ULL << ElementSizeBits) - 1;
if (ElementSize == 8) {
SourceMask = ~0ULL;
}
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Vector);
Src >>= Shift;
Src &= SourceMask;
GD = Src;
const uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Vector);
const uint64_t Result = (Src >> Shift) & SourceMask;
GD = Result;
}
}
@@ -13,22 +13,46 @@ $end_info$
namespace FEXCore::CPU {
#define DEF_OP(x) void InterpreterOps::Op_##x(IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node)
DEF_OP(VInsGPR) {
auto Op = IROp->C<IR::IROp_VInsGPR>();
const uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_VInsGPR>();
const auto OpSize = IROp->Size;
auto Src1 = *GetSrc<__uint128_t*>(Data->SSAData, Op->DestVector);
auto Src2 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Src);
const auto ElementSize = Op->Header.ElementSize;
const auto ElementSizeBits = ElementSize * 8;
constexpr auto SSEBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
uint64_t Offset = Op->DestIdx * Op->Header.ElementSize * 8;
__uint128_t Mask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
if (Op->Header.ElementSize == 8) {
const uint64_t Offset = Op->DestIdx * ElementSizeBits;
const auto InUpperLane = Offset >= SSEBitSize;
__uint128_t Mask = (1ULL << ElementSizeBits) - 1;
if (ElementSize == 8) {
Mask = ~0ULL;
}
Src2 = Src2 & Mask;
Mask <<= Offset;
const auto Src1 = *GetSrc<InterpVector256*>(Data->SSAData, Op->DestVector);
const auto Src2 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Src);
const auto Scalar = Src2 & Mask;
const auto ScaledOffset = InUpperLane ? Offset - SSEBitSize
: Offset;
// Now shift into place and set all bits but
// the ones where we're going to insert our value.
Mask <<= ScaledOffset;
Mask = ~Mask;
__uint128_t Dst = Src1 & Mask;
Dst |= Src2 << Offset;
const auto Dst = [&] {
if (InUpperLane) {
return InterpVector256{
.Lower = Src1.Lower,
.Upper = (Src1.Upper & Mask) | (Scalar << ScaledOffset),
};
} else {
return InterpVector256{
.Lower = (Src1.Lower & Mask) | (Scalar << ScaledOffset),
.Upper = Src1.Upper,
};
}
}();
memcpy(GDP, &Dst, OpSize);
}
@@ -89,63 +113,73 @@ DEF_OP(Vector_SToF) {
const uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Vector);
uint8_t Tmp[16]{};
uint8_t Tmp[Core::CPUState::XMM_AVX_REG_SIZE]{};
const uint8_t Elements = OpSize / Op->Header.ElementSize;
const uint8_t ElementSize = Op->Header.ElementSize;
const uint8_t Elements = OpSize / ElementSize;
const auto Func = [](auto a, auto min, auto max) { return a; };
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_2TYPE_OP(4, float, int32_t, Func, 0, 0)
DO_VECTOR_1SRC_2TYPE_OP(8, double, int64_t, Func, 0, 0)
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(Vector_FToZS) {
auto Op = IROp->C<IR::IROp_Vector_FToZS>();
const auto Op = IROp->C<IR::IROp_Vector_FToZS>();
const uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Vector);
uint8_t Tmp[16]{};
uint8_t Tmp[Core::CPUState::XMM_AVX_REG_SIZE]{};
const uint8_t Elements = OpSize / Op->Header.ElementSize;
const uint8_t ElementSize = Op->Header.ElementSize;
const uint8_t Elements = OpSize / ElementSize;
const auto Func = [](auto a, auto min, auto max) { return std::trunc(a); };
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_2TYPE_OP(4, int32_t, float, Func, 0, 0)
DO_VECTOR_1SRC_2TYPE_OP(8, int64_t, double, Func, 0, 0)
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(Vector_FToS) {
auto Op = IROp->C<IR::IROp_Vector_FToS>();
const auto Op = IROp->C<IR::IROp_Vector_FToS>();
const uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Vector);
uint8_t Tmp[16]{};
uint8_t Tmp[Core::CPUState::XMM_AVX_REG_SIZE]{};
const uint8_t Elements = OpSize / Op->Header.ElementSize;
const uint8_t ElementSize = Op->Header.ElementSize;
const uint8_t Elements = OpSize / ElementSize;
const auto Func = [](auto a, auto min, auto max) { return std::nearbyint(a); };
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_2TYPE_OP(4, int32_t, float, Func, 0, 0)
DO_VECTOR_1SRC_2TYPE_OP(8, int64_t, double, Func, 0, 0)
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(Vector_FToF) {
auto Op = IROp->C<IR::IROp_Vector_FToF>();
const auto Op = IROp->C<IR::IROp_Vector_FToF>();
const uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Vector);
uint8_t Tmp[16]{};
uint8_t Tmp[Core::CPUState::XMM_AVX_REG_SIZE]{};
const uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const uint16_t ElementSize = Op->Header.ElementSize;
const uint16_t Conv = (ElementSize << 8) | Op->SrcElementSize;
const auto Func = [](auto a, auto min, auto max) { return a; };
switch (Conv) {
@@ -165,19 +199,22 @@ DEF_OP(Vector_FToF) {
DO_VECTOR_1SRC_2TYPE_OP_NOSIZE(float, double, Func, 0, 0)
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Conversion Type : 0x{:04x}", Conv); break;
default:
LOGMAN_MSG_A_FMT("Unknown Conversion Type : 0x{:04x}", Conv);
break;
}
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(Vector_FToI) {
auto Op = IROp->C<IR::IROp_Vector_FToI>();
const auto Op = IROp->C<IR::IROp_Vector_FToI>();
const uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Vector);
uint8_t Tmp[16]{};
uint8_t Tmp[Core::CPUState::XMM_AVX_REG_SIZE]{};
const uint8_t Elements = OpSize / Op->Header.ElementSize;
const uint8_t ElementSize = Op->Header.ElementSize;
const uint8_t Elements = OpSize / ElementSize;
const auto Func_Nearest = [](auto a) { return std::rint(a); };
const auto Func_Neg = [](auto a) { return std::floor(a); };
const auto Func_Pos = [](auto a) { return std::ceil(a); };
@@ -186,31 +223,31 @@ DEF_OP(Vector_FToI) {
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_OP(4, float, Func_Nearest)
DO_VECTOR_1SRC_OP(8, double, Func_Nearest)
}
break;
case FEXCore::IR::Round_Negative_Infinity.Val:
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_OP(4, float, Func_Neg)
DO_VECTOR_1SRC_OP(8, double, Func_Neg)
}
break;
case FEXCore::IR::Round_Positive_Infinity.Val:
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_OP(4, float, Func_Pos)
DO_VECTOR_1SRC_OP(8, double, Func_Pos)
}
break;
case FEXCore::IR::Round_Towards_Zero.Val:
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_OP(4, float, Func_Trunc)
DO_VECTOR_1SRC_OP(8, double, Func_Trunc)
}
break;
case FEXCore::IR::Round_Host.Val:
switch (Op->Header.ElementSize) {
switch (ElementSize) {
DO_VECTOR_1SRC_OP(4, float, Func_Host)
DO_VECTOR_1SRC_OP(8, double, Func_Host)
}
@@ -159,21 +159,26 @@
break; \
}
struct InterpVector256 {
__uint128_t Lower;
__uint128_t Upper;
};
template<typename Res>
Res GetDest(void* SSAData, FEXCore::IR::OrderedNodeWrapper Op) {
auto DstPtr = &reinterpret_cast<__uint128_t*>(SSAData)[Op.ID().Value];
auto DstPtr = &reinterpret_cast<InterpVector256*>(SSAData)[Op.ID().Value];
return reinterpret_cast<Res>(DstPtr);
}
template<typename Res>
Res GetDest(void* SSAData, FEXCore::IR::NodeID Op) {
auto DstPtr = &reinterpret_cast<__uint128_t*>(SSAData)[Op.Value];
auto DstPtr = &reinterpret_cast<InterpVector256*>(SSAData)[Op.Value];
return reinterpret_cast<Res>(DstPtr);
}
template<typename Res>
Res GetSrc(void* SSAData, FEXCore::IR::OrderedNodeWrapper Src) {
auto DstPtr = &reinterpret_cast<__uint128_t*>(SSAData)[Src.ID().Value];
auto DstPtr = &reinterpret_cast<InterpVector256*>(SSAData)[Src.ID().Value];
return reinterpret_cast<Res>(DstPtr);
}
@@ -1,5 +1,6 @@
#include "Interface/Context/Context.h"
#include "Interface/Core/CPUID.h"
#include "InterpreterDefines.h"
#include "InterpreterOps.h"
#include "F80Ops.h"
@@ -180,13 +181,10 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
// Move ops
REGISTER_OP(EXTRACTELEMENTPAIR, ExtractElementPair);
REGISTER_OP(CREATEELEMENTPAIR, CreateElementPair);
REGISTER_OP(MOV, Mov);
// Vector ops
REGISTER_OP(VECTORZERO, VectorZero);
REGISTER_OP(VECTORIMM, VectorImm);
REGISTER_OP(SPLATVECTOR2, SplatVector);
REGISTER_OP(SPLATVECTOR4, SplatVector);
REGISTER_OP(VMOV, VMov);
REGISTER_OP(VAND, VAnd);
REGISTER_OP(VBIC, VBic);
@@ -245,8 +243,6 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(VUSHRS, VUShrS);
REGISTER_OP(VSSHRS, VSShrS);
REGISTER_OP(VINSELEMENT, VInsElement);
REGISTER_OP(VINSSCALARELEMENT, VInsScalarElement);
REGISTER_OP(VEXTRACTELEMENT, VExtractElement);
REGISTER_OP(VDUPELEMENT, VDupElement);
REGISTER_OP(VEXTR, VExtr);
REGISTER_OP(VSLI, VSLI);
@@ -256,7 +252,6 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(VSHLI, VShlI);
REGISTER_OP(VUSHRNI, VUShrNI);
REGISTER_OP(VUSHRNI2, VUShrNI2);
REGISTER_OP(VBITCAST, VBitcast);
REGISTER_OP(VSXTL, VSXTL);
REGISTER_OP(VSXTL2, VSXTL2);
REGISTER_OP(VUXTL, VUXTL);
@@ -336,23 +331,28 @@ void InterpreterOps::Op_NoOp(FEXCore::IR::IROp_Header *IROp, IROpData *Data, IR:
void InterpreterOps::InterpretIR(FEXCore::Core::CpuStateFrame *Frame, FEXCore::IR::IRListView const *CurrentIR) {
volatile void *StackEntry = alloca(0);
uintptr_t ListSize = CurrentIR->GetSSACount();
const uintptr_t ListSize = CurrentIR->GetSSACount();
static_assert(sizeof(FEXCore::IR::IROp_Header) == 4);
static_assert(sizeof(FEXCore::IR::OrderedNode) == 16);
auto BlockEnd = CurrentIR->GetBlocks().end();
InterpreterOps::IROpData OpData{};
OpData.State = Frame->Thread;
OpData.SSAData = alloca(ListSize * 16);
OpData.CurrentEntry = Frame->State.rip;
OpData.CurrentIR = CurrentIR;
OpData.StackEntry = StackEntry;
OpData.BlockIterator = CurrentIR->GetBlocks().begin();
constexpr size_t ListEntrySizeInBytes = sizeof(InterpVector256);
const size_t SSADataSize = ListSize * ListEntrySizeInBytes;
// Clear them all to zero. Required for Zero-extend semantics
memset(OpData.SSAData, 0, ListSize * 16);
InterpreterOps::IROpData OpData{
.State = Frame->Thread,
.CurrentEntry = Frame->State.rip,
.CurrentIR = CurrentIR,
.StackEntry = StackEntry,
.SSAData = alloca(SSADataSize),
.BlockResults = {},
.BlockIterator = CurrentIR->GetBlocks().begin(),
};
// Clear all SSAData entries to zero. Required for Zero-extend semantics
memset(OpData.SSAData, 0, SSADataSize);
while (1) {
using namespace FEXCore::IR;
@@ -207,7 +207,6 @@ namespace FEXCore::CPU {
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(SplatVector);
DEF_OP(VMov);
DEF_OP(VAnd);
DEF_OP(VBic);
@@ -264,8 +263,6 @@ namespace FEXCore::CPU {
DEF_OP(VUShrS);
DEF_OP(VSShrS);
DEF_OP(VInsElement);
DEF_OP(VInsScalarElement);
DEF_OP(VExtractElement);
DEF_OP(VDupElement);
DEF_OP(VExtr);
DEF_OP(VSLI);
@@ -275,7 +272,6 @@ namespace FEXCore::CPU {
DEF_OP(VShlI);
DEF_OP(VUShrNI);
DEF_OP(VUShrNI2);
DEF_OP(VBitcast);
DEF_OP(VSXTL);
DEF_OP(VSXTL2);
DEF_OP(VUXTL);
@@ -25,40 +25,45 @@ static inline void CacheLineFlush(char *Addr) {
#define DEF_OP(x) void InterpreterOps::Op_##x(IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node)
DEF_OP(LoadContext) {
auto Op = IROp->C<IR::IROp_LoadContext>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_LoadContext>();
const auto OpSize = IROp->Size;
const auto ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
const auto Src = ContextPtr + Op->Offset;
uintptr_t ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
ContextPtr += Op->Offset;
#define LOAD_CTX(x, y) \
case x: { \
y const *MemData = reinterpret_cast<y const*>(ContextPtr); \
y const *MemData = reinterpret_cast<y const*>(Src); \
GD = *MemData; \
break; \
}
switch (OpSize) {
LOAD_CTX(1, uint8_t)
LOAD_CTX(2, uint16_t)
LOAD_CTX(4, uint32_t)
LOAD_CTX(8, uint64_t)
case 16: {
void const *MemData = reinterpret_cast<void const*>(ContextPtr);
case 16:
case 32: {
void const *MemData = reinterpret_cast<void const*>(Src);
memcpy(GDP, MemData, OpSize);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
break;
}
#undef LOAD_CTX
}
DEF_OP(StoreContext) {
auto Op = IROp->C<IR::IROp_StoreContext>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_StoreContext>();
const auto OpSize = IROp->Size;
uintptr_t ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
ContextPtr += Op->Offset;
const auto ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
const auto Dst = ContextPtr + Op->Offset;
void *MemData = reinterpret_cast<void*>(ContextPtr);
void *MemData = reinterpret_cast<void*>(Dst);
void *Src = GetSrc<void*>(Data->SSAData, Op->Value);
memcpy(MemData, Src, OpSize);
}
@@ -72,46 +77,51 @@ DEF_OP(StoreRegister) {
}
DEF_OP(LoadContextIndexed) {
auto Op = IROp->C<IR::IROp_LoadContextIndexed>();
uint64_t Index = *GetSrc<uint64_t*>(Data->SSAData, Op->Index);
const auto Op = IROp->C<IR::IROp_LoadContextIndexed>();
const auto OpSize = IROp->Size;
uintptr_t ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
const auto Index = *GetSrc<uint64_t*>(Data->SSAData, Op->Index);
ContextPtr += Op->BaseOffset;
ContextPtr += Index * Op->Stride;
const auto ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
const auto Src = ContextPtr + Op->BaseOffset + (Index * Op->Stride);
#define LOAD_CTX(x, y) \
case x: { \
y const *MemData = reinterpret_cast<y const*>(ContextPtr); \
y const *MemData = reinterpret_cast<y const*>(Src); \
GD = *MemData; \
break; \
}
switch (IROp->Size) {
switch (OpSize) {
LOAD_CTX(1, uint8_t)
LOAD_CTX(2, uint16_t)
LOAD_CTX(4, uint32_t)
LOAD_CTX(8, uint64_t)
case 16: {
void const *MemData = reinterpret_cast<void const*>(ContextPtr);
memcpy(GDP, MemData, IROp->Size);
case 16:
case 32: {
void const *MemData = reinterpret_cast<void const*>(Src);
memcpy(GDP, MemData, OpSize);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", IROp->Size);
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", OpSize);
break;
}
#undef LOAD_CTX
}
DEF_OP(StoreContextIndexed) {
auto Op = IROp->C<IR::IROp_StoreContextIndexed>();
uint64_t Index = *GetSrc<uint64_t*>(Data->SSAData, Op->Index);
const auto Op = IROp->C<IR::IROp_StoreContextIndexed>();
const auto OpSize = IROp->Size;
uintptr_t ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
ContextPtr += Op->BaseOffset;
ContextPtr += Index * Op->Stride;
const auto Index = *GetSrc<uint64_t*>(Data->SSAData, Op->Index);
void *MemData = reinterpret_cast<void*>(ContextPtr);
const auto ContextPtr = reinterpret_cast<uintptr_t>(Data->State->CurrentFrame);
const auto Dst = ContextPtr + Op->BaseOffset + (Index * Op->Stride);
void *MemData = reinterpret_cast<void*>(Dst);
void *Src = GetSrc<void*>(Data->SSAData, Op->Value);
memcpy(MemData, Src, IROp->Size);
memcpy(MemData, Src, OpSize);
}
DEF_OP(SpillRegister) {
@@ -144,8 +154,8 @@ DEF_OP(StoreFlag) {
}
DEF_OP(LoadMem) {
auto Op = IROp->C<IR::IROp_LoadMem>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_LoadMem>();
const auto OpSize = IROp->Size;
uint8_t const *MemData = *GetSrc<uint8_t const**>(Data->SSAData, Op->Addr);
@@ -158,7 +168,8 @@ DEF_OP(LoadMem) {
case IR::MEM_OFFSET_SXTW.Val: MemData += (int32_t)Offset; break;
}
}
memset(GDP, 0, 16);
memset(GDP, 0, Core::CPUState::XMM_AVX_REG_SIZE);
switch (OpSize) {
case 1: {
auto D = reinterpret_cast<const std::atomic<uint8_t>*>(MemData);
@@ -180,16 +191,15 @@ DEF_OP(LoadMem) {
GD = D->load();
break;
}
default:
memcpy(GDP, MemData, IROp->Size);
memcpy(GDP, MemData, OpSize);
break;
}
}
DEF_OP(StoreMem) {
auto Op = IROp->C<IR::IROp_StoreMem>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_StoreMem>();
const auto OpSize = IROp->Size;
uint8_t *MemData = *GetSrc<uint8_t **>(Data->SSAData, Op->Addr);
@@ -221,7 +231,7 @@ DEF_OP(StoreMem) {
}
default:
memcpy(MemData, GetSrc<void*>(Data->SSAData, Op->Value), IROp->Size);
memcpy(MemData, GetSrc<void*>(Data->SSAData, Op->Value), OpSize);
break;
}
}
@@ -19,13 +19,6 @@ $end_info$
#include <sys/random.h>
namespace FEXCore::CPU {
[[noreturn]]
static void StopThread(FEXCore::Core::InternalThreadState *Thread) {
Thread->CTX->StopThread(Thread);
LOGMAN_MSG_A_FMT("unreachable");
FEX_UNREACHABLE;
}
#define DEF_OP(x) void InterpreterOps::Op_##x(IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node)
DEF_OP(Fence) {
@@ -30,13 +30,6 @@ DEF_OP(CreateElementPair) {
memcpy(Dst + IROp->ElementSize, Src_Upper, IROp->ElementSize);
}
DEF_OP(Mov) {
auto Op = IROp->C<IR::IROp_Mov>();
const uint8_t OpSize = IROp->Size;
memcpy(GDP, GetSrc<void*>(Data->SSAData, Op->Value), OpSize);
}
#undef DEF_OP
} // namespace FEXCore::CPU
File diff suppressed because it is too large. Load diff
+72 -18
View File
@@ -1168,25 +1168,79 @@ DEF_OP(Select) {
}
DEF_OP(VExtractToGPR) {
auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const auto OpSize = IROp->Size;
switch (OpSize) {
case 1:
umov(GetReg<RA_32>(Node), GetSrc(Op->Vector.ID()).V16B(), Op->Index);
break;
case 2:
umov(GetReg<RA_32>(Node), GetSrc(Op->Vector.ID()).V8H(), Op->Index);
break;
case 4:
umov(GetReg<RA_32>(Node), GetSrc(Op->Vector.ID()).V4S(), Op->Index);
break;
case 8:
umov(GetReg<RA_64>(Node), GetSrc(Op->Vector.ID()).V2D(), Op->Index);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled ExtractElementSize: {}", OpSize);
break;
constexpr auto AVXRegBitSize = Core::CPUState::XMM_AVX_REG_SIZE * 8;
constexpr auto SSERegBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
const auto ElementSizeBits = Op->Header.ElementSize * 8;
const auto Offset = ElementSizeBits * Op->Index;
const auto Is256Bit = Offset >= SSERegBitSize;
const auto Vector = GetSrc(Op->Vector.ID());
const auto PerformMove = [&](const aarch64::VRegister& reg, int index) {
switch (OpSize) {
case 1:
umov(GetReg<RA_32>(Node), reg.V16B(), index);
break;
case 2:
umov(GetReg<RA_32>(Node), reg.V8H(), index);
break;
case 4:
umov(GetReg<RA_32>(Node), reg.V4S(), index);
break;
case 8:
umov(GetReg<RA_64>(Node), reg.V2D(), index);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled ExtractElementSize: {}", OpSize);
break;
}
};
if (Offset < SSERegBitSize) {
// Desired data lies within the lower 128-bit lane, so we
// can treat the operation as a 128-bit operation, even
// when acting on larger register sizes.
PerformMove(Vector, Op->Index);
} else {
LOGMAN_THROW_AA_FMT(HostSupportsSVE,
"Host doesn't support SVE. Cannot perform 256-bit operation.");
LOGMAN_THROW_AA_FMT(Is256Bit,
"Can't perform 256-bit extraction with op side: {}", OpSize);
LOGMAN_THROW_AA_FMT(Offset < AVXRegBitSize,
"Trying to extract element outside bounds of register. Offset={}, Index={}",
Offset, Op->Index);
// We need to use the upper 128-bit lane, so lets move it down.
// Inverting our dedicated predicate for 128-bit operations selects
// all of the top lanes. We can then compact those into a temporary.
const auto CompactPred = p0;
not_(CompactPred.VnB(), PRED_TMP_32B.Zeroing(), PRED_TMP_16B.VnB());
compact(VTMP1.Z().VnD(), CompactPred, Vector.Z().VnD());
// Sanitize the zero-based index to work on the now-moved
// upper half of the vector.
const auto SanitizedIndex = [OpSize, Op] {
switch (OpSize) {
case 1:
return Op->Index - 16;
case 2:
return Op->Index - 8;
case 4:
return Op->Index - 4;
case 8:
return Op->Index - 2;
default:
LOGMAN_MSG_A_FMT("Unhandled OpSize: {}", OpSize);
return 0;
}
}();
// Move the value from the now-low-lane data.
PerformMove(VTMP1, SanitizedIndex);
}
}
@@ -40,7 +40,7 @@ DEF_OP(CallbackReturn) {
ResetStack();
// We can now lower the ref counter again
ldr(w2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, SignalHandlerRefCounter)));
sub(w2, w2, 1);
str(w2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, SignalHandlerRefCounter)));
@@ -197,7 +197,11 @@ DEF_OP(Syscall) {
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.SyscallHandlerFunc)));
mov(x1, STATE);
mov(x2, sp);
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, void*, void*, void*>(x3);
#else
blr(x3);
#endif
add(sp, sp, SPOffset);
@@ -239,7 +243,6 @@ DEF_OP(InlineSyscall) {
bool Intersects{};
// We always need to spill x8 since we can't know if it is live at this SSA location
uint32_t SpillMask = 1U << 8;
std::vector<vixl::aarch64::Register> IntersectRegs(FEXCore::HLE::SyscallArguments::MAX_ARGS);
for (uint32_t i = 0; i < FEXCore::HLE::SyscallArguments::MAX_ARGS-1; ++i) {
if (Op->Header.Args[i].IsInvalid()) break;
@@ -381,7 +384,11 @@ DEF_OP(Thunk) {
auto thunkFn = ThreadState->CTX->ThunkHandler->LookupThunk(Op->ThunkNameHash);
LoadConstant(x2, (uintptr_t)thunkFn);
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<void, void*, void*>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -448,7 +455,11 @@ DEF_OP(ThreadRemoveCodeEntry) {
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.ThreadRemoveCodeEntryFromJIT)));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<void, void*, void*>(x2);
#else
blr(x2);
#endif
FillStaticRegs();
// Fix the stack and any values that were stepped on
@@ -459,6 +470,7 @@ DEF_OP(CPUID) {
auto Op = IROp->C<IR::IROp_CPUID>();
PushDynamicRegsAndLR();
SpillStaticRegs();
// x0 = CPUID Handler
// x1 = CPUID Function
@@ -467,10 +479,13 @@ DEF_OP(CPUID) {
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.CPUIDFunction)));
mov(x1, GetReg<RA_64>(Op->Function.ID()));
mov(x2, GetReg<RA_64>(Op->Leaf.ID()));
SpillStaticRegs();
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, void*, uint64_t, uint64_t>(x3);
#else
blr(x3);
FillStaticRegs();
#endif
FillStaticRegs();
PopDynamicRegsAndLR();
// Results are in x0, x1
@@ -12,26 +12,115 @@ using namespace vixl;
using namespace vixl::aarch64;
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(VInsGPR) {
auto Op = IROp->C<IR::IROp_VInsGPR>();
mov(GetDst(Node), GetSrc(Op->DestVector.ID()));
switch (Op->Header.ElementSize) {
case 1: {
ins(GetDst(Node).V16B(), Op->DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
const auto Op = IROp->C<IR::IROp_VInsGPR>();
const auto OpSize = IROp->Size;
const auto DestIdx = Op->DestIdx;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto DestVector = GetSrc(Op->DestVector.ID());
if (HostSupportsSVE && Is256Bit) {
const auto ElementSizeBits = ElementSize * 8;
const auto Offset = ElementSizeBits * DestIdx;
const auto SSEBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
const auto InUpperLane = Offset >= SSEBitSize;
// This is going to be a little gross. Pls forgive me.
// Since SVE has the whole vector length agnostic programming
// thing going on, we can't exactly freely insert entries into
// arbitrary locations in the vector.
//
// SVE *does* have INSR, however this only shifts the entire
// vector to the left by an element size and inserts a value
// at the beginning of the vector. Not *quite* what we need.
// (though INSR *is* very useful for other things).
//
// The idea is (in the case of the upper lane), move the upper
// lane down, insert into it and recombine with the lower lane.
//
// In the case of the lower lane, insert and then recombine with
// the upper lane.
if (InUpperLane) {
// Move the upper lane down for the insertion.
const auto CompactPred = p0;
not_(CompactPred.VnB(), PRED_TMP_32B.Zeroing(), PRED_TMP_16B.VnB());
compact(VTMP1.Z().VnD(), CompactPred, DestVector.Z().VnD());
}
case 2: {
ins(GetDst(Node).V8H(), Op->DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
// Put data in place for destructive SPLICE below.
mov(Dst.Z().VnD(), DestVector.Z().VnD());
// Inserts the GPR value into the given V register.
// Also automatically adjusts the index in the case of using the
// moved upper lane.
const auto Insert = [&](const aarch64::VRegister& reg, int index) {
switch (ElementSize) {
case 1:
if (InUpperLane) {
index -= 16;
}
ins(reg.V16B(), index, GetReg<RA_32>(Op->Src.ID()));
break;
case 2:
if (InUpperLane) {
index -= 8;
}
ins(reg.V8H(), index, GetReg<RA_32>(Op->Src.ID()));
break;
case 4:
if (InUpperLane) {
index -= 4;
}
ins(reg.V4S(), index, GetReg<RA_32>(Op->Src.ID()));
break;
case 8:
if (InUpperLane) {
index -= 2;
}
ins(reg.V2D(), index, GetReg<RA_64>(Op->Src.ID()));
break;
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
};
if (InUpperLane) {
Insert(VTMP1, DestIdx);
splice(Dst.Z().VnD(), PRED_TMP_16B, Dst.Z().VnD(), VTMP1.Z().VnD());
} else {
Insert(Dst, DestIdx);
splice(Dst.Z().VnD(), PRED_TMP_16B, Dst.Z().VnD(), DestVector.Z().VnD());
}
case 4: {
ins(GetDst(Node).V4S(), Op->DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
} else {
mov(Dst, DestVector);
switch (ElementSize) {
case 1: {
ins(Dst.V16B(), DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
}
case 2: {
ins(Dst.V8H(), DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
}
case 4: {
ins(Dst.V4S(), DestIdx, GetReg<RA_32>(Op->Src.ID()));
break;
}
case 8: {
ins(Dst.V2D(), DestIdx, GetReg<RA_64>(Op->Src.ID()));
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
case 8: {
ins(GetDst(Node).V2D(), Op->DestIdx, GetReg<RA_64>(Op->Src.ID()));
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
}
}
@@ -57,8 +146,11 @@ DEF_OP(VCastFromGPR) {
}
DEF_OP(Float_FromGPR_S) {
auto Op = IROp->C<IR::IROp_Float_FromGPR_S>();
const uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const auto Op = IROp->C<IR::IROp_Float_FromGPR_S>();
const uint16_t ElementSize = Op->Header.ElementSize;
const uint16_t Conv = (ElementSize << 8) | Op->SrcElementSize;
switch (Conv) {
case 0x0404: { // Float <- int32_t
scvtf(GetDst(Node).S(), GetReg<RA_32>(Op->Src.ID()));
@@ -76,6 +168,10 @@ DEF_OP(Float_FromGPR_S) {
scvtf(GetDst(Node).D(), GetReg<RA_64>(Op->Src.ID()));
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled conversion mask: Mask=0x{:04x}, ElementSize={}, SrcElementSize={}",
Conv, ElementSize, Op->SrcElementSize);
break;
}
}
@@ -96,116 +192,379 @@ DEF_OP(Float_FToF) {
}
DEF_OP(Vector_SToF) {
auto Op = IROp->C<IR::IROp_Vector_SToF>();
switch (Op->Header.ElementSize) {
case 4:
scvtf(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
break;
case 8:
scvtf(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_SToF element size: {}", Op->Header.ElementSize);
const auto Op = IROp->C<IR::IROp_Vector_SToF>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
if (HostSupportsSVE && Is256Bit) {
const auto Mask = PRED_TMP_32B.Merging();
switch (ElementSize) {
case 2:
scvtf(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
scvtf(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
scvtf(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_SToF element size: {}", ElementSize);
break;
}
} else {
switch (ElementSize) {
case 2:
scvtf(Dst.V8H(), Vector.V8H());
break;
case 4:
scvtf(Dst.V4S(), Vector.V4S());
break;
case 8:
scvtf(Dst.V2D(), Vector.V2D());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_SToF element size: {}", ElementSize);
break;
}
}
}
DEF_OP(Vector_FToZS) {
auto Op = IROp->C<IR::IROp_Vector_FToZS>();
switch (Op->Header.ElementSize) {
case 4:
fcvtzs(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
break;
case 8:
fcvtzs(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToZS element size: {}", Op->Header.ElementSize);
const auto Op = IROp->C<IR::IROp_Vector_FToZS>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
if (HostSupportsSVE && Is256Bit) {
const auto Mask = PRED_TMP_32B.Merging();
switch (ElementSize) {
case 2:
fcvtzs(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
fcvtzs(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
fcvtzs(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToZS element size: {}", ElementSize);
break;
}
} else {
switch (ElementSize) {
case 2:
fcvtzs(Dst.V8H(), Vector.V8H());
break;
case 4:
fcvtzs(Dst.V4S(), Vector.V4S());
break;
case 8:
fcvtzs(Dst.V2D(), Vector.V2D());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToZS element size: {}", ElementSize);
break;
}
}
}
DEF_OP(Vector_FToS) {
auto Op = IROp->C<IR::IROp_Vector_FToS>();
switch (Op->Header.ElementSize) {
case 4:
frinti(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
fcvtzs(GetDst(Node).V4S(), GetDst(Node).V4S());
break;
case 8:
frinti(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
fcvtzs(GetDst(Node).V2D(), GetDst(Node).V2D());
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToS element size: {}", Op->Header.ElementSize);
const auto Op = IROp->C<IR::IROp_Vector_FToS>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
if (HostSupportsSVE && Is256Bit) {
const auto Mask = PRED_TMP_32B.Merging();
switch (ElementSize) {
case 2:
frinti(Dst.Z().VnH(), Mask, Vector.Z().VnH());
fcvtzs(Dst.Z().VnH(), Mask, Dst.Z().VnH());
break;
case 4:
frinti(Dst.Z().VnS(), Mask, Vector.Z().VnS());
fcvtzs(Dst.Z().VnS(), Mask, Dst.Z().VnS());
break;
case 8:
frinti(Dst.Z().VnD(), Mask, Vector.Z().VnD());
fcvtzs(Dst.Z().VnD(), Mask, Dst.Z().VnD());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToS element size: {}", ElementSize);
break;
}
} else {
switch (ElementSize) {
case 2:
frinti(Dst.V8H(), Vector.V8H());
fcvtzs(Dst.V8H(), Dst.V8H());
break;
case 4:
frinti(Dst.V4S(), Vector.V4S());
fcvtzs(Dst.V4S(), Dst.V4S());
break;
case 8:
frinti(Dst.V2D(), Vector.V2D());
fcvtzs(Dst.V2D(), Dst.V2D());
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToS element size: {}", ElementSize);
break;
}
}
}
DEF_OP(Vector_FToF) {
auto Op = IROp->C<IR::IROp_Vector_FToF>();
uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const auto Op = IROp->C<IR::IROp_Vector_FToF>();
const auto OpSize = IROp->Size;
switch (Conv) {
case 0x0804: { // Double <- Float
fcvtl(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2S());
break;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Conv = (ElementSize << 8) | Op->SrcElementSize;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
if (HostSupportsSVE && Is256Bit) {
// Curiously, FCVTLT and FCVTNT have no bottom variants,
// and also interesting is that FCVTLT will iterate the
// source vector by accessing each odd element and storing
// them consecutively in the destination.
//
// FCVTNT is somewhat like the opposite. It will read each
// consecutive element, but store each result into every odd
// element in the destination vector.
//
// We need to undo the behavior of FCVTNT with UZP2. In the case
// of FCVTLT, we instead need to set the vector up with ZIP1, so
// that the elements will be processed correctly.
const auto Mask = PRED_TMP_32B.Merging();
switch (Conv) {
case 0x0402: { // Float <- Half
zip1(Dst.Z().VnH(), Vector.Z().VnH(), Vector.Z().VnH());
fcvtlt(Dst.Z().VnS(), Mask, Dst.Z().VnH());
break;
}
case 0x0804: { // Double <- Float
zip1(Dst.Z().VnS(), Vector.Z().VnS(), Vector.Z().VnS());
fcvtlt(Dst.Z().VnD(), Mask, Dst.Z().VnS());
break;
}
case 0x0204: { // Half <- Float
fcvtnt(Dst.Z().VnH(), Mask, Vector.Z().VnS());
uzp2(Dst.Z().VnH(), Dst.Z().VnH(), Dst.Z().VnH());
break;
}
case 0x0408: { // Float <- Double
fcvtnt(Dst.Z().VnS(), Mask, Vector.Z().VnD());
uzp2(Dst.Z().VnS(), Dst.Z().VnS(), Dst.Z().VnS());
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv);
break;
}
case 0x0408: { // Float <- Double
fcvtn(GetDst(Node).V2S(), GetSrc(Op->Vector.ID()).V2D());
break;
} else {
switch (Conv) {
case 0x0402: { // Float <- Half
fcvtl(Dst.V4S(), Vector.V4H());
break;
}
case 0x0804: { // Double <- Float
fcvtl(Dst.V2D(), Vector.V2S());
break;
}
case 0x0204: { // Half <- Float
fcvtn(Dst.V4H(), Vector.V4S());
break;
}
case 0x0408: { // Float <- Double
fcvtn(Dst.V2S(), Vector.V2D());
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToF Type : 0x{:04x}", Conv); break;
}
}
DEF_OP(Vector_FToI) {
auto Op = IROp->C<IR::IROp_Vector_FToI>();
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
switch (Op->Header.ElementSize) {
case 4:
frintn(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
const auto Op = IROp->C<IR::IROp_Vector_FToI>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
if (HostSupportsSVE && Is256Bit) {
const auto Mask = PRED_TMP_32B.Merging();
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
switch (ElementSize) {
case 2:
frintn(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
frintn(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
frintn(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
}
break;
case 8:
frintn(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
case FEXCore::IR::Round_Negative_Infinity.Val:
switch (ElementSize) {
case 2:
frintm(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
frintm(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
frintm(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
}
break;
}
break;
case FEXCore::IR::Round_Negative_Infinity.Val:
switch (Op->Header.ElementSize) {
case 4:
frintm(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
case FEXCore::IR::Round_Positive_Infinity.Val:
switch (ElementSize) {
case 2:
frintp(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
frintp(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
frintp(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
}
break;
case 8:
frintm(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
case FEXCore::IR::Round_Towards_Zero.Val:
switch (ElementSize) {
case 2:
frintz(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
frintz(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
frintz(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
}
break;
}
break;
case FEXCore::IR::Round_Positive_Infinity.Val:
switch (Op->Header.ElementSize) {
case 4:
frintp(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
case FEXCore::IR::Round_Host.Val:
switch (ElementSize) {
case 2:
frinti(Dst.Z().VnH(), Mask, Vector.Z().VnH());
break;
case 4:
frinti(Dst.Z().VnS(), Mask, Vector.Z().VnS());
break;
case 8:
frinti(Dst.Z().VnD(), Mask, Vector.Z().VnD());
break;
}
break;
case 8:
frintp(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
}
} else {
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
switch (ElementSize) {
case 2:
frintn(Dst.V8H(), Vector.V8H());
break;
case 4:
frintn(Dst.V4S(), Vector.V4S());
break;
case 8:
frintn(Dst.V2D(), Vector.V2D());
break;
}
break;
}
break;
case FEXCore::IR::Round_Towards_Zero.Val:
switch (Op->Header.ElementSize) {
case 4:
frintz(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
case FEXCore::IR::Round_Negative_Infinity.Val:
switch (ElementSize) {
case 2:
frintm(Dst.V8H(), Vector.V8H());
break;
case 4:
frintm(Dst.V4S(), Vector.V4S());
break;
case 8:
frintm(Dst.V2D(), Vector.V2D());
break;
}
break;
case 8:
frintz(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
case FEXCore::IR::Round_Positive_Infinity.Val:
switch (ElementSize) {
case 2:
frintp(Dst.V8H(), Vector.V8H());
break;
case 4:
frintp(Dst.V4S(), Vector.V4S());
break;
case 8:
frintp(Dst.V2D(), Vector.V2D());
break;
}
break;
}
break;
case FEXCore::IR::Round_Host.Val:
switch (Op->Header.ElementSize) {
case 4:
frinti(GetDst(Node).V4S(), GetSrc(Op->Vector.ID()).V4S());
case FEXCore::IR::Round_Towards_Zero.Val:
switch (ElementSize) {
case 2:
frintz(Dst.V8H(), Vector.V8H());
break;
case 4:
frintz(Dst.V4S(), Vector.V4S());
break;
case 8:
frintz(Dst.V2D(), Vector.V2D());
break;
}
break;
case 8:
frinti(GetDst(Node).V2D(), GetSrc(Op->Vector.ID()).V2D());
case FEXCore::IR::Round_Host.Val:
switch (ElementSize) {
case 2:
frinti(Dst.V8H(), Vector.V8H());
break;
case 4:
frinti(Dst.V4S(), Vector.V4S());
break;
case 8:
frinti(Dst.V2D(), Vector.V2D());
break;
}
break;
}
break;
}
}
}
+78 -11
View File
@@ -28,6 +28,7 @@ $end_info$
#include <FEXCore/Utils/Allocator.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include <FEXCore/Utils/EnumUtils.h>
#include <FEXCore/Utils/Profiler.h>
#include "Interface/Core/Interpreter/InterpreterOps.h"
@@ -94,7 +95,12 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
uxth(w0, GetReg<RA_32>(IROp->Args[0].ID()));
ldr(x1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<void, uint16_t>(x1);
#else
blr(x1);
#endif
PopDynamicRegsAndLR();
@@ -109,7 +115,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
fmov(v0.S(), GetSrc(IROp->Args[0].ID()).S()) ;
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, float>(x0);
#else
blr(x0);
#endif
PopDynamicRegsAndLR();
@@ -128,7 +138,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
mov(v0.D(), GetSrc(IROp->Args[0].ID()).D());
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, double>(x0);
#else
blr(x0);
#endif
PopDynamicRegsAndLR();
@@ -153,7 +167,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
mov(w0, GetReg<RA_32>(IROp->Args[0].ID()));
}
ldr(x1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, uint32_t>(x1);
#else
blr(x1);
#endif
PopDynamicRegsAndLR();
@@ -174,7 +192,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<float, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -193,7 +215,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<double, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -210,7 +236,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
mov(v0.D(), GetSrc(IROp->Args[0].ID()).D());
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<double, double>(x0);
#else
blr(x0);
#endif
PopDynamicRegsAndLR();
@@ -229,7 +259,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
mov(v0.D(), GetSrc(IROp->Args[0].ID()).D());
mov(v1.D(), GetSrc(IROp->Args[1].ID()).D());
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<double, double, double>(x0);
#else
blr(x0);
#endif
PopDynamicRegsAndLR();
@@ -249,7 +283,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint32_t, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -267,7 +305,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint32_t, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -285,7 +327,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -306,8 +352,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w3, GetSrc(IROp->Args[1].ID()).V8H(), 4);
ldr(x4, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<uint64_t, uint64_t, uint64_t, uint64_t, uint64_t>(x4);
#else
blr(x4);
#endif
PopDynamicRegsAndLR();
FillStaticRegs();
@@ -324,7 +373,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, uint64_t, uint64_t>(x2);
#else
blr(x2);
#endif
PopDynamicRegsAndLR();
@@ -347,7 +400,11 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(w3, GetSrc(IROp->Args[1].ID()).V8H(), 4);
ldr(x4, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.FallbackHandlerPointers[Info.HandlerIndex])));
#ifdef VIXL_SIMULATOR
GenerateIndirectRuntimeCall<__uint128_t, uint64_t, uint64_t, uint64_t, uint64_t>(x4);
#else
blr(x4);
#endif
PopDynamicRegsAndLR();
@@ -426,6 +483,7 @@ void Arm64JITCore::Op_NoOp(IR::IROp_Header *IROp, IR::NodeID Node) {
Arm64JITCore::Arm64JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::InternalThreadState *Thread)
: CPUBackend(Thread, INITIAL_CODE_SIZE, MAX_CODE_SIZE)
, Arm64Emitter(ctx, 0)
, HostSupportsSVE{ctx->HostFeatures.SupportsAVX}
, CTX {ctx} {
RAPass = Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA");
@@ -472,7 +530,7 @@ Arm64JITCore::Arm64JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::Intern
// Common
auto &Common = ThreadState->CurrentFrame->Pointers.Common;
Common.PrintValue = reinterpret_cast<uint64_t>(PrintValue);
Common.PrintVectorValue = reinterpret_cast<uint64_t>(PrintVectorValue);
Common.ThreadRemoveCodeEntryFromJIT = reinterpret_cast<uintptr_t>(&Context::Context::ThreadRemoveCodeEntryFromJit);
@@ -493,7 +551,7 @@ Arm64JITCore::Arm64JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::Intern
// Platform Specific
auto &AArch64 = ThreadState->CurrentFrame->Pointers.AArch64;
AArch64.LUDIV = reinterpret_cast<uint64_t>(LUDIV);
AArch64.LDIV = reinterpret_cast<uint64_t>(LDIV);
AArch64.LUREM = reinterpret_cast<uint64_t>(LUREM);
@@ -510,6 +568,7 @@ void Arm64JITCore::InitializeSignalHandlers(FEXCore::Context::Context *CTX) {
return Thread->CTX->Dispatcher->HandleSIGILL(Thread, Signal, info, ucontext);
}, true);
#ifdef _M_ARM_64
CTX->SignalDelegation->RegisterHostSignalHandler(SIGBUS, [](FEXCore::Core::InternalThreadState *Thread, int Signal, void *info, void *ucontext) -> bool {
if (!Thread->CPUBackend->IsAddressInCodeBuffer(ArchHelpers::Context::GetPc(ucontext))) {
// Wasn't a sigbus in JIT code
@@ -518,6 +577,7 @@ void Arm64JITCore::InitializeSignalHandlers(FEXCore::Context::Context *CTX) {
return FEXCore::ArchHelpers::Arm64::HandleSIGBUS(Thread->CTX->Config.ParanoidTSO(), Signal, info, ucontext);
}, true);
#endif
}
void Arm64JITCore::EmitDetectionString() {
@@ -529,7 +589,7 @@ void Arm64JITCore::EmitDetectionString() {
void Arm64JITCore::ClearCache() {
// Get the backing code buffer
auto CodeBuffer = GetEmptyCodeBuffer();
*GetBuffer() = vixl::CodeBuffer(CodeBuffer->Ptr, CodeBuffer->Size);
EmitDetectionString();
@@ -672,6 +732,8 @@ void *Arm64JITCore::CompileCode(uint64_t Entry,
FEXCore::Core::DebugData *DebugData,
FEXCore::IR::RegisterAllocationData *RAData,
bool GDBEnabled) {
FEXCORE_PROFILE_SCOPED("Arm64::CompileCode");
using namespace aarch64;
JumpTargets.clear();
uint32_t SSACount = IR->GetSSACount();
@@ -724,10 +786,12 @@ void *Arm64JITCore::CompileCode(uint64_t Entry,
SpillSlots = RAData->SpillSlots();
if (SpillSlots) {
if (IsImmAddSub(SpillSlots * 16)) {
sub(sp, sp, SpillSlots * 16);
const auto TotalSpillSlotsSize = SpillSlots * MaxSpillSlotSize;
if (IsImmAddSub(TotalSpillSlotsSize)) {
sub(sp, sp, TotalSpillSlotsSize);
} else {
LoadConstant(x0, SpillSlots * 16);
LoadConstant(x0, TotalSpillSlotsSize);
sub(sp, sp, x0);
}
}
@@ -795,14 +859,17 @@ void *Arm64JITCore::CompileCode(uint64_t Entry,
}
void Arm64JITCore::ResetStack() {
if (SpillSlots == 0)
if (SpillSlots == 0) {
return;
}
if (IsImmAddSub(SpillSlots * 16)) {
add(sp, sp, SpillSlots * 16);
const auto TotalSpillSlotsSize = SpillSlots * MaxSpillSlotSize;
if (IsImmAddSub(TotalSpillSlotsSize)) {
add(sp, sp, TotalSpillSlotsSize);
} else {
// Too big to fit in a 12bit immediate
LoadConstant(x0, SpillSlots * 16);
LoadConstant(x0, TotalSpillSlotsSize);
add(sp, sp, x0);
}
}
+12 -16
View File
@@ -23,16 +23,6 @@ $end_info$
#include <utility>
#include <vector>
#define STATE x28
#define TMP1 x0
#define TMP2 x1
#define TMP3 x2
#define TMP4 x3
#define VTMP1 v1
#define VTMP2 v2
#define VTMP3 v3
namespace FEXCore::Core {
struct InternalThreadState;
}
@@ -66,6 +56,7 @@ public:
private:
FEX_CONFIG_OPT(ParanoidTSO, PARANOIDTSO);
const bool HostSupportsSVE{};
Label *PendingTargetLabel;
FEXCore::Context::Context *CTX;
@@ -127,6 +118,17 @@ private:
IR::MemOffsetType OffsetType,
uint8_t OffsetScale);
// NOTE: Will use TMP1 as a way to encode immediates that happen to fall outside
// the limits of the scalar plus immediate variant of SVE load/stores.
//
// TMP1 is safe to use again once this memory operand is used with its
// equivalent loads or stores that this was called for.
[[nodiscard]] SVEMemOperand GenerateSVEMemOperand(uint8_t AccessSize,
aarch64::Register Base,
IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType,
uint8_t OffsetScale);
[[nodiscard]] bool IsInlineConstant(const IR::OrderedNodeWrapper& Node, uint64_t* Value = nullptr) const;
[[nodiscard]] bool IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode, uint64_t* Value) const;
@@ -364,13 +366,10 @@ private:
///< Move ops
DEF_OP(ExtractElementPair);
DEF_OP(CreateElementPair);
DEF_OP(Mov);
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(SplatVector2);
DEF_OP(SplatVector4);
DEF_OP(VMov);
DEF_OP(VAnd);
DEF_OP(VBic);
@@ -429,8 +428,6 @@ private:
DEF_OP(VUShrS);
DEF_OP(VSShrS);
DEF_OP(VInsElement);
DEF_OP(VInsScalarElement);
DEF_OP(VExtractElement);
DEF_OP(VDupElement);
DEF_OP(VExtr);
DEF_OP(VSLI);
@@ -440,7 +437,6 @@ private:
DEF_OP(VShlI);
DEF_OP(VUShrNI);
DEF_OP(VUShrNI2);
DEF_OP(VBitcast);
DEF_OP(VSXTL);
DEF_OP(VSXTL2);
DEF_OP(VUXTL);
File diff suppressed because it is too large. Load diff
@@ -119,8 +119,11 @@ DEF_OP(SetRoundingMode) {
mrs(TMP1, FPCR);
// vixl simulator doesn't support anything beyond ties-to-even rounding
#ifndef VIXL_SIMULATOR
// Insert the rounding flags
bfi(TMP1, TMP2, 22, 2);
#endif
// Insert the FTZ flag
lsr(TMP2, Src, 2);
@@ -134,6 +137,7 @@ DEF_OP(Print) {
auto Op = IROp->C<IR::IROp_Print>();
PushDynamicRegsAndLR();
SpillStaticRegs();
if (IsGPR(Op->Value.ID())) {
mov(x0, GetReg<RA_64>(Op->Value.ID()));
@@ -145,10 +149,10 @@ DEF_OP(Print) {
fmov(x1, GetSrc(Op->Value.ID()).V1D(), 1);
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.PrintVectorValue)));
}
SpillStaticRegs();
blr(x3);
FillStaticRegs();
blr(x3);
FillStaticRegs();
PopDynamicRegsAndLR();
}
@@ -68,17 +68,11 @@ DEF_OP(CreateElementPair) {
}
}
DEF_OP(Mov) {
auto Op = IROp->C<IR::IROp_Mov>();
mov(GetReg<RA_64>(Node), GetReg<RA_64>(Op->Value.ID()));
}
#undef DEF_OP
void Arm64JITCore::RegisterMoveHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
REGISTER_OP(EXTRACTELEMENTPAIR, ExtractElementPair);
REGISTER_OP(CREATEELEMENTPAIR, CreateElementPair);
REGISTER_OP(MOV, Mov);
#undef REGISTER_OP
}
}
File diff suppressed because it is too large. Load diff
+44 -11
View File
@@ -1143,26 +1143,59 @@ DEF_OP(Select) {
}
DEF_OP(VExtractToGPR) {
auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const auto Op = IROp->C<IR::IROp_VExtractToGPR>();
switch (Op->Header.ElementSize) {
constexpr auto SSERegSize = Core::CPUState::XMM_SSE_REG_SIZE;
constexpr auto SSEBitSize = SSERegSize * 8;
const auto ElementSize = Op->Header.ElementSize;
const auto ElementSizeBits = ElementSize * 8;
const auto Offset = ElementSizeBits * Op->Index;
const auto Is256Bit = Offset >= SSEBitSize;
const auto Vector = GetSrc(Op->Vector.ID());
switch (ElementSize) {
case 1: {
pextrb(GetDst<RA_32>(Node), GetSrc(Op->Vector.ID()), Op->Index);
break;
if (Is256Bit) {
vextracti128(xmm15, ToYMM(Vector), 1);
pextrb(GetDst<RA_32>(Node), xmm15, Op->Index - 16);
} else {
pextrb(GetDst<RA_32>(Node), Vector, Op->Index);
}
break;
}
case 2: {
pextrw(GetDst<RA_32>(Node), GetSrc(Op->Vector.ID()), Op->Index);
break;
if (Is256Bit) {
vextracti128(xmm15, ToYMM(Vector), 1);
pextrw(GetDst<RA_32>(Node), xmm15, Op->Index - 8);
} else {
pextrw(GetDst<RA_32>(Node), Vector, Op->Index);
}
break;
}
case 4: {
pextrd(GetDst<RA_32>(Node), GetSrc(Op->Vector.ID()), Op->Index);
break;
if (Is256Bit) {
vextracti128(xmm15, ToYMM(Vector), 1);
pextrd(GetDst<RA_32>(Node), xmm15, Op->Index - 4);
} else {
pextrd(GetDst<RA_32>(Node), Vector, Op->Index);
}
break;
}
case 8: {
pextrq(GetDst<RA_64>(Node), GetSrc(Op->Vector.ID()), Op->Index);
break;
if (Is256Bit) {
vextracti128(xmm15, ToYMM(Vector), 1);
pextrq(GetDst<RA_64>(Node), xmm15, Op->Index - 2);
} else {
pextrq(GetDst<RA_64>(Node), Vector, Op->Index);
}
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
}
@@ -33,7 +33,7 @@ namespace FEXCore::CPU {
DEF_OP(SignalReturn) {
// Adjust the stack first for a regular return
if (SpillSlots) {
add(rsp, SpillSlots * 16); // + 8 to consume return address
add(rsp, SpillSlots * MaxSpillSlotSize); // + 8 to consume return address
}
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.SignalReturnHandler)]);
@@ -42,7 +42,7 @@ DEF_OP(SignalReturn) {
DEF_OP(CallbackReturn) {
// Adjust the stack first for a regular return
if (SpillSlots) {
add(rsp, SpillSlots * 16); // + 8 to consume return address
add(rsp, SpillSlots * MaxSpillSlotSize); // + 8 to consume return address
}
// Make sure to adjust the refcounter so we don't clear the cache now
@@ -71,7 +71,7 @@ DEF_OP(ExitFunction) {
if (SpillSlots) {
add(rsp, SpillSlots * 16);
add(rsp, SpillSlots * MaxSpillSlotSize);
}
uint64_t NewRIP;
@@ -17,27 +17,76 @@ namespace FEXCore::CPU {
#define DEF_OP(x) void X86JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(VInsGPR) {
auto Op = IROp->C<IR::IROp_VInsGPR>();
movapd(GetDst(Node), GetSrc(Op->DestVector.ID()));
const auto Op = IROp->C<IR::IROp_VInsGPR>();
const auto OpSize = IROp->Size;
switch (Op->Header.ElementSize) {
case 1: {
pinsrb(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
const auto Dst = GetDst(Node);
const auto DestVector = GetSrc(Op->DestVector.ID());
const auto DestIdx = Op->DestIdx;
const auto ElementSize = Op->Header.ElementSize;
const auto ElementSizeBits = ElementSize * 8;
const auto Offset = ElementSizeBits * DestIdx;
constexpr auto SSEBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto InUpperLane = Offset >= SSEBitSize;
if (InUpperLane && !Is256Bit) {
LOGMAN_MSG_A_FMT("Attempt to access upper 128-bit lane in 128-bit operation! Offset={}",
Offset);
return;
}
if (Is256Bit) {
vmovapd(ToYMM(Dst), ToYMM(DestVector));
} else {
vmovapd(Dst, DestVector);
}
const auto Insert = [&](const Xbyak::Xmm& reg, int index) {
switch (ElementSize) {
case 1: {
if (InUpperLane) {
index -= 16;
}
pinsrb(reg, GetSrc<RA_32>(Op->Src.ID()), index);
break;
}
case 2: {
if (InUpperLane) {
index -= 8;
}
pinsrw(reg, GetSrc<RA_32>(Op->Src.ID()), index);
break;
}
case 4: {
if (InUpperLane) {
index -= 4;
}
pinsrd(reg, GetSrc<RA_32>(Op->Src.ID()), index);
break;
}
case 8: {
if (InUpperLane) {
index -= 2;
}
pinsrq(reg, GetSrc<RA_64>(Op->Src.ID()), index);
break;
}
default:
LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize);
break;
}
case 2: {
pinsrw(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
}
case 4: {
pinsrd(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
}
case 8: {
pinsrq(GetDst(Node), GetSrc<RA_64>(Op->Src.ID()), Op->DestIdx);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
};
if (InUpperLane) {
vextracti128(xmm15, ToYMM(Dst), 1);
Insert(xmm15, DestIdx);
vinserti128(ToYMM(Dst), ToYMM(Dst), xmm15, 1);
} else {
Insert(Dst, DestIdx);
}
}
@@ -63,8 +112,10 @@ DEF_OP(VCastFromGPR) {
}
DEF_OP(Float_FromGPR_S) {
auto Op = IROp->C<IR::IROp_Float_FromGPR_S>();
const uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const auto Op = IROp->C<IR::IROp_Float_FromGPR_S>();
const uint16_t ElementSize = Op->Header.ElementSize;
const uint16_t Conv = (ElementSize << 8) | Op->SrcElementSize;
switch (Conv) {
case 0x0404: { // Float <- int32_t
@@ -83,6 +134,10 @@ DEF_OP(Float_FromGPR_S) {
cvtsi2sd(GetDst(Node), GetSrc<RA_64>(Op->Src.ID()));
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled conversion mask: Mask=0x{:04x}, ElementSize={}, SrcElementSize={}",
Conv, ElementSize, Op->SrcElementSize);
break;
}
}
@@ -104,99 +159,194 @@ DEF_OP(Float_FToF) {
}
DEF_OP(Vector_SToF) {
auto Op = IROp->C<IR::IROp_Vector_SToF>();
switch (Op->Header.ElementSize) {
const auto Op = IROp->C<IR::IROp_Vector_SToF>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
switch (ElementSize) {
case 4:
cvtdq2ps(GetDst(Node), GetSrc(Op->Vector.ID()));
break;
if (Is256Bit) {
vcvtdq2ps(ToYMM(Dst), ToYMM(Vector));
} else {
vcvtdq2ps(Dst, Vector);
}
break;
case 8:
// This operation is a bit disgusting in x86
// There is no vector form of this instruction until AVX512VL + AVX512DQ (vcvtqq2pd)
// 1) First extract the top 64bits
// 2) Do a scalar conversion on each
// 3) Make sure to merge them together at the end
pextrq(rax, GetSrc(Op->Vector.ID()), 1);
pextrq(rcx, GetSrc(Op->Vector.ID()), 0);
cvtsi2sd(GetDst(Node), rcx);
pextrq(rax, Vector, 1);
pextrq(rcx, Vector, 0);
cvtsi2sd(Dst, rcx);
cvtsi2sd(xmm15, rax);
movlhps(GetDst(Node), xmm15);
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_SToF element size: {}", Op->Header.ElementSize);
vmovlhps(Dst, Dst, xmm15);
if (Is256Bit) {
vextracti128(xmm15, ToYMM(Vector), 1);
pextrq(rax, xmm15, 1);
pextrq(rcx, xmm15, 0);
cvtsi2sd(xmm15, rcx);
cvtsi2sd(xmm14, rax);
movlhps(xmm15, xmm14);
vinserti128(ToYMM(Dst), ToYMM(Dst), xmm15, 1);
}
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_SToF element size: {}", ElementSize);
break;
}
}
DEF_OP(Vector_FToZS) {
auto Op = IROp->C<IR::IROp_Vector_FToZS>();
switch (Op->Header.ElementSize) {
const auto Op = IROp->C<IR::IROp_Vector_FToZS>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
switch (ElementSize) {
case 4:
cvttps2dq(GetDst(Node), GetSrc(Op->Vector.ID()));
break;
if (Is256Bit) {
vcvttps2dq(ToYMM(Dst), ToYMM(Vector));
} else {
vcvttps2dq(Dst, Vector);
}
break;
case 8:
cvttpd2dq(GetDst(Node), GetSrc(Op->Vector.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToZS element size: {}", Op->Header.ElementSize);
if (Is256Bit) {
vcvttpd2dq(ToYMM(Dst), ToYMM(Vector));
} else {
vcvttpd2dq(Dst, Vector);
}
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToZS element size: {}", ElementSize);
break;
}
}
DEF_OP(Vector_FToS) {
auto Op = IROp->C<IR::IROp_Vector_FToS>();
switch (Op->Header.ElementSize) {
const auto Op = IROp->C<IR::IROp_Vector_FToS>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
switch (ElementSize) {
case 4:
cvtps2dq(GetDst(Node), GetSrc(Op->Vector.ID()));
break;
if (Is256Bit) {
vcvtps2dq(ToYMM(Dst), ToYMM(Vector));
} else {
vcvtps2dq(Dst, Vector);
}
break;
case 8:
cvtpd2dq(GetDst(Node), GetSrc(Op->Vector.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToS element size: {}", Op->Header.ElementSize);
if (Is256Bit) {
vcvtpd2dq(ToYMM(Dst), ToYMM(Vector));
} else {
vcvtpd2dq(Dst, Vector);
}
break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToS element size: {}", ElementSize);
break;
}
}
DEF_OP(Vector_FToF) {
auto Op = IROp->C<IR::IROp_Vector_FToF>();
const uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const auto Op = IROp->C<IR::IROp_Vector_FToF>();
const auto OpSize = IROp->Size;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Conv = (ElementSize << 8) | Op->SrcElementSize;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
switch (Conv) {
case 0x0804: { // Double <- Float
cvtps2pd(GetDst(Node), GetSrc(Op->Vector.ID()));
if (Is256Bit) {
vcvtps2pd(ToYMM(Dst), Vector);
} else {
vcvtps2pd(Dst, Vector);
}
break;
}
case 0x0408: { // Float <- Double
cvtpd2ps(GetDst(Node), GetSrc(Op->Vector.ID()));
if (Is256Bit) {
vcvtpd2ps(Dst, ToYMM(Vector));
} else {
vcvtpd2ps(Dst, Vector);
}
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Vector_FToF conversion type : 0x{:04x}", Conv); break;
default:
LOGMAN_MSG_A_FMT("Unknown Vector_FToF conversion type : 0x{:04x}", Conv);
break;
}
}
DEF_OP(Vector_FToI) {
auto Op = IROp->C<IR::IROp_Vector_FToI>();
uint8_t RoundMode{};
const auto Op = IROp->C<IR::IROp_Vector_FToI>();
const auto OpSize = IROp->Size;
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
RoundMode = 0b0000'0'0'00;
break;
case FEXCore::IR::Round_Negative_Infinity.Val:
RoundMode = 0b0000'0'0'01;
break;
case FEXCore::IR::Round_Positive_Infinity.Val:
RoundMode = 0b0000'0'0'10;
break;
case FEXCore::IR::Round_Towards_Zero.Val:
RoundMode = 0b0000'0'0'11;
break;
case FEXCore::IR::Round_Host.Val:
RoundMode = 0b0000'0'1'00;
break;
}
const uint8_t RoundMode = [Op] {
switch (Op->Round) {
case FEXCore::IR::Round_Nearest.Val:
return 0b0000'0'0'00;
case FEXCore::IR::Round_Negative_Infinity.Val:
return 0b0000'0'0'01;
case FEXCore::IR::Round_Positive_Infinity.Val:
return 0b0000'0'0'10;
case FEXCore::IR::Round_Towards_Zero.Val:
return 0b0000'0'0'11;
case FEXCore::IR::Round_Host.Val:
return 0b0000'0'1'00;
default:
LOGMAN_MSG_A_FMT("Unhandled rounding mode");
return 0;
}
}();
switch (Op->Header.ElementSize) {
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Dst = GetDst(Node);
const auto Vector = GetSrc(Op->Vector.ID());
switch (ElementSize) {
case 4:
roundps(GetDst(Node), GetSrc(Op->Vector.ID()), RoundMode);
break;
if (Is256Bit) {
vroundps(ToYMM(Dst), ToYMM(Vector), RoundMode);
} else {
vroundps(Dst, Vector, RoundMode);
}
break;
case 8:
roundpd(GetDst(Node), GetSrc(Op->Vector.ID()), RoundMode);
break;
if (Is256Bit) {
vroundpd(ToYMM(Dst), ToYMM(Vector), RoundMode);
} else {
vroundpd(Dst, Vector, RoundMode);
}
break;
default:
LOGMAN_MSG_A_FMT("Unhandled element size: {}", ElementSize);
break;
}
}
+33 -18
View File
@@ -25,7 +25,9 @@ $end_info$
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/IR/RegisterAllocationData.h>
#include <FEXCore/Utils/Allocator.h>
#include <FEXCore/Utils/EnumUtils.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <algorithm>
#include <array>
@@ -59,32 +61,42 @@ static void PrintVectorValue(uint64_t Value, uint64_t ValueUpper) {
namespace FEXCore::CPU {
void X86JITCore::PushRegs() {
sub(rsp, 16 * RAXMM_x.size());
const auto AVXRegSize = Core::CPUState::XMM_AVX_REG_SIZE;
sub(rsp, AVXRegSize * RAXMM_x.size());
for (size_t i = 0; i < RAXMM_x.size(); ++i) {
movaps(ptr[rsp + i * 16], RAXMM_x[i]);
vmovups(ptr[rsp + i * AVXRegSize], ToYMM(RAXMM_x[i]));
}
for (auto &Reg : RA64)
for (const auto &Reg : RA64) {
push(Reg);
}
auto NumPush = RA64.size();
if (NumPush & 1)
sub(rsp, 8); // Align
const auto NumPush = RA64.size();
if ((NumPush & 1) != 0) {
// Align
sub(rsp, 8);
}
}
void X86JITCore::PopRegs() {
auto NumPush = RA64.size();
const auto AVXRegSize = Core::CPUState::XMM_AVX_REG_SIZE;
const auto NumPush = RA64.size();
if (NumPush & 1)
add(rsp, 8); // Align
for (uint32_t i = RA64.size(); i > 0; --i)
pop(RA64[i - 1]);
for (size_t i = 0; i < RAXMM_x.size(); ++i) {
movaps(RAXMM_x[i], ptr[rsp + i * 16]);
if ((NumPush & 1) != 0) {
// Align
add(rsp, 8);
}
add(rsp, 16 * RAXMM_x.size());
for (uint32_t i = RA64.size(); i > 0; --i) {
pop(RA64[i - 1]);
}
for (size_t i = 0; i < RAXMM_x.size(); ++i) {
vmovups(ToYMM(RAXMM_x[i]), ptr[rsp + i * AVXRegSize]);
}
add(rsp, AVXRegSize * RAXMM_x.size());
}
void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
@@ -293,7 +305,8 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
case FABI_UNKNOWN:
default:
#if defined(ASSERTIONS_ENABLED) && ASSERTIONS_ENABLED
LOGMAN_MSG_A_FMT("Unhandled IR Fallback ABI: {} {}", FEXCore::IR::GetName(IROp->Op), Info.ABI);
LOGMAN_MSG_A_FMT("Unhandled IR Fallback ABI: {} {}",
IR::GetName(IROp->Op), ToUnderlying(Info.ABI));
#endif
break;
}
@@ -358,7 +371,7 @@ X86JITCore::X86JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::InternalTh
{
auto &Common = ThreadState->CurrentFrame->Pointers.Common;
Common.PrintValue = reinterpret_cast<uint64_t>(PrintValue);
Common.PrintVectorValue = reinterpret_cast<uint64_t>(PrintVectorValue);
Common.ThreadRemoveCodeEntryFromJIT = reinterpret_cast<uintptr_t>(&Context::Context::ThreadRemoveCodeEntryFromJit);
@@ -570,6 +583,8 @@ std::tuple<X86JITCore::SetCC, X86JITCore::CMovCC, X86JITCore::JCC> X86JITCore::G
}
void *X86JITCore::CompileCode(uint64_t Entry, [[maybe_unused]] FEXCore::IR::IRListView const *IR, [[maybe_unused]] FEXCore::Core::DebugData *DebugData, FEXCore::IR::RegisterAllocationData *RAData, bool GDBEnabled) {
FEXCORE_PROFILE_SCOPED("x86::CompileCode");
JumpTargets.clear();
uint32_t SSACount = IR->GetSSACount();
@@ -597,7 +612,7 @@ void *X86JITCore::CompileCode(uint64_t Entry, [[maybe_unused]] FEXCore::IR::IRLi
SpillSlots = RAData->SpillSlots();
if (SpillSlots) {
sub(rsp, SpillSlots * 16);
sub(rsp, SpillSlots * MaxSpillSlotSize);
}
#ifdef BLOCKSTATS
@@ -181,6 +181,10 @@ private:
[[nodiscard]] Xbyak::Xmm GetSrc(IR::NodeID Node) const;
[[nodiscard]] Xbyak::Xmm GetDst(IR::NodeID Node) const;
[[nodiscard]] static Xbyak::Ymm ToYMM(const Xbyak::Xmm& xmm) {
return Xbyak::Ymm{xmm.getIdx()};
}
[[nodiscard]] Xbyak::RegExp GenerateModRM(Xbyak::Reg Base, IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType, uint8_t OffsetScale) const;
@@ -205,7 +209,7 @@ private:
* @brief Current guest RIP entrypoint
*/
uint8_t *GuestEntry{};
using SetCC = void (X86JITCore::*)(const Operand& op);
using CMovCC = void (X86JITCore::*)(const Reg& reg, const Operand& op);
using JCC = void (X86JITCore::*)(const Label& label, LabelType type);
@@ -362,12 +366,10 @@ private:
///< Move ops
DEF_OP(ExtractElementPair);
DEF_OP(CreateElementPair);
DEF_OP(Mov);
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(SplatVector);
DEF_OP(VMov);
DEF_OP(VAnd);
DEF_OP(VBic);
@@ -426,8 +428,6 @@ private:
DEF_OP(VUShrS);
DEF_OP(VSShrS);
DEF_OP(VInsElement);
DEF_OP(VInsScalarElement);
DEF_OP(VExtractElement);
DEF_OP(VDupElement);
DEF_OP(VExtr);
DEF_OP(VSLI);
@@ -437,7 +437,6 @@ private:
DEF_OP(VShlI);
DEF_OP(VUShrNI);
DEF_OP(VUShrNI2);
DEF_OP(VBitcast);
DEF_OP(VSXTL);
DEF_OP(VSXTL2);
DEF_OP(VUXTL);
+265 -177
View File
@@ -21,130 +21,160 @@ namespace FEXCore::CPU {
#define DEF_OP(x) void X86JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(LoadContext) {
auto Op = IROp->C<IR::IROp_LoadContext>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_LoadContext>();
const auto OpSize = IROp->Size;
if (Op->Class == IR::GPRClass) {
switch (OpSize) {
case 1: {
movzx(GetDst<RA_32>(Node), byte [STATE + Op->Offset]);
break;
}
break;
case 2: {
movzx(GetDst<RA_32>(Node), word [STATE + Op->Offset]);
break;
}
break;
case 4: {
mov(GetDst<RA_32>(Node), dword [STATE + Op->Offset]);
break;
}
break;
case 8: {
mov(GetDst<RA_64>(Node), qword [STATE + Op->Offset]);
break;
}
break;
case 16: {
LOGMAN_MSG_A_FMT("Invalid GPR load of size 16");
break;
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
break;
}
}
else {
const auto Dst = GetDst(Node);
switch (OpSize) {
case 1: {
movzx(rax, byte [STATE + Op->Offset]);
vmovq(GetDst(Node), rax);
vmovq(Dst, rax);
break;
}
break;
case 2: {
movzx(rax, word [STATE + Op->Offset]);
vmovq(GetDst(Node), rax);
vmovq(Dst, rax);
break;
}
break;
case 4: {
vmovd(GetDst(Node), dword [STATE + Op->Offset]);
vmovd(Dst, dword [STATE + Op->Offset]);
break;
}
break;
case 8: {
vmovq(GetDst(Node), qword [STATE + Op->Offset]);
vmovq(Dst, qword [STATE + Op->Offset]);
break;
}
break;
case 16: {
if (Op->Offset % 16 == 0)
movaps(GetDst(Node), xword [STATE + Op->Offset]);
else
movups(GetDst(Node), xword [STATE + Op->Offset]);
if (Op->Offset % 16 == 0) {
vmovaps(Dst, xword [STATE + Op->Offset]);
} else {
vmovups(Dst, xword [STATE + Op->Offset]);
}
break;
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
case 32: {
if (Op->Offset % 32 == 0) {
vmovaps(ToYMM(Dst), yword [STATE + Op->Offset]);
} else {
vmovups(ToYMM(Dst), yword [STATE + Op->Offset]);
}
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContext size: {}", OpSize);
break;
}
}
}
DEF_OP(StoreContext) {
auto Op = IROp->C<IR::IROp_StoreContext>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_StoreContext>();
const auto OpSize = IROp->Size;
if (Op->Class == IR::GPRClass) {
switch (OpSize) {
case 1: {
mov(byte [STATE + Op->Offset], GetSrc<RA_8>(Op->Value.ID()));
break;
}
break;
case 2: {
mov(word [STATE + Op->Offset], GetSrc<RA_16>(Op->Value.ID()));
break;
}
break;
case 4: {
mov(dword [STATE + Op->Offset], GetSrc<RA_32>(Op->Value.ID()));
break;
}
break;
case 8: {
mov(qword [STATE + Op->Offset], GetSrc<RA_64>(Op->Value.ID()));
break;
}
break;
case 16:
LogMan::Msg::DFmt("Invalid store size of 16");
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreContext size: {}", OpSize);
case 16: {
LOGMAN_MSG_A_FMT("Invalid store size of 16");
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled StoreContext size: {}", OpSize);
break;
}
}
else {
const auto Value = GetSrc(Op->Value.ID());
switch (OpSize) {
case 1: {
pextrb(byte [STATE + Op->Offset], GetSrc(Op->Value.ID()), 0);
pextrb(byte [STATE + Op->Offset], Value, 0);
break;
}
break;
case 2: {
pextrw(word [STATE + Op->Offset], GetSrc(Op->Value.ID()), 0);
pextrw(word [STATE + Op->Offset], Value, 0);
break;
}
break;
case 4: {
vmovd(dword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
vmovd(dword [STATE + Op->Offset], Value);
break;
}
break;
case 8: {
vmovq(qword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
vmovq(qword [STATE + Op->Offset], Value);
break;
}
break;
case 16: {
if (Op->Offset % 16 == 0)
movaps(xword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
else
movups(xword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
if (Op->Offset % 16 == 0) {
vmovaps(xword [STATE + Op->Offset], Value);
} else {
vmovups(xword [STATE + Op->Offset], Value);
}
break;
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreContext size: {}", OpSize);
case 32: {
if (Op->Offset % 32 == 0) {
vmovaps(yword [STATE + Op->Offset], ToYMM(Value));
} else {
vmovups(yword [STATE + Op->Offset], ToYMM(Value));
}
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled StoreContext size: {}", OpSize);
break;
}
}
}
DEF_OP(LoadContextIndexed) {
auto Op = IROp->C<IR::IROp_LoadContextIndexed>();
size_t size = IROp->Size;
Reg index = GetSrc<RA_64>(Op->Index.ID());
const auto Op = IROp->C<IR::IROp_LoadContextIndexed>();
const auto OpSize = IROp->Size;
const Reg Index = GetSrc<RA_64>(Op->Index.ID());
if (Op->Class == IR::GPRClass) {
switch (Op->Stride) {
@@ -153,21 +183,21 @@ DEF_OP(LoadContextIndexed) {
case 4:
case 8: {
lea(rax, dword [STATE + Op->BaseOffset]);
switch (size) {
switch (OpSize) {
case 1:
movzx(GetDst<RA_32>(Node), byte [rax + index * Op->Stride]);
movzx(GetDst<RA_32>(Node), byte [rax + Index * Op->Stride]);
break;
case 2:
movzx(GetDst<RA_32>(Node), word [rax + index * Op->Stride]);
movzx(GetDst<RA_32>(Node), word [rax + Index * Op->Stride]);
break;
case 4:
mov(GetDst<RA_32>(Node), dword [rax + index * Op->Stride]);
mov(GetDst<RA_32>(Node), dword [rax + Index * Op->Stride]);
break;
case 8:
mov(GetDst<RA_64>(Node), qword [rax + index * Op->Stride]);
mov(GetDst<RA_64>(Node), qword [rax + Index * Op->Stride]);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", IROp->Size);
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", OpSize);
break;
}
break;
@@ -186,53 +216,67 @@ DEF_OP(LoadContextIndexed) {
case 2:
case 4:
case 8: {
const auto Dst = GetDst(Node);
lea(rax, dword [STATE + Op->BaseOffset]);
switch (size) {
switch (OpSize) {
case 1:
movzx(eax, byte [rax + index * Op->Stride]);
vmovd(GetDst(Node), eax);
movzx(eax, byte [rax + Index * Op->Stride]);
vmovd(Dst, eax);
break;
case 2:
movzx(eax, word [rax + index * Op->Stride]);
vmovd(GetDst(Node), eax);
movzx(eax, word [rax + Index * Op->Stride]);
vmovd(Dst, eax);
break;
case 4:
vmovd(GetDst(Node), dword [rax + index * Op->Stride]);
vmovd(Dst, dword [rax + Index * Op->Stride]);
break;
case 8:
vmovq(GetDst(Node), qword [rax + index * Op->Stride]);
vmovq(Dst, qword [rax + Index * Op->Stride]);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", IROp->Size);
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", OpSize);
break;
}
break;
}
case 16: {
mov(rax, index);
shl(rax, 4);
case 16:
case 32: {
const auto Dst = GetDst(Node);
const auto Shift = Op->Stride == 16 ? 4 : 5;
mov(rax, Index);
shl(rax, Shift);
lea(rax, dword [rax + Op->BaseOffset]);
switch (size) {
switch (OpSize) {
case 1:
pinsrb(GetDst(Node), byte [STATE + rax], 0);
pinsrb(Dst, byte [STATE + rax], 0);
break;
case 2:
pinsrw(GetDst(Node), word [STATE + rax], 0);
pinsrw(Dst, word [STATE + rax], 0);
break;
case 4:
vmovd(GetDst(Node), dword [STATE + rax]);
vmovd(Dst, dword [STATE + rax]);
break;
case 8:
vmovq(GetDst(Node), qword [STATE + rax]);
vmovq(Dst, qword [STATE + rax]);
break;
case 16:
if (Op->BaseOffset % 16 == 0)
movaps(GetDst(Node), xword [STATE + rax]);
else
movups(GetDst(Node), xword [STATE + rax]);
if (Op->BaseOffset % 16 == 0) {
vmovaps(Dst, xword [STATE + rax]);
} else {
vmovups(Dst, xword [STATE + rax]);
}
break;
case 32:
if (Op->BaseOffset % 32 == 0) {
vmovaps(ToYMM(Dst), yword [STATE + rax]);
} else {
vmovups(ToYMM(Dst), yword [STATE + rax]);
}
break;
default:
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", IROp->Size);
LOGMAN_MSG_A_FMT("Unhandled LoadContextIndexed size: {}", OpSize);
break;
}
break;
@@ -245,12 +289,13 @@ DEF_OP(LoadContextIndexed) {
}
DEF_OP(StoreContextIndexed) {
auto Op = IROp->C<IR::IROp_StoreContextIndexed>();
Reg index = GetSrc<RA_64>(Op->Index.ID());
size_t size = IROp->Size;
const auto Op = IROp->C<IR::IROp_StoreContextIndexed>();
const auto OpSize = IROp->Size;
const Reg Index = GetSrc<RA_64>(Op->Index.ID());
if (Op->Class == IR::GPRClass) {
auto value = GetSrc<RA_64>(Op->Value.ID());
const auto Value = GetSrc<RA_64>(Op->Value.ID());
lea(rax, dword [STATE + Op->BaseOffset]);
switch (Op->Stride) {
@@ -258,10 +303,10 @@ DEF_OP(StoreContextIndexed) {
case 2:
case 4:
case 8: {
if (!(size == 1 || size == 2 || size == 4 || size == 8)) {
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", IROp->Size);
if (!(OpSize == 1 || OpSize == 2 || OpSize == 4 || OpSize == 8)) {
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", OpSize);
}
mov(AddressFrame(IROp->Size * 8) [rax + index * Op->Stride], value);
mov(AddressFrame(OpSize * 8) [rax + Index * Op->Stride], Value);
break;
}
default:
@@ -270,57 +315,68 @@ DEF_OP(StoreContextIndexed) {
}
}
else {
auto value = GetSrc(Op->Value.ID());
const auto Value = GetSrc(Op->Value.ID());
switch (Op->Stride) {
case 1:
case 2:
case 4:
case 8: {
lea(rax, dword [STATE + Op->BaseOffset]);
switch (size) {
switch (OpSize) {
case 1:
pextrb(AddressFrame(IROp->Size * 8) [rax + index * Op->Stride], value, 0);
pextrb(AddressFrame(OpSize * 8) [rax + Index * Op->Stride], Value, 0);
break;
case 2:
pextrw(AddressFrame(IROp->Size * 8) [rax + index * Op->Stride], value, 0);
pextrw(AddressFrame(OpSize * 8) [rax + Index * Op->Stride], Value, 0);
break;
case 4:
vmovd(AddressFrame(IROp->Size * 8) [rax + index * Op->Stride], value);
vmovd(AddressFrame(OpSize * 8) [rax + Index * Op->Stride], Value);
break;
case 8:
vmovq(AddressFrame(IROp->Size * 8) [rax + index * Op->Stride], value);
vmovq(AddressFrame(OpSize * 8) [rax + Index * Op->Stride], Value);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", size);
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", OpSize);
break;
}
break;
}
case 16: {
mov(rax, index);
shl(rax, 4);
case 16:
case 32: {
const auto Shift = Op->Stride == 16 ? 4 : 5;
mov(rax, Index);
shl(rax, Shift);
lea(rax, dword [rax + Op->BaseOffset]);
switch (size) {
switch (OpSize) {
case 1:
pextrb(AddressFrame(IROp->Size * 8) [STATE + rax], value, 0);
pextrb(AddressFrame(OpSize * 8) [STATE + rax], Value, 0);
break;
case 2:
pextrw(AddressFrame(IROp->Size * 8) [STATE + rax], value, 0);
pextrw(AddressFrame(OpSize * 8) [STATE + rax], Value, 0);
break;
case 4:
vmovd(AddressFrame(IROp->Size * 8) [STATE + rax], value);
vmovd(AddressFrame(OpSize * 8) [STATE + rax], Value);
break;
case 8:
vmovq(AddressFrame(IROp->Size * 8) [STATE + rax], value);
vmovq(AddressFrame(OpSize * 8) [STATE + rax], Value);
break;
case 16:
if (Op->BaseOffset % 16 == 0)
movaps(xword [STATE + rax], value);
else
movups(xword [STATE + rax], value);
if (Op->BaseOffset % 16 == 0) {
vmovaps(xword [STATE + rax], Value);
} else {
vmovups(xword [STATE + rax], Value);
}
break;
case 32:
if (Op->BaseOffset % 32 == 0) {
vmovaps(yword [STATE + rax], ToYMM(Value));
} else {
vmovups(yword [STATE + rax], ToYMM(Value));
}
break;
default:
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", size);
LOGMAN_MSG_A_FMT("Unhandled StoreContextIndexed size: {}", OpSize);
break;
}
break;
@@ -333,10 +389,10 @@ DEF_OP(StoreContextIndexed) {
}
DEF_OP(SpillRegister) {
auto Op = IROp->C<IR::IROp_SpillRegister>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_SpillRegister>();
const uint8_t OpSize = IROp->Size;
const uint32_t SlotOffset = Op->Slot * MaxSpillSlotSize;
uint32_t SlotOffset = Op->Slot * 16;
if (Op->Class == FEXCore::IR::GPRClass) {
switch (OpSize) {
case 1: {
@@ -355,36 +411,44 @@ DEF_OP(SpillRegister) {
mov(qword [rsp + SlotOffset], GetSrc<RA_64>(Op->Value.ID()));
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
default:
LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
break;
}
} else if (Op->Class == FEXCore::IR::FPRClass) {
const auto Src = GetSrc(Op->Value.ID());
switch (OpSize) {
case 4: {
movss(dword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
movss(dword [rsp + SlotOffset], Src);
break;
}
case 8: {
movsd(qword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
movsd(qword [rsp + SlotOffset], Src);
break;
}
case 16: {
movaps(xword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
movaps(xword [rsp + SlotOffset], Src);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
case 32: {
vmovaps(yword [rsp + SlotOffset], ToYMM(Src));
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
break;
}
} else {
LOGMAN_MSG_A_FMT("Unhandled SpillRegister class: {}", Op->Class.Val);
}
}
DEF_OP(FillRegister) {
auto Op = IROp->C<IR::IROp_FillRegister>();
uint8_t OpSize = IROp->Size;
const auto Op = IROp->C<IR::IROp_FillRegister>();
const uint8_t OpSize = IROp->Size;
const uint32_t SlotOffset = Op->Slot * MaxSpillSlotSize;
uint32_t SlotOffset = Op->Slot * 16;
if (Op->Class == FEXCore::IR::GPRClass) {
switch (OpSize) {
case 1: {
@@ -403,23 +467,33 @@ DEF_OP(FillRegister) {
mov(GetDst<RA_64>(Node), qword [rsp + SlotOffset]);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled FillRegister size: {}", OpSize);
default:
LOGMAN_MSG_A_FMT("Unhandled FillRegister size: {}", OpSize);
break;
}
} else if (Op->Class == FEXCore::IR::FPRClass) {
const auto Dst = GetDst(Node);
switch (OpSize) {
case 4: {
movss(GetDst(Node), dword [rsp + SlotOffset]);
vmovss(Dst, dword [rsp + SlotOffset]);
break;
}
case 8: {
movsd(GetDst(Node), qword [rsp + SlotOffset]);
vmovsd(Dst, qword [rsp + SlotOffset]);
break;
}
case 16: {
movaps(GetDst(Node), xword [rsp + SlotOffset]);
vmovaps(Dst, xword [rsp + SlotOffset]);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled FillRegister size: {}", OpSize);
case 32: {
vmovaps(ToYMM(Dst), yword [rsp + SlotOffset]);
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled FillRegister size: {}", OpSize);
break;
}
} else {
LOGMAN_MSG_A_FMT("Unhandled FillRegister class: {}", Op->Class.Val);
@@ -464,118 +538,132 @@ Xbyak::RegExp X86JITCore::GenerateModRM(Xbyak::Reg Base, IR::OrderedNodeWrapper
}
DEF_OP(LoadMem) {
auto Op = IROp->C<IR::IROp_LoadMem>();
const auto Op = IROp->C<IR::IROp_LoadMem>();
const auto OpSize = IROp->Size;
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
const Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
const auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
if (Op->Class == IR::GPRClass) {
auto Dst = GetDst<RA_64>(Node);
const auto Dst = GetDst<RA_64>(Node);
switch (IROp->Size) {
switch (OpSize) {
case 1: {
movzx (Dst, byte [MemPtr]);
movzx(Dst, byte [MemPtr]);
break;
}
break;
case 2: {
movzx (Dst, word [MemPtr]);
movzx(Dst, word [MemPtr]);
break;
}
break;
case 4: {
mov(Dst.cvt32(), dword [MemPtr]);
break;
}
break;
case 8: {
mov(Dst, qword [MemPtr]);
break;
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled LoadMem size: {}", IROp->Size);
default:
LOGMAN_MSG_A_FMT("Unhandled LoadMem size: {}", OpSize);
break;
}
}
else
{
auto Dst = GetDst(Node);
const auto Dst = GetDst(Node);
switch (IROp->Size) {
switch (OpSize) {
case 1: {
movzx(eax, byte [MemPtr]);
vmovd(Dst, eax);
break;
}
break;
case 2: {
movzx(eax, word [MemPtr]);
vmovd(Dst, eax);
break;
}
break;
case 4: {
vmovd(Dst, dword [MemPtr]);
break;
}
break;
case 8: {
vmovq(Dst, qword [MemPtr]);
break;
}
break;
case 16: {
if (IROp->Size == Op->Align)
movups(GetDst(Node), xword [MemPtr]);
else
movups(GetDst(Node), xword [MemPtr]);
if (MemoryDebug) {
movq(rcx, GetDst(Node));
}
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled LoadMem size: {}", IROp->Size);
vmovups(Dst, xword [MemPtr]);
if (MemoryDebug) {
movq(rcx, Dst);
}
break;
}
case 32: {
vmovups(ToYMM(Dst), yword [MemPtr]);
if (MemoryDebug) {
movq(rcx, Dst);
}
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled LoadMem size: {}", OpSize);
break;
}
}
}
DEF_OP(StoreMem) {
auto Op = IROp->C<IR::IROp_StoreMem>();
const auto Op = IROp->C<IR::IROp_StoreMem>();
const auto OpSize = IROp->Size;
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
const Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
const auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
if (Op->Class == IR::GPRClass) {
switch (IROp->Size) {
switch (OpSize) {
case 1:
mov(byte [MemPtr], GetSrc<RA_8>(Op->Value.ID()));
break;
break;
case 2:
mov(word [MemPtr], GetSrc<RA_16>(Op->Value.ID()));
break;
break;
case 4:
mov(dword [MemPtr], GetSrc<RA_32>(Op->Value.ID()));
break;
break;
case 8:
mov(qword [MemPtr], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", IROp->Size);
break;
default:
LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", OpSize);
break;
}
}
else {
switch (IROp->Size) {
const auto Value = GetSrc(Op->Value.ID());
switch (OpSize) {
case 1:
pextrb(byte [MemPtr], GetSrc(Op->Value.ID()), 0);
break;
pextrb(byte [MemPtr], Value, 0);
break;
case 2:
pextrw(word [MemPtr], GetSrc(Op->Value.ID()), 0);
break;
pextrw(word [MemPtr], Value, 0);
break;
case 4:
vmovd(dword [MemPtr], GetSrc(Op->Value.ID()));
break;
vmovd(dword [MemPtr], Value);
break;
case 8:
vmovq(qword [MemPtr], GetSrc(Op->Value.ID()));
break;
vmovq(qword [MemPtr], Value);
break;
case 16:
if (IROp->Size == Op->Align)
movups(xword [MemPtr], GetSrc(Op->Value.ID()));
else
movups(xword [MemPtr], GetSrc(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", IROp->Size);
vmovups(xword [MemPtr], Value);
break;
case 32:
vmovups(yword [MemPtr], ToYMM(Value));
break;
default:
LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", OpSize);
break;
}
}
}
@@ -47,7 +47,7 @@ DEF_OP(Break) {
auto Op = IROp->C<IR::IROp_Break>();
if (SpillSlots) {
add(rsp, SpillSlots * 16);
add(rsp, SpillSlots * MaxSpillSlotSize);
}
mov(byte [STATE + offsetof(FEXCore::Core::CpuStateFrame, SynchronousFaultData.FaultToTopAndGeneratedException)], 1);
@@ -73,17 +73,11 @@ DEF_OP(CreateElementPair) {
}
}
DEF_OP(Mov) {
auto Op = IROp->C<IR::IROp_Mov>();
mov (GetDst<RA_64>(Node), GetSrc<RA_64>(Op->Value.ID()));
}
#undef DEF_OP
void X86JITCore::RegisterMoveHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
REGISTER_OP(EXTRACTELEMENTPAIR, ExtractElementPair);
REGISTER_OP(CREATEELEMENTPAIR, CreateElementPair);
REGISTER_OP(MOV, Mov);
#undef REGISTER_OP
}
}
File diff suppressed because it is too large. Load diff
+359 -60
View File
@@ -126,10 +126,20 @@ void OpDispatchBuilder::ThunkOp(OpcodeArgs) {
const uint8_t GPRSize = CTX->GetGPRSize();
uint8_t *sha256 = (uint8_t *)(Op->PC + 2);
_Thunk(
_LoadContext(GPRSize, GPRClass, GPROffset(X86State::REG_RDI)),
*reinterpret_cast<SHA256Sum*>(sha256)
);
if (CTX->Config.Is64BitMode) {
// x86-64 ABI puts the function argument in RDI
_Thunk(
_LoadContext(GPRSize, GPRClass, GPROffset(X86State::REG_RDI)),
*reinterpret_cast<SHA256Sum*>(sha256)
);
}
else {
// x86 fastcall ABI puts the function argument in ECX
_Thunk(
_LoadContext(GPRSize, GPRClass, GPROffset(X86State::REG_RCX)),
*reinterpret_cast<SHA256Sum*>(sha256)
);
}
auto Constant = _Constant(GPRSize);
auto OldSP = _LoadContext(GPRSize, GPRClass, RSPOffset);
@@ -230,8 +240,11 @@ void OpDispatchBuilder::IRETOp(OpcodeArgs) {
// RIP (64/32/16 bits)
auto NewRIP = _LoadMem(GPRClass, GPRSize, SP, GPRSize);
SP = _Add(SP, Constant);
//CS (lower 16 used)
_StoreContext(2, GPRClass, _LoadMem(GPRClass, GPRSize, SP, GPRSize), offsetof(FEXCore::Core::CPUState, cs));
// CS (lower 16 used)
auto NewSegmentCS = _LoadMem(GPRClass, GPRSize, SP, GPRSize);
_StoreContext(2, GPRClass, NewSegmentCS, offsetof(FEXCore::Core::CPUState, cs_idx));
UpdatePrefixFromSegment(NewSegmentCS, FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX);
SP = _Add(SP, Constant);
//eflags (lower 16 used)
auto eflags = _LoadMem(GPRClass, GPRSize, SP, GPRSize);
@@ -243,8 +256,11 @@ void OpDispatchBuilder::IRETOp(OpcodeArgs) {
// FEX doesn't support a CPL mode switch, so don't need to worry about this on 32-bit
_StoreContext(GPRSize, GPRClass, _LoadMem(GPRClass, GPRSize, SP, GPRSize), RSPOffset);
SP = _Add(SP, Constant);
//ss
_StoreContext(2, GPRClass, _LoadMem(GPRClass, GPRSize, SP, GPRSize), offsetof(FEXCore::Core::CPUState, ss));
// ss
auto NewSegmentSS = _LoadMem(GPRClass, GPRSize, SP, GPRSize);
_StoreContext(2, GPRClass, NewSegmentSS, offsetof(FEXCore::Core::CPUState, ss_idx));
UpdatePrefixFromSegment(NewSegmentSS, FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX);
SP = _Add(SP, Constant);
}
else {
@@ -562,26 +578,51 @@ void OpDispatchBuilder::PUSHSegmentOp(OpcodeArgs) {
_StoreContext(GPRSize, GPRClass, NewSP, RSPOffset);
OrderedNode *Src{};
switch (SegmentReg) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, es));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, cs));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ss));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ds));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs));
break;
default: break; // Do nothing
if (!CTX->Config.Is64BitMode()) {
switch (SegmentReg) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, es_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, cs_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ss_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ds_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs_idx));
break;
default: break; // Do nothing
}
}
else {
switch (SegmentReg) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, es_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, cs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ss_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, ds_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
Src = _LoadContext(SrcSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs_cached));
break;
default: break; // Do nothing
}
}
// Store our value to the new stack location
@@ -679,25 +720,27 @@ void OpDispatchBuilder::POPSegmentOp(OpcodeArgs) {
switch (SegmentReg) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, es));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, es_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, cs));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, cs_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ss));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ss_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ds));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ds_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, fs));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, fs_idx));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, gs));
_StoreContext(DstSize, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, gs_idx));
break;
default: break; // Do nothing
}
UpdatePrefixFromSegment(NewSegment, SegmentReg);
}
void OpDispatchBuilder::LEAVEOp(OpcodeArgs) {
@@ -1581,11 +1624,13 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
switch (Op->Dest.Data.GPR.GPR) {
case 0: // ES
case FEXCore::X86State::REG_R8: // ES
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, es));
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, es_idx));
UpdatePrefixFromSegment(Src, FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX);
break;
case 1: // DS
case FEXCore::X86State::REG_R11: // DS
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, ds));
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, ds_idx));
UpdatePrefixFromSegment(Src, FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX);
break;
case 2: // CS
case FEXCore::X86State::REG_R9: // CS
@@ -1599,12 +1644,14 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
break;
case 3: // SS
case FEXCore::X86State::REG_R10: // SS
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, ss));
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, ss_idx));
UpdatePrefixFromSegment(Src, FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX);
break;
case 6: // GS
case FEXCore::X86State::REG_R13: // GS
if (!CTX->Config.Is64BitMode) {
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, gs));
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, gs_idx));
UpdatePrefixFromSegment(Src, FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX);
} else {
LogMan::Msg::EFmt("We don't support modifying GS selector in 64bit mode!");
DecodeFailure = true;
@@ -1613,7 +1660,8 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
case 7: // FS
case FEXCore::X86State::REG_R12: // FS
if (!CTX->Config.Is64BitMode) {
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, fs));
_StoreContext(2, GPRClass, Src, offsetof(FEXCore::Core::CPUState, fs_idx));
UpdatePrefixFromSegment(Src, FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX);
} else {
LogMan::Msg::EFmt("We don't support modifying FS selector in 64bit mode!");
DecodeFailure = true;
@@ -1631,19 +1679,19 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
switch (Op->Src[0].Data.GPR.GPR) {
case 0: // ES
case FEXCore::X86State::REG_R8: // ES
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, es));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, es_idx));
break;
case 1: // DS
case FEXCore::X86State::REG_R11: // DS
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ds));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ds_idx));
break;
case 2: // CS
case FEXCore::X86State::REG_R9: // CS
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, cs));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, cs_idx));
break;
case 3: // SS
case FEXCore::X86State::REG_R10: // SS
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ss));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ss_idx));
break;
case 6: // GS
case FEXCore::X86State::REG_R13: // GS
@@ -1651,7 +1699,7 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
Segment = _Constant(0);
}
else {
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, gs));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, gs_idx));
}
break;
case 7: // FS
@@ -1660,7 +1708,7 @@ void OpDispatchBuilder::MOVSegOp(OpcodeArgs) {
Segment = _Constant(0);
}
else {
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, fs));
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, fs_idx));
}
break;
default:
@@ -3332,6 +3380,222 @@ void OpDispatchBuilder::PopcountOp(OpcodeArgs) {
GenerateFlags_POPCOUNT(Op, Src);
}
void OpDispatchBuilder::DAAOp(OpcodeArgs) {
CalculateDeferredFlags();
auto CF = GetRFLAG(FEXCore::X86State::RFLAG_CF_LOC);
auto AF = GetRFLAG(FEXCore::X86State::RFLAG_AF_LOC);
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
auto Cond = _Or(AF, _Select(FEXCore::IR::COND_UGT, _And(AL, _Constant(0xF)), _Constant(9), _Constant(1), _Constant(0)));
auto FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
auto TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
auto EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
auto NewAL = _Add(AL, _Constant(0x6));
_StoreContext(1, GPRClass, NewAL, GPROffset(X86State::REG_RAX));
CalculateDeferredFlags();
auto NewCF = GetRFLAG(FEXCore::X86State::RFLAG_CF_LOC);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Or(CF, NewCF));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
Cond = _Or(CF, _Select(FEXCore::IR::COND_UGT, AL, _Constant(0x99), _Constant(1), _Constant(0)));
FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto NewAL = _Add(AL, _Constant(0x60));
_StoreContext(1, GPRClass, NewAL, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
// Update Flags
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(_Select(FEXCore::IR::COND_UGE, _And(AL, _Constant(0x80)), _Constant(0), _Constant(1), _Constant(0)));
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Select(FEXCore::IR::COND_EQ, _And(AL, _Constant(0xFF)), _Constant(0), _Constant(1), _Constant(0)));
auto EightBitMask = _Constant(0xFF);
auto PopCountOp = _Popcount(_And(AL, EightBitMask));
auto XorOp = _Xor(PopCountOp, _Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(XorOp);
}
void OpDispatchBuilder::DASOp(OpcodeArgs) {
CalculateDeferredFlags();
auto CF = GetRFLAG(FEXCore::X86State::RFLAG_CF_LOC);
auto AF = GetRFLAG(FEXCore::X86State::RFLAG_AF_LOC);
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
auto Cond = _Or(AF, _Select(FEXCore::IR::COND_UGT, _And(AL, _Constant(0xf)), _Constant(9), _Constant(1), _Constant(0)));
auto FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
auto TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
auto EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
auto NewAL = _Sub(AL, _Constant(0x6));
_StoreContext(1, GPRClass, NewAL, GPROffset(X86State::REG_RAX));
CalculateDeferredFlags();
auto NewCF = GetRFLAG(FEXCore::X86State::RFLAG_CF_LOC);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Or(CF, NewCF));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
Cond = _Or(CF, _Select(FEXCore::IR::COND_UGT, AL, _Constant(0x99), _Constant(1), _Constant(0)));
FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto NewAL = _Sub(AL, _Constant(0x60));
_StoreContext(1, GPRClass, NewAL, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
// Update Flags
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(_Select(FEXCore::IR::COND_UGE, _And(AL, _Constant(0x80)), _Constant(0), _Constant(1), _Constant(0)));
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Select(FEXCore::IR::COND_EQ, _And(AL, _Constant(0xFF)), _Constant(0), _Constant(1), _Constant(0)));
auto EightBitMask = _Constant(0xFF);
auto PopCountOp = _Popcount(_And(AL, EightBitMask));
auto XorOp = _Xor(PopCountOp, _Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(XorOp);
}
void OpDispatchBuilder::AAAOp(OpcodeArgs) {
auto AF = GetRFLAG(FEXCore::X86State::RFLAG_AF_LOC);
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto AX = _LoadContext(2, GPRClass, GPROffset(X86State::REG_RAX));
auto Cond = _Or(AF, _Select(FEXCore::IR::COND_UGT, _And(AL, _Constant(0xF)), _Constant(9), _Constant(1), _Constant(0)));
auto FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
auto TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
auto EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
auto NewAX = _And(AX, _Constant(0xFF0F));
_StoreContext(2, GPRClass, NewAX, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
auto NewAX = _Add(AX, _Constant(0x106));
auto Result = _And(NewAX, _Constant(0xFF0F));
_StoreContext(2, GPRClass, Result, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
}
void OpDispatchBuilder::AASOp(OpcodeArgs) {
auto AF = GetRFLAG(FEXCore::X86State::RFLAG_AF_LOC);
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto AX = _LoadContext(2, GPRClass, GPROffset(X86State::REG_RAX));
auto Cond = _Or(AF, _Select(FEXCore::IR::COND_UGT, _And(AL, _Constant(0xF)), _Constant(9), _Constant(1), _Constant(0)));
auto FalseBlock = CreateNewCodeBlockAfter(GetCurrentBlock());
auto TrueBlock = CreateNewCodeBlockAfter(FalseBlock);
auto EndBlock = CreateNewCodeBlockAfter(TrueBlock);
_CondJump(Cond, TrueBlock, FalseBlock);
SetCurrentCodeBlock(FalseBlock);
{
auto NewAX = _And(AX, _Constant(0xFF0F));
_StoreContext(2, GPRClass, NewAX, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(0));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(0));
_Jump(EndBlock);
}
SetCurrentCodeBlock(TrueBlock);
{
auto NewAX = _Sub(AX, _Constant(6));
NewAX = _Sub(NewAX, _Constant(0x100));
auto Result = _And(NewAX, _Constant(0xFF0F));
_StoreContext(2, GPRClass, Result, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(_Constant(1));
_Jump(EndBlock);
}
SetCurrentCodeBlock(EndBlock);
}
void OpDispatchBuilder::AAMOp(OpcodeArgs) {
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto Imm8 = _Constant(Op->Src[0].Data.Literal.Value & 0xFF);
auto UDivOp = _UDiv(AL, Imm8);
auto URemOp = _URem(AL, Imm8);
auto AH = _Lshl(UDivOp, _Constant(8));
auto AX = _Add(AH, URemOp);
_StoreContext(2, GPRClass, AX, GPROffset(X86State::REG_RAX));
// Update Flags
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(_Select(FEXCore::IR::COND_UGE, _And(AL, _Constant(0x80)), _Constant(0), _Constant(1), _Constant(0)));
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Select(FEXCore::IR::COND_EQ, _And(AL, _Constant(0xFF)), _Constant(0), _Constant(1), _Constant(0)));
auto EightBitMask = _Constant(0xFF);
auto PopCountOp = _Popcount(_And(AL, EightBitMask));
auto XorOp = _Xor(PopCountOp, _Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(XorOp);
}
void OpDispatchBuilder::AADOp(OpcodeArgs) {
auto AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
auto AH = _Lshr(_LoadContext(2, GPRClass, GPROffset(X86State::REG_RAX)), _Constant(8));
auto Imm8 = _Constant(Op->Src[0].Data.Literal.Value & 0xFF);
auto NewAL = _Add(AL, _Mul(AH, Imm8));
auto Result = _And(NewAL, _Constant(0xFF));
_StoreContext(2, GPRClass, Result, GPROffset(X86State::REG_RAX));
// Update Flags
AL = _LoadContext(1, GPRClass, GPROffset(X86State::REG_RAX));
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(_Select(FEXCore::IR::COND_UGE, _And(AL, _Constant(0x80)), _Constant(0), _Constant(1), _Constant(0)));
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Select(FEXCore::IR::COND_EQ, _And(AL, _Constant(0xFF)), _Constant(0), _Constant(1), _Constant(0)));
auto EightBitMask = _Constant(0xFF);
auto PopCountOp = _Popcount(_And(AL, EightBitMask));
auto XorOp = _Xor(PopCountOp, _Constant(1));
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(XorOp);
}
void OpDispatchBuilder::XLATOp(OpcodeArgs) {
const uint32_t RAXOffset = GPROffset(X86State::REG_RAX);
const uint32_t RBXOffset = GPROffset(X86State::REG_RBX);
@@ -3350,13 +3614,15 @@ void OpDispatchBuilder::XLATOp(OpcodeArgs) {
template<OpDispatchBuilder::Segment Seg>
void OpDispatchBuilder::ReadSegmentReg(OpcodeArgs) {
// 64-bit only
// Doesn't hit the segment register optimization
auto Size = GetSrcSize(Op);
OrderedNode *Src{};
if constexpr (Seg == Segment::FS) {
Src = _LoadContext(Size, GPRClass, offsetof(FEXCore::Core::CPUState, fs));
Src = _LoadContext(Size, GPRClass, offsetof(FEXCore::Core::CPUState, fs_cached));
}
else {
Src = _LoadContext(Size, GPRClass, offsetof(FEXCore::Core::CPUState, gs));
Src = _LoadContext(Size, GPRClass, offsetof(FEXCore::Core::CPUState, gs_cached));
}
StoreResult(GPRClass, Op, Src, -1);
@@ -3369,10 +3635,10 @@ void OpDispatchBuilder::WriteSegmentReg(OpcodeArgs) {
auto Size = GetDstSize(Op);
OrderedNode *Src = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags, -1);
if constexpr (Seg == Segment::FS) {
_StoreContext(Size, GPRClass, Src, offsetof(FEXCore::Core::CPUState, fs));
_StoreContext(Size, GPRClass, Src, offsetof(FEXCore::Core::CPUState, fs_cached));
}
else {
_StoreContext(Size, GPRClass, Src, offsetof(FEXCore::Core::CPUState, gs));
_StoreContext(Size, GPRClass, Src, offsetof(FEXCore::Core::CPUState, gs_cached));
}
}
@@ -4570,10 +4836,10 @@ OrderedNode *OpDispatchBuilder::AppendSegmentOffset(OrderedNode *Value, uint32_t
if (CTX->Config.Is64BitMode) {
if (Flags & FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX) {
Value = _Add(Value, _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs)));
Value = _Add(Value, _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs_cached)));
}
else if (Flags & FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX) {
Value = _Add(Value, _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs)));
Value = _Add(Value, _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs_cached)));
}
// If there was any other segment in 64bit then it is ignored
}
@@ -4585,38 +4851,65 @@ OrderedNode *OpDispatchBuilder::AppendSegmentOffset(OrderedNode *Value, uint32_t
// Or the argument only uses a specific prefix (with override set)
Prefix = DefaultPrefix;
}
// With the segment register optimization we store the GDT bases directly in the segment register to remove indexed loads
switch (Prefix) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, es));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, es_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, cs));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, cs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ss));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, ss_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, ds));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, ds_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, fs));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, fs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
Segment = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, gs));
Segment = _LoadContext(GPRSize, GPRClass, offsetof(FEXCore::Core::CPUState, gs_cached));
break;
default: break; // Do nothing
}
if (Segment) {
Segment = _Lshr(Segment, _Constant(3));
auto data = _LoadContextIndexed(Segment, 4, offsetof(FEXCore::Core::CPUState, gdt[0]), 4, GPRClass);
Value = _Add(Value, data);
Value = _Add(Value, Segment);
}
}
return Value;
}
void OpDispatchBuilder::UpdatePrefixFromSegment(OrderedNode *Segment, uint32_t SegmentReg) {
// Use BFE to extract the selector index in bits [15,3] of the segment register.
// In some cases the upper 16-bits of the 32-bit GPR contain garbage to ignore.
Segment = _Bfe(4, 16 - 3, 3, Segment);
auto NewSegment = _LoadContextIndexed(Segment, 4, offsetof(FEXCore::Core::CPUState, gdt[0]), 4, GPRClass);
switch (SegmentReg) {
case FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, es_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, cs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ss_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, ds_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, fs_cached));
break;
case FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX:
_StoreContext(4, GPRClass, NewSegment, offsetof(FEXCore::Core::CPUState, gs_cached));
break;
default: break; // Do nothing
}
}
OrderedNode *OpDispatchBuilder::LoadSource_WithOpSize(FEXCore::IR::RegisterClassType Class, FEXCore::X86Tables::DecodedOp const& Op, FEXCore::X86Tables::DecodedOperand const& Operand, uint8_t OpSize, uint32_t Flags, int8_t Align, bool LoadData, bool ForceLoad, MemoryAccessType AccessType) {
LOGMAN_THROW_A_FMT(Operand.IsGPR() ||
Operand.IsLiteral() ||
@@ -5495,12 +5788,18 @@ void InstallOpcodeHandlers(Context::OperatingMode Mode) {
{0x17, 1, &OpDispatchBuilder::POPSegmentOp<FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX>},
{0x1E, 1, &OpDispatchBuilder::PUSHSegmentOp<FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX>},
{0x1F, 1, &OpDispatchBuilder::POPSegmentOp<FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX>},
{0x27, 1, &OpDispatchBuilder::DAAOp},
{0x2F, 1, &OpDispatchBuilder::DASOp},
{0x37, 1, &OpDispatchBuilder::AAAOp},
{0x3F, 1, &OpDispatchBuilder::AASOp},
{0x40, 8, &OpDispatchBuilder::INCOp},
{0x48, 8, &OpDispatchBuilder::DECOp},
{0x60, 1, &OpDispatchBuilder::PUSHAOp},
{0x61, 1, &OpDispatchBuilder::POPAOp},
{0xCE, 1, &OpDispatchBuilder::INTOp},
{0xD4, 1, &OpDispatchBuilder::AAMOp},
{0xD5, 1, &OpDispatchBuilder::AADOp},
};
constexpr std::tuple<uint8_t, uint8_t, X86Tables::OpDispatchPtr> BaseOpTable_64[] = {
@@ -278,6 +278,12 @@ public:
void NOTOp(OpcodeArgs);
void XADDOp(OpcodeArgs);
void PopcountOp(OpcodeArgs);
void DAAOp(OpcodeArgs);
void DASOp(OpcodeArgs);
void AAAOp(OpcodeArgs);
void AASOp(OpcodeArgs);
void AAMOp(OpcodeArgs);
void AADOp(OpcodeArgs);
void XLATOp(OpcodeArgs);
template<bool Reseed>
void RDRANDOp(OpcodeArgs);
@@ -646,6 +652,7 @@ private:
OrderedNode *Current_HeaderNode{};
OrderedNode *AppendSegmentOffset(OrderedNode *Value, uint32_t Flags, uint32_t DefaultPrefix = 0, bool Override = false);
void UpdatePrefixFromSegment(OrderedNode *Segment, uint32_t SegmentReg);
enum class MemoryAccessType {
// Choose TSO or Non-TSO depending on access type
@@ -769,7 +769,11 @@ void OpDispatchBuilder::CalculcateFlags_ShiftLeftImmediate(uint8_t SrcSize, Orde
// CF
{
// Extract the last bit shifted in to CF
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Bfe(1, SrcSize * 8 - Shift, Src1));
auto OpSize = SrcSize * 8;
if (OpSize < Shift) {
Shift &= (OpSize - 1);
}
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Bfe(1, OpSize - Shift, Src1));
}
// PF
@@ -934,6 +938,7 @@ void OpDispatchBuilder::CalculcateFlags_RotateRight(uint8_t SrcSize, OrderedNode
auto OldOF = GetRFLAG(FEXCore::X86State::RFLAG_OF_LOC);
// OF is set to the XOR of the new CF bit and the most significant bit of the result
// OF is architecturally only defined for 1-bit rotate, which is why this only happens when the shift is one.
auto NewOF = _Xor(_Bfe(1, OpSize - 2, Res), NewCF);
// If shift == 0, don't update flags
@@ -963,7 +968,9 @@ void OpDispatchBuilder::CalculcateFlags_RotateLeft(uint8_t SrcSize, OrderedNode
// OF
{
auto OldOF = GetRFLAG(FEXCore::X86State::RFLAG_OF_LOC);
// OF is set to the XOR of the new CF bit and the most significant bit of the result
// OF is the LSB and MSB XOR'd together.
// OF is set to the XOR of the new CF bit and the most significant bit of the result.
// OF is architecturally only defined for 1-bit rotate, which is why this only happens when the shift is one.
auto NewOF = _Xor(_Bfe(1, OpSize - 1, Res), NewCF);
auto OF = _Select(FEXCore::IR::COND_EQ, Src2, _Constant(0), OldOF, NewOF);
@@ -977,8 +984,7 @@ void OpDispatchBuilder::CalculcateFlags_RotateRightImmediate(uint8_t SrcSize, Or
if (Shift == 0) return;
auto OpSize = SrcSize * 8;
auto NewCF = _Bfe(1, OpSize - Shift, Src1);
auto NewCF = _Bfe(1, OpSize - 1, Res);
// CF
{
@@ -989,8 +995,10 @@ void OpDispatchBuilder::CalculcateFlags_RotateRightImmediate(uint8_t SrcSize, Or
// OF
{
if (Shift == 1) {
// OF is set to the XOR of the new CF bit and the most significant bit of the result
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(_Xor(_Bfe(1, OpSize - 1, Res), NewCF));
// OF is the top two MSBs XOR'd together
// OF is architecturally only defined for 1-bit rotate, which is why this only happens when the shift is one.
auto NewOF = _Xor(_Bfe(1, OpSize - 2, Res), NewCF);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(NewOF);
}
}
}
@@ -1000,17 +1008,22 @@ void OpDispatchBuilder::CalculcateFlags_RotateLeftImmediate(uint8_t SrcSize, Ord
auto OpSize = SrcSize * 8;
auto NewCF = _Bfe(1, 0, Res);
// CF
{
// Extract the last bit shifted in to CF
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Bfe(1, Shift, Src1));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(NewCF);
}
// OF
{
if (Shift == 1) {
// OF is the top two MSBs XOR'd together
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(_Xor(_Bfe(1, OpSize - 1, Src1), _Bfe(1, OpSize - 2, Src1)));
// OF is the LSB and MSB XOR'd together.
// OF is set to the XOR of the new CF bit and the most significant bit of the result.
// OF is architecturally only defined for 1-bit rotate, which is why this only happens when the shift is one.
auto NewOF = _Xor(_Bfe(1, OpSize - 1, Res), NewCF);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(NewOF);
}
}
}
@@ -74,13 +74,12 @@ void OpDispatchBuilder::MOVLPOp(OpcodeArgs) {
// xmm, xmm is movhlps special case
if (Op->Src[0].IsGPR()) {
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, 8, 16);
Src = _VExtractElement(16, 8, Src, 1);
auto Result = _VInsScalarElement(16, 8, 0, Dest, Src);
auto Result = _VInsElement(16, 8, 0, 1, Dest, Src);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Result, 16, 16);
}
else {
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, 8, 16);
auto Result = _VInsScalarElement(16, 8, 0, Dest, Src);
auto Result = _VInsElement(16, 8, 0, 0, Dest, Src);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Result, 8, 16);
}
}
@@ -112,7 +111,7 @@ void OpDispatchBuilder::MOVSSOp(OpcodeArgs) {
// MOVSS xmm1, xmm2
OrderedNode *Dest = LoadSource_WithOpSize(FPRClass, Op, Op->Dest, 16, Op->Flags, -1);
OrderedNode *Src = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], 4, Op->Flags, -1);
auto Result = _VInsScalarElement(16, 4, 0, Dest, Src);
auto Result = _VInsElement(16, 4, 0, 0, Dest, Src);
StoreResult(FPRClass, Op, Result, -1);
}
else if (Op->Dest.IsGPR()) {
@@ -133,7 +132,7 @@ void OpDispatchBuilder::MOVSDOp(OpcodeArgs) {
// xmm1[63:0] <- xmm2[63:0]
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
auto Result = _VInsScalarElement(16, 8, 0, Dest, Src);
auto Result = _VInsElement(16, 8, 0, 0, Dest, Src);
StoreResult(FPRClass, Op, Result, -1);
}
else if (Op->Dest.IsGPR()) {
@@ -335,7 +334,7 @@ void OpDispatchBuilder::VectorScalarALUOp(OpcodeArgs) {
if (Size != ElementSize) {
// Insert the lower bits
Result = _VInsScalarElement(Size, ElementSize, 0, Dest, Result);
Result = _VInsElement(Size, ElementSize, 0, 0, Dest, Result);
}
StoreResult(FPRClass, Op, Result, -1);
@@ -381,7 +380,7 @@ void OpDispatchBuilder::VectorUnaryOp(OpcodeArgs) {
if constexpr (Scalar) {
// Insert the lower bits
auto Result = _VInsScalarElement(GetSrcSize(Op), ElementSize, 0, Dest, ALUOp);
auto Result = _VInsElement(GetSrcSize(Op), ElementSize, 0, 0, Dest, ALUOp);
StoreResult(FPRClass, Op, Result, -1);
}
else {
@@ -978,7 +977,8 @@ void OpDispatchBuilder::PAVGOp<2>(OpcodeArgs);
void OpDispatchBuilder::MOVDDUPOp(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Res = _SplatVector2(Src);
OrderedNode *Res = _VDupElement(16, GetSrcSize(Op), Src, 0);
StoreResult(FPRClass, Op, Res, -1);
}
@@ -992,7 +992,7 @@ void OpDispatchBuilder::CVTGPR_To_FPR(OpcodeArgs) {
OrderedNode *Dest = LoadSource_WithOpSize(FPRClass, Op, Op->Dest, 16, Op->Flags, -1);
Src = _VInsScalarElement(16, DstElementSize, 0, Dest, Src);
Src = _VInsElement(16, DstElementSize, 0, 0, Dest, Src);
StoreResult(FPRClass, Op, Src, -1);
}
@@ -1091,7 +1091,7 @@ void OpDispatchBuilder::Scalar_CVT_Float_To_Float(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
Src = _Float_FToF(DstElementSize, SrcElementSize, Src);
Src = _VInsScalarElement(16, DstElementSize, 0, Dest, Src);
Src = _VInsElement(16, DstElementSize, 0, 0, Dest, Src);
StoreResult(FPRClass, Op, Src, -1);
}
@@ -1281,7 +1281,7 @@ void OpDispatchBuilder::VFCMPOp(OpcodeArgs) {
if constexpr (Scalar) {
// Insert the lower bits
Result = _VInsScalarElement(GetDstSize(Op), ElementSize, 0, Dest, Result);
Result = _VInsElement(GetDstSize(Op), ElementSize, 0, 0, Dest, Result);
}
StoreResult(FPRClass, Op, Result, -1);
@@ -1563,6 +1563,9 @@ void OpDispatchBuilder::PACKSSOp<4>(OpcodeArgs);
template<size_t ElementSize, bool Signed>
void OpDispatchBuilder::PMULLOp(OpcodeArgs) {
static_assert(ElementSize == sizeof(uint32_t),
"Currently only handles 32-bit -> 64-bit");
auto Size = GetSrcSize(Op);
OrderedNode *Src1 = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
@@ -1579,17 +1582,8 @@ void OpDispatchBuilder::PMULLOp(OpcodeArgs) {
}
}
else {
OrderedNode* Srcs1[2]{};
OrderedNode* Srcs2[2]{};
Srcs1[0] = _VExtr(Size, ElementSize, Src1, Src1, 0);
Srcs1[1] = _VExtr(Size, ElementSize, Src1, Src1, 2);
Srcs2[0] = _VExtr(Size, ElementSize, Src2, Src2, 0);
Srcs2[1] = _VExtr(Size, ElementSize, Src2, Src2, 2);
Src1 = _VInsElement(Size, ElementSize, 1, 0, Srcs1[0], Srcs1[1]);
Src2 = _VInsElement(Size, ElementSize, 1, 0, Srcs2[0], Srcs2[1]);
Src1 = _VInsElement(Size, ElementSize, 1, 2, Src1, Src1);
Src2 = _VInsElement(Size, ElementSize, 1, 2, Src2, Src2);
if constexpr (Signed) {
Res = _VSMull(Size, ElementSize, Src1, Src2);
@@ -1714,8 +1708,8 @@ void OpDispatchBuilder::PFNACCOp(OpcodeArgs) {
OrderedNode *ResSubSrc{};
OrderedNode *ResSubDest{};
auto UpperSubDest = _VExtractElement(Size, 4, Dest, 1);
auto UpperSubSrc = _VExtractElement(Size, 4, Src, 1);
auto UpperSubDest = _VDupElement(Size, 4, Dest, 1);
auto UpperSubSrc = _VDupElement(Size, 4, Src, 1);
ResSubDest = _VFSub(4, 4, Dest, UpperSubDest);
ResSubSrc = _VFSub(4, 4, Src, UpperSubSrc);
@@ -1733,7 +1727,7 @@ void OpDispatchBuilder::PFPNACCOp(OpcodeArgs) {
OrderedNode *ResAdd{};
OrderedNode *ResSub{};
auto UpperSubDest = _VExtractElement(Size, 4, Dest, 1);
auto UpperSubDest = _VDupElement(Size, 4, Dest, 1);
ResSub = _VFSub(4, 4, Dest, UpperSubDest);
ResAdd = _VFAddP(Size, 4, Src, Src);
@@ -1866,8 +1860,6 @@ void OpDispatchBuilder::PMADDWD(OpcodeArgs) {
if (Size == 8) {
Size <<= 1;
Src1 = _VBitcast(Size, 2, Src1);
Src2 = _VBitcast(Size, 2, Src2);
}
auto Src1_L = _VSXTL(Size, 2, Src1); // [15:0 ], [31:16], [32:47 ], [63:48 ]
@@ -1954,9 +1946,6 @@ void OpDispatchBuilder::PMULHW(OpcodeArgs) {
OrderedNode *Res{};
if (Size == 8) {
Dest = _VBitcast(Size * 2, 2, Dest);
Src = _VBitcast(Size * 2, 2, Src);
// Implementation is more efficient for 8byte registers
if (Signed)
Res = _VSMull(Size * 2, 2, Dest, Src);
@@ -2333,7 +2322,7 @@ void OpDispatchBuilder::VectorRound(OpcodeArgs) {
if constexpr (Scalar) {
// Insert the lower bits
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
auto Result = _VInsScalarElement(GetDstSize(Op), ElementSize, 0, Dest, Src);
auto Result = _VInsElement(GetDstSize(Op), ElementSize, 0, 0, Dest, Src);
StoreResult(FPRClass, Op, Result, -1);
}
else {
@@ -782,6 +782,12 @@ void OpDispatchBuilder::X87UnaryOp(OpcodeArgs) {
// Overwrite the op
result.first->Header.Op = IROp;
if constexpr (IROp == IR::OP_F80SIN ||
IROp == IR::OP_F80COS) {
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
}
// Write to ST[TOP]
_StoreContextIndexed(result, top, 16, MMBaseOffset(), 16, FPRClass);
}
@@ -809,7 +815,8 @@ void OpDispatchBuilder::X87BinaryOp(OpcodeArgs) {
// Overwrite the op
result.first->Header.Op = IROp;
if constexpr (IROp == IR::OP_F80FPREM) {
if constexpr (IROp == IR::OP_F80FPREM ||
IROp == IR::OP_F80FPREM1) {
//TODO: Set C0 to Q2, C3 to Q1, C1 to Q0
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
}
@@ -854,6 +861,9 @@ void OpDispatchBuilder::X87SinCos(OpcodeArgs) {
auto sin = _F80SIN(a);
auto cos = _F80COS(a);
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
// Write to ST[TOP]
_StoreContextIndexed(sin, orig_top, 16, MMBaseOffset(), 16, FPRClass);
_StoreContextIndexed(cos, top, 16, MMBaseOffset(), 16, FPRClass);
@@ -900,6 +910,9 @@ void OpDispatchBuilder::X87TAN(OpcodeArgs) {
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, 1, data, high);
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
// Write to ST[TOP]
_StoreContextIndexed(result, orig_top, 16, MMBaseOffset(), 16, FPRClass);
_StoreContextIndexed(data, top, 16, MMBaseOffset(), 16, FPRClass);
@@ -1185,7 +1198,7 @@ void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
// upper 16 bits [79:64]
_StoreMem(FPRClass, 8, ST0Location, data, 1);
ST0Location = _Add(ST0Location, _Constant(8));
auto topBytes = _VExtractElement(16, 2, data, 4);
auto topBytes = _VDupElement(16, 2, data, 4);
_StoreMem(FPRClass, 2, ST0Location, topBytes, 1);
// reset to default
@@ -778,6 +778,12 @@ void OpDispatchBuilder::X87UnaryOpF64(OpcodeArgs) {
// Overwrite the op
result.first->Header.Op = IROp;
if constexpr (IROp == IR::OP_F64SIN ||
IROp == IR::OP_F64COS) {
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
}
// Write to ST[TOP]
_StoreContextIndexed(result, top, 8, MMBaseOffset(), 16, FPRClass);
}
@@ -804,7 +810,8 @@ void OpDispatchBuilder::X87BinaryOpF64(OpcodeArgs) {
// Overwrite the op
result.first->Header.Op = IROp;
if constexpr (IROp == IR::OP_F64FPREM) {
if constexpr (IROp == IR::OP_F80FPREM ||
IROp == IR::OP_F80FPREM1) {
//TODO: Set C0 to Q2, C3 to Q1, C1 to Q0
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
}
@@ -831,6 +838,9 @@ void OpDispatchBuilder::X87SinCosF64(OpcodeArgs) {
auto sin = _F64SIN(a);
auto cos = _F64COS(a);
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
// Write to ST[TOP]
_StoreContextIndexed(sin, orig_top, 8, MMBaseOffset(), 16, FPRClass);
_StoreContextIndexed(cos, top, 8, MMBaseOffset(), 16, FPRClass);
@@ -871,6 +881,9 @@ void OpDispatchBuilder::X87TANF64(OpcodeArgs) {
auto one = _VCastFromGPR(8, 8, _Constant(0x3FF0000000000000));
// TODO: ACCURACY: should check source is in range –2^63 to +2^63
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(_Constant(0));
// Write to ST[TOP]
_StoreContextIndexed(result, orig_top, 8, MMBaseOffset(), 16, FPRClass);
_StoreContextIndexed(one, top, 8, MMBaseOffset(), 16, FPRClass);
@@ -996,7 +1009,7 @@ void OpDispatchBuilder::X87FNSAVEF64(OpcodeArgs) {
// upper 16 bits [79:64]
_StoreMem(FPRClass, 8, ST0Location, data, 1);
ST0Location = _Add(ST0Location, _Constant(8));
auto topBytes = _VExtractElement(16, 2, data, 4);
auto topBytes = _VDupElement(16, 2, data, 4);
_StoreMem(FPRClass, 2, ST0Location, topBytes, 1);
// reset to default
@@ -266,10 +266,10 @@ void InitializeBaseTables(Context::OperatingMode Mode) {
{0x17, 1, X86InstInfo{"POP SS", TYPE_INST, GenFlagsSizes(SIZE_16BIT, SIZE_DEF) | FLAGS_DEBUG_MEM_ACCESS, 0, nullptr}},
{0x1E, 1, X86InstInfo{"PUSH DS", TYPE_INST, GenFlagsSrcSize(SIZE_16BIT) | FLAGS_DEBUG_MEM_ACCESS, 0, nullptr}},
{0x1F, 1, X86InstInfo{"POP DS", TYPE_INST, GenFlagsSizes(SIZE_16BIT, SIZE_DEF) | FLAGS_DEBUG_MEM_ACCESS, 0, nullptr}},
{0x27, 1, X86InstInfo{"DAA", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x2F, 1, X86InstInfo{"DAS", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x37, 1, X86InstInfo{"AAA", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x3F, 1, X86InstInfo{"AAS", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x27, 1, X86InstInfo{"DAA", TYPE_INST, GenFlagsDstSize(SIZE_8BIT) | FLAGS_SF_DST_RAX, 0, nullptr}},
{0x2F, 1, X86InstInfo{"DAS", TYPE_INST, GenFlagsDstSize(SIZE_8BIT) | FLAGS_SF_DST_RAX, 0, nullptr}},
{0x37, 1, X86InstInfo{"AAA", TYPE_INST, GenFlagsDstSize(SIZE_16BIT) | FLAGS_SF_DST_RAX, 0, nullptr}},
{0x3F, 1, X86InstInfo{"AAS", TYPE_INST, GenFlagsDstSize(SIZE_16BIT) | FLAGS_SF_DST_RAX, 0, nullptr}},
{0x40, 8, X86InstInfo{"INC", TYPE_INST, FLAGS_SF_REX_IN_BYTE, 0, nullptr}},
{0x48, 8, X86InstInfo{"DEC", TYPE_INST, FLAGS_SF_REX_IN_BYTE, 0, nullptr}},
@@ -283,8 +283,8 @@ void InitializeBaseTables(Context::OperatingMode Mode) {
{0xA1, 1, X86InstInfo{"MOV", TYPE_INST, FLAGS_SF_DST_RAX | FLAGS_MEM_OFFSET, 4, nullptr}},
{0xA3, 1, X86InstInfo{"MOV", TYPE_INST, FLAGS_SF_SRC_RAX | FLAGS_MEM_OFFSET, 4, nullptr}},
{0xCE, 1, X86InstInfo{"INTO", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0xD4, 1, X86InstInfo{"AAM", TYPE_INST, FLAGS_NONE, 1, nullptr}},
{0xD5, 1, X86InstInfo{"AAD", TYPE_INST, FLAGS_NONE, 1, nullptr}},
{0xD4, 1, X86InstInfo{"AAM", TYPE_INST, GenFlagsSameSize(SIZE_8BIT) | FLAGS_SF_DST_RAX, 1, nullptr}},
{0xD5, 1, X86InstInfo{"AAD", TYPE_INST, GenFlagsSameSize(SIZE_8BIT) | FLAGS_SF_DST_RAX, 1, nullptr}},
{0xEA, 1, X86InstInfo{"JMPF", TYPE_INST, FLAGS_NONE, 0, nullptr}},
};
+19 -6
View File
@@ -28,7 +28,9 @@ $end_info$
#include <string>
#include <utility>
#ifdef ENABLE_JEMALLOC
#include "jemalloc/jemalloc.h"
#endif
struct LoadlibArgs {
const char *Name;
@@ -137,7 +139,7 @@ namespace FEXCore {
// sha256(fex:allocate_host_trampoline_for_guest_function)
{ 0x9b, 0xb2, 0xf4, 0xb4, 0x83, 0x7d, 0x28, 0x93, 0x40, 0xcb, 0xf4, 0x7a, 0x0b, 0x47, 0x85, 0x87, 0xf9, 0xbc, 0xb5, 0x27, 0xca, 0xa6, 0x93, 0xa5, 0xc0, 0x73, 0x27, 0x24, 0xae, 0xc8, 0xb8, 0x5a },
&AllocateHostTrampolineForGuestFunction
}
},
};
// Can't be a string_view. We need to keep a copy of the library name in-case string_view pointer goes away.
@@ -241,12 +243,17 @@ namespace FEXCore {
* and host heap pointers.
*/
static void IsHostHeapAllocation(void* ArgsRV) {
#ifdef ENABLE_JEMALLOC
struct ArgsRV_t {
void* ptr;
bool rv;
} *args = reinterpret_cast<ArgsRV_t*>(ArgsRV);
args->rv = je_is_known_allocation(args->ptr);
#else
// Thunks usage without jemalloc isn't supported
ERROR_AND_DIE_FMT("Unsupported: Thunks querying for host heap allocation information");
#endif
}
static void LoadLib(void *ArgsV) {
@@ -313,7 +320,7 @@ namespace FEXCore {
}
}
ThunkedFunction* LookupThunk(const IR::SHA256Sum &sha256) {
ThunkedFunction* LookupThunk(const IR::SHA256Sum &sha256) override {
std::shared_lock lk(ThunksMutex);
@@ -326,13 +333,19 @@ namespace FEXCore {
}
}
void RegisterTLSState(FEXCore::Core::InternalThreadState *Thread) {
void RegisterTLSState(FEXCore::Core::InternalThreadState *Thread) override {
::Thread = Thread;
}
void AppendThunkDefinitions(std::vector<FEXCore::IR::ThunkDefinition> const& Definitions) override {
for (auto & Definition : Definitions) {
Thunks.emplace(Definition.Sum, Definition.ThunkFunction);
}
}
};
ThunkHandler* ThunkHandler::Create() {
return new ThunkHandler_impl();
return new ThunkHandler_impl();
}
/**
@@ -358,7 +371,7 @@ namespace FEXCore {
* is to allocate the trampoline for a given GuestTarget/GuestUnpacker on the guest-side,
* and provide the HostPacker host-side.
*/
__attribute__((visibility("default")))
FEX_DEFAULT_VISIBILITY
HostToGuestTrampolinePtr* MakeHostTrampolineForGuestFunction(void* HostPacker, uintptr_t GuestTarget, uintptr_t GuestUnpacker) {
LOGMAN_THROW_AA_FMT(GuestTarget, "Tried to create host-trampoline to null pointer guest function");
@@ -416,7 +429,7 @@ namespace FEXCore {
return HostTrampoline;
}
__attribute__((visibility("default")))
FEX_DEFAULT_VISIBILITY
void FinalizeHostTrampolineForGuestFunction(HostToGuestTrampolinePtr* TrampolineAddress, void* HostPacker) {
auto& Trampoline = GetInstanceInfo(TrampolineAddress);
+6
View File
@@ -6,6 +6,10 @@ $end_info$
#pragma once
#include <FEXCore/IR/IR.h>
#include <vector>
namespace FEXCore::Context {
struct Context;
}
@@ -28,5 +32,7 @@ namespace FEXCore {
virtual ~ThunkHandler() { }
static ThunkHandler* Create();
virtual void AppendThunkDefinitions(std::vector<FEXCore::IR::ThunkDefinition> const& Definitions) = 0;
};
};
+31 -59
View File
@@ -302,10 +302,6 @@
}
},
"Moves": {
"GPR = Mov GPR:$Value": {
"DestSize": "GetOpSize(_Value)"
},
"GPR = ExtractElementPair GPRPair:$Pair, u8:$Element": {
"Desc": ["Extracts a register for the register pair"],
"DestSize": "GetOpSize(_Pair) >> 1"
@@ -359,22 +355,22 @@
"DestSize": "ByteSize",
"EmitValidation": [
"($Class == GPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8)) || $Class == FPRClass",
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16)) || $Class == GPRClass"
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16 || #ByteSize == 32)) || $Class == GPRClass"
]
},
"StoreContext u8:#ByteSize, RegisterClass:$Class, SSA:$Value, u32:$Offset": {
"Desc": ["Stores a value to the context with offset",
"Ctx[Offset] = Value",
"Zero Extends if value's type is too small",
"Truncates if value's type is too large"
],
"Desc": ["Stores a value to the context with offset",
"Ctx[Offset] = Value",
"Zero Extends if value's type is too small",
"Truncates if value's type is too large"
],
"HasSideEffects": true,
"DestSize": "ByteSize",
"EmitValidation": [
"WalkFindRegClass($Value) == $Class",
"($Class == GPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8)) || $Class == FPRClass",
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16)) || $Class == GPRClass"
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16 || #ByteSize == 32)) || $Class == GPRClass"
]
},
@@ -385,7 +381,7 @@
"DestSize": "ByteSize",
"EmitValidation": [
"($Class == GPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8)) || $Class == FPRClass",
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16)) || $Class == GPRClass"
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16 || #ByteSize == 32)) || $Class == GPRClass"
]
},
"StoreContextIndexed SSA:$Value, GPR:$Index, u8:#ByteSize, u32:$BaseOffset, u32:$Stride, RegisterClass:$Class": {
@@ -397,7 +393,7 @@
"EmitValidation": [
"WalkFindRegClass($Value) == $Class",
"($Class == GPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8)) || $Class == FPRClass",
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16)) || $Class == GPRClass"
"($Class == FPRClass && (#ByteSize == 1 || #ByteSize == 2 || #ByteSize == 4 || #ByteSize == 8 || #ByteSize == 16 || #ByteSize == 32)) || $Class == GPRClass"
]
},
@@ -845,27 +841,25 @@
"DestSize": "std::max<uint8_t>(4, std::max<uint8_t>(GetOpSize(_TrueVal), GetOpSize(_FalseVal)))"
},
"GPR = Extr GPR:$Upper, GPR:$Lower, u8:$LSB": {
"Desc": ["Concats the two GPRs to create a value that is the size of the full two GPRs",
"It then extracts a bitfield width that size of a GPR from the LSB",
"Valid LSB range is 0-31 for 32bit and 0-63 for 64bit",
"<Size * 2> ConcatValue = $Upper:$Lower",
"Result = ConcatValue<LSB+Size - 1: LSB>"
]
"Desc": ["Concats the two GPRs to create a value that is the size of the full two GPRs",
"It then extracts a bitfield width that size of a GPR from the LSB",
"Valid LSB range is 0-31 for 32bit and 0-63 for 64bit",
"<Size * 2> ConcatValue = $Upper:$Lower",
"Result = ConcatValue<LSB+Size - 1: LSB>"
]
},
"GPR = PDep GPR:$Input, GPR:$Mask": {
"Desc": [
"Performs a parallel bit deposit.",
"Takes the contiguous low-order bits and deposits them into",
"the destination at the locations specified by the Mask."
]
"Desc": ["Performs a parallel bit deposit.",
"Takes the contiguous low-order bits and deposits them into",
"the destination at the locations specified by the Mask."
]
},
"GPR = PExt GPR:$Input, GPR:$Mask": {
"Desc": [
"Performs a parallel bit extract.",
"Each bit set in the mask will select the corresponding bit in the Input",
"and transfers them to the lower contiguous bits in the destination."
]
"Desc": ["Performs a parallel bit extract.",
"Each bit set in the mask will select the corresponding bit in the Input",
"and transfers them to the lower contiguous bits in the destination."
]
},
"GPR = LDiv GPR:$Lower, GPR:$Upper, GPR:$Divisor": {
@@ -924,15 +918,6 @@
}
},
"Vector": {
"FPR = SplatVector2 FPR:$Scalar": {
"NumElements": "2",
"DestSize": "GetOpSize(_Scalar) * 2"
},
"FPR = SplatVector4 FPR:$Scalar": {
"NumElements": "4",
"DestSize": "GetOpSize(_Scalar) * 4"
},
"FPR = VMov u8:#RegisterSize, FPR:$Source": {
"Desc" : ["Copy vector register",
"When Register size is smaller than Source register size,",
@@ -941,12 +926,6 @@
"DestSize": "RegisterSize"
},
"FPR = VBitcast u8:#RegisterSize, u8:#ElementSize, FPR:$Source": {
"Desc": ["Workaround for issue with LLVM breaking when loading scalar elements to vectors"],
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
},
"FPR = VectorZero u8:#RegisterSize": {
"Desc": ["Generates a vector zero",
"Useful to generate a zero vector without any previous dependencies"
@@ -1038,9 +1017,6 @@
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
},
"FPR = VExtractElement u8:#RegisterSize, u8:#ElementSize, FPR:$Vector, u8:$Index": {
"DestSize": "ElementSize"
},
"FPR = VDupElement u8:#RegisterSize, u8:#ElementSize, FPR:$Vector, u8:$Index": {
"Desc": ["Duplicates one element from the source register across the whole register"],
"DestSize": "RegisterSize",
@@ -1089,7 +1065,7 @@
},
"FPR = VSXTL2 u8:#RegisterSize, u8:#ElementSize, FPR:$Vector": {
"Desc": ["Sign extends elements from the source element size to the next size up",
"Source elements come from the upper 64bits of the register"
"Source elements come from the upper half of the register"
],
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / (ElementSize << 1)"
@@ -1101,7 +1077,7 @@
},
"FPR = VUXTL2 u8:#RegisterSize, u8:#ElementSize, FPR:$Vector": {
"Desc": ["Zero extends elements from the source element size to the next size up",
"Source elements come from the upper 64bits of the register"
"Source elements come from the upper half of the register"
],
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / (ElementSize << 1)"
@@ -1124,9 +1100,9 @@
},
"FPR = VRev64 u8:#RegisterSize, u8:#ElementSize, FPR:$Vector": {
"Desc" : ["Reverses elements in 64-bit halfwords",
"Available element size: 1byte, 2 byte, 4 byte"
],
"Desc" : ["Reverses elements in 64-bit halfwords",
"Available element size: 1byte, 2 byte, 4 byte"
],
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
},
@@ -1320,10 +1296,6 @@
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
},
"FPR = VInsScalarElement u8:#RegisterSize, u8:#ElementSize, u8:$DestIdx, FPR:$DestVector, FPR:$SrcScalar": {
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
},
"FPR = VInsGPR u8:#RegisterSize, u8:#ElementSize, u8:$DestIdx, FPR:$DestVector, GPR:$Src": {
"DestSize": "RegisterSize",
"NumElements": "RegisterSize / ElementSize"
@@ -1587,9 +1559,9 @@
"DestSize": "16"
},
"GPR = F80Cmp FPR:$X80Src1, FPR:$X80Src2, u32:$Flags": {
"Desc": ["Does a scalar unordered compare and stores the asked for flags in to a GPR",
"Ordering flag result is true if either float input is NaN"
],
"Desc": ["Does a scalar unordered compare and stores the asked for flags in to a GPR",
"Ordering flag result is true if either float input is NaN"
],
"DestSize": "4"
},
"FPR = F80BCDLoad FPR:$X80Src": {
+3
View File
@@ -12,6 +12,7 @@ $end_info$
#include "Interface/IR/Passes/RegisterAllocationPass.h"
#include <FEXCore/Config/Config.h>
#include <FEXCore/Utils/Profiler.h>
namespace FEXCore::IR {
class IREmitter;
@@ -66,6 +67,8 @@ void PassManager::InsertRegisterAllocationPass(bool OptimizeSRA, bool SupportsAV
}
bool PassManager::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::Run");
bool Changed = false;
for (auto const &Pass : Passes) {
Changed |= Pass->Run(IREmit);
+12 -9
View File
@@ -6,7 +6,7 @@ $end_info$
*/
#if defined(_M_ARM_64)
#if JIT_ARM64
//aarch64 heuristics
#include "aarch64/assembler-aarch64.h"
#include "aarch64/cpu-aarch64.h"
@@ -20,6 +20,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <bit>
#include <cstdint>
@@ -45,20 +46,20 @@ uint64_t getMask(IROp_Header* Op) {
return (~0ULL) >> (64 - NumBits);
}
#ifdef _M_X86_64
// very lazy heuristics
static bool IsImmLogical(uint64_t imm, unsigned width) { return imm < 0x8000'0000; }
static bool IsImmAddSub(uint64_t imm) { return imm < 0x8000'0000; }
static bool IsMemoryScale(uint64_t Scale, uint8_t AccessSize) {
return Scale == 1 || Scale == 2 || Scale == 4 || Scale == 8;
}
#elif defined(_M_ARM_64)
#if JIT_ARM64
//aarch64 heuristics
static bool IsImmLogical(uint64_t imm, unsigned width) { if (width < 32) width = 32; return vixl::aarch64::Assembler::IsImmLogical(imm, width); }
static bool IsImmAddSub(uint64_t imm) { return vixl::aarch64::Assembler::IsImmAddSub(imm); }
static bool IsMemoryScale(uint64_t Scale, uint8_t AccessSize) {
return Scale == AccessSize;
}
#elif JIT_X86_64
// very lazy heuristics
static bool IsImmLogical(uint64_t imm, unsigned width) { return imm < 0x8000'0000; }
static bool IsImmAddSub(uint64_t imm) { return imm < 0x8000'0000; }
static bool IsMemoryScale(uint64_t Scale, uint8_t AccessSize) {
return Scale == 1 || Scale == 2 || Scale == 4 || Scale == 8;
}
#else
#error No inline constant heuristics for this target
#endif
@@ -1028,6 +1029,8 @@ bool ConstProp::ConstantInlining(IREmitter *IREmit, const IRListView& CurrentIR)
}
bool ConstProp::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::ConstProp");
bool Changed = false;
auto CurrentIR = IREmit->ViewIR();
auto OriginalWriteCursor = IREmit->GetWriteCursor();
@@ -9,6 +9,7 @@ $end_info$
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/Profiler.h>
#include <memory>
@@ -22,6 +23,7 @@ private:
};
bool DeadCodeElimination::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::DCE");
auto CurrentIR = IREmit->ViewIR();
int NumRemoved = 0;
@@ -13,6 +13,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <array>
#include <memory>
@@ -76,24 +77,6 @@ namespace {
std::vector<ContextMemberInfo> ClassificationInfo;
};
constexpr static std::array<LastAccessType, 15> DefaultAccess = {
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_INVALID, // SSE padding in non-AVX case
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
ACCESS_NONE,
};
static void ClassifyContextStruct(ContextInfo *ContextClassificationInfo, bool SupportsAVX) {
auto ContextClassification = &ContextClassificationInfo->ClassificationInfo;
@@ -102,7 +85,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, rip),
sizeof(FEXCore::Core::CPUState::rip),
},
DefaultAccess[0],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
@@ -112,62 +95,134 @@ namespace {
offsetof(FEXCore::Core::CPUState, gregs[0]) + sizeof(FEXCore::Core::CPUState::gregs[0]) * i,
FEXCore::Core::CPUState::GPR_REG_SIZE,
},
DefaultAccess[1],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, es),
sizeof(FEXCore::Core::CPUState::es),
offsetof(FEXCore::Core::CPUState, es_idx),
sizeof(FEXCore::Core::CPUState::es_idx),
},
DefaultAccess[2],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, cs),
sizeof(FEXCore::Core::CPUState::cs),
offsetof(FEXCore::Core::CPUState, cs_idx),
sizeof(FEXCore::Core::CPUState::cs_idx),
},
DefaultAccess[3],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, ss),
sizeof(FEXCore::Core::CPUState::ss),
offsetof(FEXCore::Core::CPUState, ss_idx),
sizeof(FEXCore::Core::CPUState::ss_idx),
},
DefaultAccess[4],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, ds),
sizeof(FEXCore::Core::CPUState::ds),
offsetof(FEXCore::Core::CPUState, ds_idx),
sizeof(FEXCore::Core::CPUState::ds_idx),
},
DefaultAccess[5],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, gs),
sizeof(FEXCore::Core::CPUState::gs),
offsetof(FEXCore::Core::CPUState, gs_idx),
sizeof(FEXCore::Core::CPUState::gs_idx),
},
DefaultAccess[6],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, fs),
sizeof(FEXCore::Core::CPUState::fs),
offsetof(FEXCore::Core::CPUState, fs_idx),
sizeof(FEXCore::Core::CPUState::fs_idx),
},
DefaultAccess[7],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, _pad),
sizeof(FEXCore::Core::CPUState::_pad),
},
ACCESS_INVALID,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, es_cached),
sizeof(FEXCore::Core::CPUState::es_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, cs_cached),
sizeof(FEXCore::Core::CPUState::cs_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, ss_cached),
sizeof(FEXCore::Core::CPUState::ss_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, ds_cached),
sizeof(FEXCore::Core::CPUState::ds_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, gs_cached),
sizeof(FEXCore::Core::CPUState::gs_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, fs_cached),
sizeof(FEXCore::Core::CPUState::fs_cached),
},
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
ContextClassification->emplace_back(ContextMemberInfo{
ContextMemberClassification {
offsetof(FEXCore::Core::CPUState, _pad2),
sizeof(FEXCore::Core::CPUState::_pad2),
},
ACCESS_INVALID,
FEXCore::IR::InvalidClass,
});
@@ -178,7 +233,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, xmm.avx.data[0][0]) + FEXCore::Core::CPUState::XMM_AVX_REG_SIZE * i,
FEXCore::Core::CPUState::XMM_AVX_REG_SIZE,
},
DefaultAccess[8],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
@@ -189,7 +244,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, xmm.sse.data[0][0]) + FEXCore::Core::CPUState::XMM_SSE_REG_SIZE * i,
FEXCore::Core::CPUState::XMM_SSE_REG_SIZE,
},
DefaultAccess[8],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
@@ -199,7 +254,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, xmm.sse.pad[0][0]),
static_cast<uint16_t>(FEXCore::Core::CPUState::XMM_SSE_REG_SIZE * FEXCore::Core::CPUState::NUM_XMMS),
},
DefaultAccess[9],
ACCESS_INVALID,
FEXCore::IR::InvalidClass,
});
}
@@ -210,7 +265,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, flags[0]) + sizeof(FEXCore::Core::CPUState::flags[0]) * i,
FEXCore::Core::CPUState::FLAG_SIZE,
},
DefaultAccess[10],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
@@ -221,7 +276,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, mm[0][0]) + sizeof(FEXCore::Core::CPUState::mm[0]) * i,
FEXCore::Core::CPUState::MM_REG_SIZE
},
DefaultAccess[11],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
@@ -233,7 +288,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, gdt[0]) + sizeof(FEXCore::Core::CPUState::gdt[0]) * i,
sizeof(FEXCore::Core::CPUState::gdt[0]),
},
DefaultAccess[12],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
}
@@ -244,7 +299,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, FCW),
sizeof(FEXCore::Core::CPUState::FCW),
},
DefaultAccess[13],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
@@ -254,7 +309,7 @@ namespace {
offsetof(FEXCore::Core::CPUState, FTW),
sizeof(FEXCore::Core::CPUState::FTW),
},
DefaultAccess[14],
ACCESS_NONE,
FEXCore::IR::InvalidClass,
});
@@ -288,40 +343,55 @@ namespace {
ContextClassification->at(Offset).StoreNode = nullptr;
};
size_t Offset = 0;
SetAccess(Offset++, DefaultAccess[0]);
SetAccess(Offset++, ACCESS_NONE);
for (size_t i = 0; i < FEXCore::Core::CPUState::NUM_GPRS; ++i) {
SetAccess(Offset++, DefaultAccess[1]);
SetAccess(Offset++, ACCESS_NONE);
}
SetAccess(Offset++, DefaultAccess[2]);
SetAccess(Offset++, DefaultAccess[3]);
SetAccess(Offset++, DefaultAccess[4]);
SetAccess(Offset++, DefaultAccess[5]);
SetAccess(Offset++, DefaultAccess[6]);
SetAccess(Offset++, DefaultAccess[7]);
// Segment indexes
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
// Pad
SetAccess(Offset++, ACCESS_INVALID);
// Segments
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
// Pad2
SetAccess(Offset++, ACCESS_INVALID);
for (size_t i = 0; i < FEXCore::Core::CPUState::NUM_XMMS; ++i) {
SetAccess(Offset++, DefaultAccess[8]);
SetAccess(Offset++, ACCESS_NONE);
}
if (!SupportsAVX) {
SetAccess(Offset++, DefaultAccess[9]);
SetAccess(Offset++, ACCESS_NONE);
}
for (size_t i = 0; i < FEXCore::Core::CPUState::NUM_FLAGS; ++i) {
SetAccess(Offset++, DefaultAccess[10]);
SetAccess(Offset++, ACCESS_NONE);
}
for (size_t i = 0; i < FEXCore::Core::CPUState::NUM_MMS; ++i) {
SetAccess(Offset++, DefaultAccess[11]);
SetAccess(Offset++, ACCESS_NONE);
}
for (size_t i = 0; i < FEXCore::Core::CPUState::NUM_GDTS; ++i) {
SetAccess(Offset++, DefaultAccess[12]);
SetAccess(Offset++, ACCESS_NONE);
}
SetAccess(Offset++, DefaultAccess[13]);
SetAccess(Offset++, DefaultAccess[14]);
SetAccess(Offset++, ACCESS_NONE);
SetAccess(Offset++, ACCESS_NONE);
}
struct BlockInfo {
@@ -449,7 +519,6 @@ void RCLSE::CalculateControlFlowInfo(FEXCore::IR::IREmitter *IREmit) {
* %ssa26 i128 = LoadMem %ssa25 i64, 0x10
* (%%ssa27) StoreContext %ssa26 i128, 0x10, 0xb0
* %ssa28 i128 = LoadContext 0x10, 0x90
* %ssa29 i128 = VBitcast %ssa26 i128
*
* eg.
* %ssa6 i128 = LoadContext 0x10, 0x90
@@ -462,13 +531,11 @@ void RCLSE::CalculateControlFlowInfo(FEXCore::IR::IREmitter *IREmit) {
* eg.
* (%%ssa189) StoreContext %ssa188 i128, 0x10, 0xa0
* %ssa190 i128 = LoadContext 0x10, 0x90
* %ssa191 i128 = VBitcast %ssa188 i128
* %ssa192 i128 = VAdd %ssa191 i128, %ssa190 i128, 0x10, 0x4
* %ssa192 i128 = VAdd %ssa188 i128, %ssa190 i128, 0x10, 0x4
* (%%ssa193) StoreContext %ssa192 i128, 0x10, 0xa0
* Converts to
* %ssa173 i128 = LoadContext 0x10, 0x90
* %ssa174 i128 = VBitcast %ssa172 i128
* %ssa175 i128 = VAdd %ssa174 i128, %ssa173 i128, 0x10, 0x4
* %ssa175 i128 = VAdd %ssa172 i128, %ssa173 i128, 0x10, 0x4
* (%%ssa176) StoreContext %ssa175 i128, 0x10, 0xa0
*/
@@ -698,6 +765,7 @@ bool RCLSE::RedundantStoreLoadElimination(FEXCore::IR::IREmitter *IREmit) {
}
bool RCLSE::Run(FEXCore::IR::IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::RCLSE");
// XXX: We don't do cross-block optimizations yet
//CalculateControlFlowInfo(IREmit);
bool Changed = false;
@@ -12,6 +12,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <memory>
#include <stddef.h>
@@ -154,6 +155,8 @@ struct Info {
*
*/
bool DeadStoreElimination::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::DSE");
std::unordered_map<OrderedNode*, Info> InfoMap;
bool Changed = false;
@@ -13,6 +13,7 @@ $end_info$
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/MathUtils.h>
#include <FEXCore/Utils/Profiler.h>
#include <algorithm>
#include <cstdint>
@@ -52,6 +53,8 @@ IRCompaction::IRCompaction(FEXCore::Utils::IntrusivePooledAllocator &Allocator)
}
bool IRCompaction::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::IRCompaction");
LocalBuilder.ReownOrClaimBuffer();
auto CurrentIR = IREmit->ViewIR();
@@ -14,6 +14,7 @@ $end_info$
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/IR/RegisterAllocationData.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <cstdint>
#include <memory>
@@ -32,6 +33,8 @@ IRValidation::~IRValidation() {
}
bool IRValidation::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::IRValidation");
bool HadError = false;
bool HadWarning = false;
@@ -9,6 +9,7 @@ $end_info$
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/Profiler.h>
#include <memory>
#include <stdint.h>
@@ -53,6 +54,8 @@ bool LongDivideEliminationPass::IsSextOp(IREmitter *IREmit, OrderedNodeWrapper L
}
bool LongDivideEliminationPass::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::LDE");
bool Changed = false;
auto CurrentIR = IREmit->ViewIR();
auto OriginalWriteCursor = IREmit->GetWriteCursor();
@@ -9,6 +9,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include "Interface/IR/PassManager.h"
@@ -24,6 +25,8 @@ public:
};
bool PhiValidation::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::PHIValidation");
bool HadError = false;
auto CurrentIR = IREmit->ViewIR();
@@ -6,7 +6,7 @@
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/IR/RegisterAllocationData.h>
#include <FEXCore/Utils/Profiler.h>
#include <algorithm>
#include <deque>
@@ -191,6 +191,8 @@ private:
bool RAValidation::Run(IREmitter *IREmit) {
if (!Manager->HasPass("RA")) return false;
FEXCORE_PROFILE_SCOPED("PassManager::RAValidation");
IR::RegisterAllocationData* RAData = Manager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData();
BlockExitState.clear();
// BlocksToVisit will already be empty
@@ -8,6 +8,8 @@ $end_info$
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/Profiler.h>
#include "Interface/IR/PassManager.h"
#include <array>
@@ -32,6 +34,8 @@ public:
*
*/
bool DeadFlagCalculationEliminination::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::DFE");
std::array<OrderedNode*, 32> LastValidFlagStores{};
bool Changed = false;
@@ -15,6 +15,8 @@ $end_info$
#include <FEXCore/Utils/BucketList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/MathUtils.h>
#include <FEXCore/Utils/Profiler.h>
#include <FEXHeaderUtils/TypeDefines.h>
#include <algorithm>
@@ -1527,6 +1529,7 @@ namespace {
}
bool ConstrainedRAPass::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::RA");
bool Changed = false;
auto IR = IREmit->ViewIR();
@@ -11,6 +11,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <memory>
#include <stddef.h>
@@ -76,6 +77,8 @@ private:
*
*/
bool StaticRegisterAllocationPass::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::SRA");
auto CurrentIR = IREmit->ViewIR();
for (auto [BlockNode, BlockIROp] : CurrentIR.GetBlocks()) {
@@ -11,6 +11,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/HLE/SyscallHandler.h>
#include <FEXCore/Utils/Profiler.h>
#include <memory>
#include <stdint.h>
@@ -23,6 +24,8 @@ public:
};
bool SyscallOptimization::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::SyscallOpt");
bool Changed = false;
auto CurrentIR = IREmit->ViewIR();
@@ -11,6 +11,7 @@ $end_info$
#include <FEXCore/IR/IREmitter.h>
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#include <functional>
#include <memory>
@@ -36,6 +37,8 @@ public:
};
bool ValueDominanceValidation::Run(IREmitter *IREmit) {
FEXCORE_PROFILE_SCOPED("PassManager::ValueDominanceValidation");
bool HadError = false;
auto CurrentIR = IREmit->ViewIR();
+133 -79
View File
@@ -6,6 +6,10 @@
#include <FEXHeaderUtils/TypeDefines.h>
#include <array>
#include <asm-generic/errno-base.h>
#include <cctype>
#include <cstdio>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/user.h>
#ifdef ENABLE_JEMALLOC
@@ -41,6 +45,8 @@ namespace FEXCore::Allocator {
FREE_Hook free {::free};
#endif
uint64_t HostVASize{};
using GLIBC_MALLOC_Hook = void*(*)(size_t, const void *caller);
using GLIBC_REALLOC_Hook = void*(*)(void*, size_t, const void *caller);
using GLIBC_FREE_Hook = void(*)(void*, const void *caller);
@@ -93,6 +99,10 @@ namespace FEXCore::Allocator {
#pragma GCC diagnostic pop
FEX_DEFAULT_VISIBILITY size_t DetermineVASize() {
if (HostVASize) {
return HostVASize;
}
static constexpr std::array<uintptr_t, 7> TLBSizes = {
57,
52,
@@ -123,6 +133,7 @@ namespace FEXCore::Allocator {
};
if (Find(Size)) {
HostVASize = Bits;
return Bits;
}
}
@@ -131,91 +142,143 @@ namespace FEXCore::Allocator {
FEX_UNREACHABLE;
}
PtrCache* StealMemoryRegion(uintptr_t Begin, uintptr_t End) {
PtrCache *Cache{};
uint64_t CacheSize{};
uint64_t CurrentCacheOffset = 0;
constexpr std::array<size_t, 10> ReservedVMARegionSizes = {{
// Anything larger than 64GB fails out
64ULL * 1024 * 1024 * 1024, // 64GB
32ULL * 1024 * 1024 * 1024, // 32GB
16ULL * 1024 * 1024 * 1024, // 16GB
4ULL * 1024 * 1024 * 1024, // 4GB
1ULL * 1024 * 1024 * 1024, // 1GB
512ULL * 1024 * 1024, // 512MB
128ULL * 1024 * 1024, // 128MB
32ULL * 1024 * 1024, // 32MB
1ULL * 1024 * 1024, // 1MB
4096ULL // One page
}};
constexpr size_t AllocationSizeMaxIndex = ReservedVMARegionSizes.size() - 1;
uint64_t CurrentSizeIndex = 0;
#define STEAL_LOG(...) // fprintf(stderr, __VA_ARGS__)
int PROT_FLAGS = PROT_READ | PROT_WRITE;
for (size_t MemoryOffset = Begin; MemoryOffset < End;) {
size_t AllocationSize = ReservedVMARegionSizes[CurrentSizeIndex];
size_t MemoryOffsetUpper = MemoryOffset + AllocationSize;
std::vector<MemoryRegion> StealMemoryRegion(uintptr_t Begin, uintptr_t End) {
void * const StackLocation = alloca(0);
const uintptr_t StackLocation_u64 = reinterpret_cast<uintptr_t>(StackLocation);
std::vector<MemoryRegion> Regions;
// If we would go above the upper bound on size then try the next size
if (MemoryOffsetUpper > End) {
++CurrentSizeIndex;
continue;
int MapsFD = open("/proc/self/maps", O_RDONLY);
LogMan::Throw::AFmt(MapsFD != -1, "Failed to open /proc/self/maps");
enum {ParseBegin, ParseEnd, ScanEnd} State = ParseBegin;
uintptr_t RegionBegin = 0;
uintptr_t RegionEnd = 0;
uintptr_t PreviousMapEnd = 0;
char Buffer[2048];
const char *Cursor;
ssize_t Remaining = 0;
for(;;) {
if (Remaining == 0) {
do {
Remaining = read(MapsFD, Buffer, sizeof(Buffer));
} while ( Remaining == -1 && errno == EAGAIN);
Cursor = Buffer;
}
void *Ptr = ::mmap(reinterpret_cast<void*>(MemoryOffset), AllocationSize, PROT_FLAGS, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE | MAP_FIXED_NOREPLACE, -1, 0);
if (Remaining == 0 && State == ParseBegin) {
STEAL_LOG("[%d] EndOfFile; RegionBegin: %016lX RegionEnd: %016lX\n", __LINE__, RegionBegin, RegionEnd);
// If we managed to allocate and not get the address we want then unmap it
// This happens with kernels older than 4.17
if (reinterpret_cast<uintptr_t>(Ptr) + AllocationSize > End) {
::munmap(Ptr, AllocationSize);
Ptr = reinterpret_cast<void*>(~0ULL);
}
const auto MapBegin = std::max(RegionEnd, Begin);
const auto MapEnd = End;
// If we failed to allocate and we are on the smallest allocation size then just continue onward
// This page was unmappable
if (reinterpret_cast<uintptr_t>(Ptr) == ~0ULL && CurrentSizeIndex == AllocationSizeMaxIndex) {
CurrentSizeIndex = 0;
MemoryOffset += AllocationSize;
continue;
}
STEAL_LOG(" MapBegin: %016lX MapEnd: %016lX\n", MapBegin, MapEnd);
// Congratulations we were able to map this bit
// Reset and claim it was available
if (reinterpret_cast<uintptr_t>(Ptr) != ~0ULL) {
if (!Cache) {
Cache = reinterpret_cast<PtrCache *>(Ptr);
CacheSize = AllocationSize;
PROT_FLAGS = PROT_NONE;
}
else {
Cache[CurrentCacheOffset] = {
.Ptr = static_cast<uint64_t>(reinterpret_cast<uint64_t>(Ptr)),
.Size = static_cast<uint64_t>(AllocationSize)
};
++CurrentCacheOffset;
if (MapEnd > MapBegin) {
STEAL_LOG(" Reserving\n");
auto MapSize = MapEnd - MapBegin;
auto Alloc = mmap((void*)MapBegin, MapSize, PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE | MAP_FIXED_NOREPLACE, -1, 0);
LogMan::Throw::AFmt(Alloc != MAP_FAILED, "mmap({:x},{:x}) failed", MapBegin, MapSize);
LogMan::Throw::AFmt(Alloc == (void*)MapBegin, "mmap({},{:x}) returned {} instead of {:x}", Alloc, MapBegin);
Regions.push_back({(void*)MapBegin, MapSize});
}
CurrentSizeIndex = 0;
MemoryOffset += AllocationSize;
close(MapsFD);
return Regions;
}
LogMan::Throw::AFmt(Remaining > 0, "Failed to parse /proc/self/maps");
auto c = *Cursor++;
Remaining--;
if (State == ScanEnd) {
if (c == '\n') {
State = ParseBegin;
}
continue;
}
// Couldn't allocate at this size
// Increase and continue
++CurrentSizeIndex;
if (State == ParseBegin) {
if (c == '-') {
STEAL_LOG("[%d] ParseBegin; RegionBegin: %016lX RegionEnd: %016lX\n", __LINE__, RegionBegin, RegionEnd);
const auto MapBegin = std::max(RegionEnd, Begin);
const auto MapEnd = std::min(RegionBegin, End);
// Store the location we are going to map.
PreviousMapEnd = MapEnd;
STEAL_LOG(" MapBegin: %016lX MapEnd: %016lX\n", MapBegin, MapEnd);
if (MapEnd > MapBegin) {
STEAL_LOG(" Reserving\n");
auto MapSize = MapEnd - MapBegin;
auto Alloc = mmap((void*)MapBegin, MapSize, PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE | MAP_FIXED_NOREPLACE, -1, 0);
LogMan::Throw::AFmt(Alloc != MAP_FAILED, "mmap({:x},{:x}) failed", MapBegin, MapSize);
LogMan::Throw::AFmt(Alloc == (void*)MapBegin, "mmap({},{:x}) returned {} instead of {:x}", Alloc, MapBegin);
Regions.push_back({(void*)MapBegin, MapSize});
}
RegionBegin = 0;
RegionEnd = 0;
State = ParseEnd;
continue;
} else {
LogMan::Throw::AFmt(std::isalpha(c) || std::isdigit(c), "Unexpected char '{}' in ParseBegin", c);
RegionBegin = (RegionBegin << 4) | (c <= '9' ? (c - '0') : (c - 'a' + 10));
}
}
if (State == ParseEnd) {
if (c == ' ') {
STEAL_LOG("[%d] ParseEnd; RegionBegin: %016lX RegionEnd: %016lX\n", __LINE__, RegionBegin, RegionEnd);
State = ScanEnd;
// If the previous map's ending and the region we just parsed overlap the stack then we need to save the stack mapping.
// Otherwise we will have severely limited stack size which crashes quickly.
if (PreviousMapEnd <= StackLocation_u64 && RegionEnd > StackLocation_u64) {
auto BelowStackRegion = Regions.back();
LOGMAN_THROW_AA_FMT(reinterpret_cast<uint64_t>(BelowStackRegion.Ptr) + BelowStackRegion.Size == PreviousMapEnd,
"This needs to match");
// Allocate the region under the stack as READ | WRITE so the stack can still grow
auto Alloc = mmap(BelowStackRegion.Ptr, BelowStackRegion.Size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE | MAP_FIXED, -1, 0);
LogMan::Throw::AFmt(Alloc != MAP_FAILED, "mmap({:x},{:x}) failed", BelowStackRegion.Ptr, BelowStackRegion.Size);
LogMan::Throw::AFmt(Alloc == BelowStackRegion.Ptr, "mmap({},{:x}) returned {} instead of {:x}", Alloc, BelowStackRegion.Ptr);
Regions.pop_back();
}
continue;
} else {
LogMan::Throw::AFmt(std::isalpha(c) || std::isdigit(c), "Unexpected char '{}' in ParseEnd", c);
RegionEnd = (RegionEnd << 4) | (c <= '9' ? (c - '0') : (c - 'a' + 10));
}
}
}
Cache[CurrentCacheOffset] = {
.Ptr = static_cast<uint64_t>(reinterpret_cast<uint64_t>(Cache)),
.Size = CacheSize,
};
return Cache;
ERROR_AND_DIE_FMT("unreachable");
}
PtrCache* Steal48BitVA() {
std::vector<MemoryRegion> Steal48BitVA() {
size_t Bits = FEXCore::Allocator::DetermineVASize();
if (Bits < 48) {
return nullptr;
return {};
}
uintptr_t Begin48BitVA = 0x0'8000'0000'0000ULL;
@@ -223,18 +286,9 @@ namespace FEXCore::Allocator {
return StealMemoryRegion(Begin48BitVA, End48BitVA);
}
void ReclaimMemoryRegion(PtrCache* Regions) {
if (Regions == nullptr) {
return;
}
for (size_t i = 0;; ++i) {
void *Ptr = reinterpret_cast<void*>(Regions[i].Ptr);
size_t Size = Regions[i].Size;
::munmap(Ptr, Size);
if (Ptr == Regions) {
break;
}
void ReclaimMemoryRegion(const std::vector<MemoryRegion> &Regions) {
for (const auto &Region: Regions) {
::munmap(Region.Ptr, Region.Size);
}
}
}
+130 -195
View File
@@ -69,8 +69,14 @@ namespace Alloc::OSAllocator {
struct LiveVMARegion {
ReservedVMARegion *SlabInfo;
uint64_t FreeSpace{};
uint64_t NumManagedPages{};
uint32_t LastPageAllocation{};
FEXCore::FlexBitSet<uint64_t> UsedPages;
bool HadMunmap{};
// Align UsedPages so it pads to the next page.
// Necessary to take advantage of madvise zero page pooling.
using FlexBitElementType = uint64_t;
alignas(4096) FEXCore::FlexBitSet<FlexBitElementType> UsedPages;
// This returns the size of the LiveVMARegion in addition to the flex set that tracks the used data
// The LiveVMARegion lives at the start of the VMA region which means on initialization we need to set that
@@ -82,8 +88,8 @@ namespace Alloc::OSAllocator {
// 0x100'0000 Pages
// 1 bit per page for tracking means 0x20'0000 (Pages / 8) bytes of flex space
// Which is 2MB of tracking
uint64_t NumElements = (Size >> FHU::FEX_PAGE_SHIFT) * sizeof(uint64_t);
return sizeof(LiveVMARegion) + FEXCore::FlexBitSet<uint64_t>::Size(NumElements);
uint64_t NumElements = (Size >> FHU::FEX_PAGE_SHIFT) * sizeof(FlexBitElementType);
return sizeof(LiveVMARegion) + FEXCore::FlexBitSet<FlexBitElementType>::Size(NumElements);
}
static void InitializeVMARegionUsed(LiveVMARegion *Region, size_t AdditionalSize) {
@@ -92,17 +98,26 @@ namespace Alloc::OSAllocator {
Region->FreeSpace = Region->SlabInfo->RegionSize - SizePlusManagedData;
size_t NumPages = SizePlusManagedData >> FHU::FEX_PAGE_SHIFT;
// Memset the full tracking to zero to state nothing used
Region->UsedPages.MemSet(Region->SlabInfo->RegionSize >> FHU::FEX_PAGE_SHIFT);
size_t NumManagedPages = SizePlusManagedData >> FHU::FEX_PAGE_SHIFT;
size_t ManagedSize = NumManagedPages << FHU::FEX_PAGE_SHIFT;
// Use madvise to set the full tracking region to zero.
// This ensures unused pages are zero, while not having the backing pages consuming memory.
::madvise(Region->UsedPages.Memory + ManagedSize, (Region->SlabInfo->RegionSize >> FHU::FEX_PAGE_SHIFT) - ManagedSize, MADV_DONTNEED);
// Use madvise to claim WILLNEED on the beginning pages for initial state tracking.
// Improves performance of the following MemClear by not doing a page level fault dance for data necessary to track >170TB of used pages.
::madvise(Region->UsedPages.Memory, ManagedSize, MADV_WILLNEED);
// Set our reserved pages
for (size_t i = 0; i < NumPages; ++i) {
// Set our used pages
Region->UsedPages.Set(i);
}
Region->LastPageAllocation = NumPages;
Region->UsedPages.MemSet(NumManagedPages);
Region->LastPageAllocation = NumManagedPages;
Region->NumManagedPages = NumManagedPages;
}
};
static_assert(sizeof(LiveVMARegion) == 4096, "Needs to be the size of a page");
static_assert(std::is_trivially_copyable<LiveVMARegion>::value, "Needs to be trivially copyable");
static_assert(offsetof(LiveVMARegion, UsedPages) == sizeof(LiveVMARegion), "FlexBitSet needs to be at the end");
@@ -119,6 +134,7 @@ namespace Alloc::OSAllocator {
ReservedVMARegion *ReservedRegion = *ReservedIterator;
ReservedRegions->erase(ReservedIterator);
// mprotect the new region we've allocated
size_t SizeOfLiveRegion = FEXCore::AlignUp(LiveVMARegion::GetSizeWithFlexSet(ReservedRegion->RegionSize), FHU::FEX_PAGE_SIZE);
size_t SizePlusManagedData = UsedSize + SizeOfLiveRegion;
@@ -131,26 +147,71 @@ namespace Alloc::OSAllocator {
// Copy over the reserved data
LiveRange->SlabInfo = ReservedRegion;
// Initialize VMA
LiveVMARegion::InitializeVMARegionUsed(LiveRange, UsedSize);
// Add to our active tracked ranges
auto LiveIter = LiveRegions->emplace_back(LiveRange);
return LiveIter;
}
// 32-bit old kernel workarounds
FEXCore::Allocator::PtrCache *Steal32BitIfOldKernel();
std::vector<FEXCore::Allocator::MemoryRegion> Steal32BitIfOldKernel();
void AllocateMemoryRegions(std::vector<FEXCore::Allocator::MemoryRegion> const &Ranges);
LiveVMARegion *FindLiveRegionForAddress(uintptr_t Addr, uintptr_t AddrEnd);
};
void OSAllocator_64Bit::DetermineVASize() {
size_t Bits = FEXCore::Allocator::DetermineVASize();
uintptr_t Size = 1ULL << Bits;
UPPER_BOUND = Size;
#if _M_X86_64 // Last page cannot be allocated on x86
UPPER_BOUND -= FHU::FEX_PAGE_SIZE;
#endif
UPPER_BOUND_PAGE = UPPER_BOUND / FHU::FEX_PAGE_SIZE;
}
OSAllocator_64Bit::LiveVMARegion *OSAllocator_64Bit::FindLiveRegionForAddress(uintptr_t Addr, uintptr_t AddrEnd) {
LiveVMARegion *LiveRegion{};
// Check active slabs to see if we can fit this
for (auto it = LiveRegions->begin(); it != LiveRegions->end(); ++it) {
uintptr_t RegionBegin = (*it)->SlabInfo->Base;
uintptr_t RegionEnd = RegionBegin + (*it)->SlabInfo->RegionSize;
if (Addr >= RegionBegin &&
Addr < RegionEnd) {
LiveRegion = *it;
// Leave our loop
break;
}
}
// Couldn't find an active region that fit
// Check reserved regions
if (!LiveRegion) {
// Didn't have a slab that fit this range
// Check our reserved regions to see if we have one that fits
for (auto it = ReservedRegions->begin(); it != ReservedRegions->end(); ++it) {
ReservedVMARegion *ReservedRegion = *it;
uintptr_t RegionEnd = ReservedRegion->Base + ReservedRegion->RegionSize;
if (Addr >= ReservedRegion->Base &&
AddrEnd < RegionEnd) {
// Found one, let's make it active
LiveRegion = MakeRegionActive(it, 0);
break;
}
}
}
return LiveRegion;
}
void *OSAllocator_64Bit::Mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset) {
if (addr != 0 &&
addr < reinterpret_cast<void*>(LOWER_BOUND)) {
@@ -189,41 +250,13 @@ void *OSAllocator_64Bit::Mmap(void *addr, size_t length, int prot, int flags, in
LiveVMARegion *LiveRegion{};
if (Fixed || Addr != 0) {
// Check active slabs to see if we can fit this
for (auto it = LiveRegions->begin(); it != LiveRegions->end(); ++it) {
uintptr_t RegionBegin = (*it)->SlabInfo->Base;
uintptr_t RegionEnd = RegionBegin + (*it)->SlabInfo->RegionSize;
if (Addr >= RegionBegin &&
Addr < RegionEnd) {
LiveRegion = *it;
// Leave our loop
break;
}
}
// Couldn't find an active region that fit
// Check reserved regions
if (!LiveRegion) {
// Didn't have a slab that fit this range
// Check our reserved regions to see if we have one that fits
for (auto it = ReservedRegions->begin(); it != ReservedRegions->end(); ++it) {
ReservedVMARegion *ReservedRegion = *it;
uintptr_t RegionEnd = ReservedRegion->Base + ReservedRegion->RegionSize;
if (Addr >= ReservedRegion->Base &&
AddrEnd < RegionEnd) {
// Found one, let's make it active
LiveRegion = MakeRegionActive(it, 0);
break;
}
}
}
LiveRegion = FindLiveRegionForAddress(Addr, AddrEnd);
}
again:
auto CheckIfRangeFits = [&AllocatedOffset](LiveVMARegion *Region, uint64_t length, int prot, int flags, int fd, off_t offset, uint64_t StartingPosition = 0) -> std::pair<LiveVMARegion*, void*> {
uint64_t AllocatedPage{};
uint64_t AllocatedPage{~0ULL};
uint64_t NumberOfPages = length >> FHU::FEX_PAGE_SHIFT;
if (Region->FreeSpace >= length) {
@@ -233,72 +266,29 @@ void *OSAllocator_64Bit::Mmap(void *addr, size_t length, int prot, int flags, in
: Region->LastPageAllocation;
size_t RegionNumberOfPages = Region->SlabInfo->RegionSize >> FHU::FEX_PAGE_SHIFT;
// Backward scan
// We need to do a backward scan first to fill any holes
// Otherwise we will very quickly run out of VMA regions (65k maximum)
for (size_t CurrentPage = LastAllocation;
CurrentPage >= NumberOfPages;) {
size_t Remaining = NumberOfPages;
assert(Remaining <= CurrentPage);
while (Remaining) {
if (Region->UsedPages[CurrentPage - Remaining]) {
// Has an intersecting range
break;
}
--Remaining;
}
if (Region->HadMunmap) {
// Backward scan
// We need to do a backward scan first to fill any holes
// Otherwise we will very quickly run out of VMA regions (65k maximum)
auto SearchResult = Region->UsedPages.BackwardScanForRange<true>(LastAllocation, NumberOfPages, Region->NumManagedPages);
if (Remaining) {
// Didn't find a slab range
CurrentPage -= Remaining;
}
else {
// We have a slab range
CurrentPage -= NumberOfPages;
AllocatedPage = SearchResult.FoundElement;
// Keep scanning backwards to not introduce ANOTHER gap
while (CurrentPage >= 1) {
if (Region->UsedPages[CurrentPage - 1]) {
// Found a used page, we can leave now
break;
}
--CurrentPage;
}
AllocatedPage = CurrentPage;
break;
// If we didn't even have a one page free in the backward search, then unclaim HadMunmap.
// Switching over to default forward search.
if (SearchResult.FoundElement == ~0ULL && !SearchResult.FoundHole) {
Region->HadMunmap = false;
}
}
// Foward Scan
if (AllocatedPage == 0) {
for (size_t CurrentPage = LastAllocation;
CurrentPage < (RegionNumberOfPages - NumberOfPages);) {
// If we have enough free space, check if we have enough free pages that are contiguous
size_t Remaining = NumberOfPages;
assert((CurrentPage + Remaining - 1) < RegionNumberOfPages);
while (Remaining) {
if (Region->UsedPages[CurrentPage + Remaining - 1]) {
// Has an intersecting range
break;
}
--Remaining;
}
if (Remaining) {
// Didn't find a slab range
CurrentPage += Remaining;
}
else {
// We have a slab range
AllocatedPage = CurrentPage;
break;
}
}
if (AllocatedPage == ~0ULL) {
auto SearchResult = Region->UsedPages.ForwardScanForRange<true>(LastAllocation, NumberOfPages, RegionNumberOfPages);
AllocatedPage = SearchResult.FoundElement;
}
if (AllocatedPage) {
if (AllocatedPage != ~0ULL) {
AllocatedOffset = Region->SlabInfo->Base + AllocatedPage * FHU::FEX_PAGE_SIZE;
// We need to setup protections for this
@@ -481,6 +471,8 @@ int OSAllocator_64Bit::Munmap(void *addr, size_t length) {
// This will let us more quickly fill holes
(*it)->LastPageAllocation = std::min((*it)->LastPageAllocation, SlabPageBegin);
(*it)->HadMunmap = true;
// XXX: Move region back to reserved list
return 0;
}
@@ -490,11 +482,11 @@ int OSAllocator_64Bit::Munmap(void *addr, size_t length) {
return 0;
}
FEXCore::Allocator::PtrCache *OSAllocator_64Bit::Steal32BitIfOldKernel() {
std::vector<FEXCore::Allocator::MemoryRegion> OSAllocator_64Bit::Steal32BitIfOldKernel() {
// First calculate kernel version
struct utsname buf{};
if (uname(&buf) == -1) {
return nullptr;
return {};
}
int32_t Major{};
@@ -512,7 +504,7 @@ FEXCore::Allocator::PtrCache *OSAllocator_64Bit::Steal32BitIfOldKernel() {
if (Version >= ((4 << 24) | (17 << 16) | 0)) {
// If the kernel is >= 4.17 then it supports MAP_FIXED_NOREPLACE
return nullptr;
return {};
}
constexpr size_t LOWER_BOUND_32 = 0x1'0000;
@@ -521,103 +513,46 @@ FEXCore::Allocator::PtrCache *OSAllocator_64Bit::Steal32BitIfOldKernel() {
return FEXCore::Allocator::StealMemoryRegion(LOWER_BOUND_32, UPPER_BOUND_32);
}
void OSAllocator_64Bit::AllocateMemoryRegions(std::vector<FEXCore::Allocator::MemoryRegion> const &Ranges) {
for (auto [Ptr, AllocationSize]: Ranges) {
if (!ObjectAlloc) {
auto MaxSize = std::min(size_t(64) * 1024 * 1024, AllocationSize);
// Allocate up to 64 MiB the first allocation for an intrusive allocator
mprotect(Ptr, MaxSize, PROT_READ | PROT_WRITE);
// This enables the kernel to use transparent large pages in the allocator which can reduce memory pressure
::madvise(Ptr, MaxSize, MADV_HUGEPAGE);
ObjectAlloc = new (Ptr) Alloc::ForwardOnlyIntrusiveArenaAllocator(Ptr, MaxSize);
ReservedRegions = ObjectAlloc->new_construct(ReservedRegions, ObjectAlloc);
LiveRegions = ObjectAlloc->new_construct(LiveRegions, ObjectAlloc);
if (AllocationSize > MaxSize) {
AllocationSize -= MaxSize;
(uint8_t*&)Ptr += MaxSize;
} else {
continue;
}
}
ReservedVMARegion *Region = ObjectAlloc->new_construct<ReservedVMARegion>();
Region->Base = reinterpret_cast<uint64_t>(Ptr);
Region->RegionSize = AllocationSize;
ReservedRegions->emplace_back(Region);
}
}
OSAllocator_64Bit::OSAllocator_64Bit() {
DetermineVASize();
auto ArrayPtr = Steal32BitIfOldKernel();
auto LowMem = Steal32BitIfOldKernel();
// On allocation try and steal the entire upper 64bits of address space for mapping
constexpr std::array<size_t, 8> ReservedVMARegionSizes = {{
// Anything larger than 64GB fails out
64ULL * 1024 * 1024 * 1024, // 64GB
32ULL * 1024 * 1024 * 1024, // 32GB
16ULL * 1024 * 1024 * 1024, // 16GB
4ULL * 1024 * 1024 * 1024, // 4GB
1ULL * 1024 * 1024 * 1024, // 1GB
512ULL * 1024 * 1024, // 512MB
128ULL * 1024 * 1024, // 128MB
4096ULL // One page
}};
auto Ranges = FEXCore::Allocator::StealMemoryRegion(LOWER_BOUND, UPPER_BOUND);
constexpr size_t AllocationSizeMaxIndex = ReservedVMARegionSizes.size() - 1;
AllocateMemoryRegions(Ranges);
// Have the first region only be 4GB VMA
// Avoids conflicts with some tests
uint64_t CurrentSizeIndex = 3;
ReservedVMARegion *PrevReserved{};
for (size_t MemoryOffset = LOWER_BOUND; MemoryOffset < UPPER_BOUND;) {
size_t AllocationSize = ReservedVMARegionSizes[CurrentSizeIndex];
size_t MemoryOffsetUpper = MemoryOffset + AllocationSize;
// If we would go above the upper bound on size then try the next size
if (MemoryOffsetUpper > UPPER_BOUND) {
++CurrentSizeIndex;
continue;
}
void *Ptr = ::mmap(reinterpret_cast<void*>(MemoryOffset), AllocationSize, PROT_NONE, MAP_FIXED_NOREPLACE | MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
// If we managed to allocate and not get the address we want then unmap it
// This happens with kernels older than 4.17
if (reinterpret_cast<uintptr_t>(Ptr) != MemoryOffset &&
reinterpret_cast<uintptr_t>(Ptr) < LOWER_BOUND) {
::munmap(Ptr, AllocationSize);
Ptr = reinterpret_cast<void*>(~0ULL);
}
// If we failed to allocate and we are on the smallest allocation size then just continue onward
// This page was unmappable
if (reinterpret_cast<uintptr_t>(Ptr) == ~0ULL && CurrentSizeIndex == AllocationSizeMaxIndex) {
CurrentSizeIndex = 0;
MemoryOffset += AllocationSize;
continue;
}
// Congratulations we were able to map this bit
// Reset and claim it was available
if (reinterpret_cast<uintptr_t>(Ptr) != ~0ULL) {
if (!ObjectAlloc) {
// Steal the first allocation for an intrusive allocator
// Will be mprotected correctly already
mprotect(Ptr, AllocationSize, PROT_READ | PROT_WRITE);
ObjectAlloc = new (Ptr) Alloc::ForwardOnlyIntrusiveArenaAllocator(Ptr, AllocationSize);
ReservedRegions = ObjectAlloc->new_construct(ReservedRegions, ObjectAlloc);
LiveRegions = ObjectAlloc->new_construct(LiveRegions, ObjectAlloc);
}
else {
// If the allocation size is large than a page, then try allowing it to be a huge page
// This enables the kernel to use transparent large pages in the allocator which can reduce memory pressure
// Considering we are allocating the entire VA space, this is a good thing
// If MADV_HUGEPAGE isn't support then this will fail harmlessly
if (AllocationSize > 4096) {
::madvise(Ptr, AllocationSize, MADV_HUGEPAGE);
}
bool Merged = false;
if (PrevReserved) {
Merged = MergeReservedRegionIfPossible(PrevReserved, reinterpret_cast<uint64_t>(Ptr), AllocationSize);
}
if (!Merged) {
ReservedVMARegion *Region = ObjectAlloc->new_construct<ReservedVMARegion>();
Region->Base = reinterpret_cast<uint64_t>(Ptr);
Region->RegionSize = AllocationSize;
ReservedRegions->emplace_back(Region);
PrevReserved = Region;
}
}
CurrentSizeIndex = 0;
MemoryOffset += AllocationSize;
continue;
}
// Couldn't allocate at this size
// Increase and continue
++CurrentSizeIndex;
}
FEXCore::Allocator::ReclaimMemoryRegion(ArrayPtr);
FEXCore::Allocator::ReclaimMemoryRegion(LowMem);
}
OSAllocator_64Bit::~OSAllocator_64Bit() {
+104
View File
@@ -1,6 +1,7 @@
#pragma once
#include <FEXCore/Utils/MathUtils.h>
#include <FEXCore/Utils/LogManager.h>
#include <cstddef>
#include <cstdint>
@@ -38,6 +39,109 @@ struct FlexBitSet final {
memset(Memory, 0xFF, FEXCore::AlignUp(Elements / MinimumSizeBits, MinimumSizeBits));
}
// Range scanning results
struct BitsetScanResults {
// Which element was found. ~0ULL if not found.
size_t FoundElement;
// During the scan, found a hole in the allocations that didn't fit.
bool FoundHole;
};
// TODO: Make {Forward,Backward}ScanForRange faster
// Currently these functions test a single bit at a time, which is fairly costly.
// The compiler emits a full element load per iteration, wasting a bunch of time on loads.
// If we change these functions to have a pre-amble and post-amble to align the primary loop to the element size then this can go significantly
// faster.
//
// Once the element scanning is aligned to the element size, we can then use native count leading zero(CLZ) and count trailing zero(CTZ)
// instructions on a full element to scan uint64_t elements per loop iteration.
// Implementation details:
// Template argument WantUnset
// Used to determine if the desired range is for set or unset ranges.
// Typically `WantUnset` should be true. Used for finding a unset range inside of a range will set elements.
//
// @param BeginningElement - The first element in the set to start scanning from.
// @param ElementCount - How many elements to find a range for fitting.
// @param MinimumElement - Minimum element in the set to search to
//
// @return The scan results
template<bool WantUnset>
BitsetScanResults BackwardScanForRange(size_t BeginningElement, size_t ElementCount, size_t MinimumElement) {
bool FoundHole {};
for (size_t CurrentPage = BeginningElement;
CurrentPage >= (MinimumElement + ElementCount);) {
size_t Remaining = ElementCount;
LOGMAN_THROW_AA_FMT(Remaining <= CurrentPage, "Scanning less than available range");
while (Remaining) {
if (this->Get(CurrentPage - Remaining) == WantUnset) {
// Has an intersecting range
break;
}
--Remaining;
}
if (Remaining) {
// If we found at least one Element hole then track that
if (Remaining != ElementCount) {
FoundHole = true;
}
// Didn't find a slab range
CurrentPage -= Remaining;
}
else {
// We have a slab range
return BitsetScanResults{CurrentPage - ElementCount, FoundHole};
}
}
return BitsetScanResults {~0ULL, FoundHole};
}
// @param BeginningElement - The first element in the set to start scanning from.
// @param ElementCount - How many elements to find a range for fitting.
// @param ElementsInSet - How many elements are in the full set.
//
// @return The scan results
template<bool WantUnset>
BitsetScanResults ForwardScanForRange(size_t BeginningElement, size_t ElementCount, size_t ElementsInSet) {
bool FoundHole {};
for (size_t CurrentElement = BeginningElement;
CurrentElement < (ElementsInSet - ElementCount);) {
// If we have enough free space, check if we have enough free pages that are contiguous
size_t Remaining = ElementCount;
LOGMAN_THROW_AA_FMT((CurrentElement + Remaining - 1) < ElementsInSet, "Scanning less than available range");
while (Remaining) {
if (this->Get(CurrentElement + Remaining - 1) == WantUnset) {
// Has an intersecting range
break;
}
--Remaining;
}
if (Remaining) {
// If we found at least one Element hole then track that
if (Remaining != ElementCount) {
FoundHole = true;
}
// Didn't find a slab range
CurrentElement += Remaining;
}
else {
// We have a slab range
return BitsetScanResults {CurrentElement, FoundHole};
}
}
return BitsetScanResults {~0ULL, FoundHole};
}
// This very explicitly doesn't let you take an address
// Is only a getter
bool operator[](size_t Element) const {
+116
View File
@@ -0,0 +1,116 @@
#include <array>
#include <cstdint>
#include <fcntl.h>
#include <limits.h>
#include <linux/magic.h>
#include <string>
#include <sys/stat.h>
#include <sys/vfs.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Profiler.h>
#define BACKEND_OFF 0
#define BACKEND_GPUVIS 1
#ifdef ENABLE_FEXCORE_PROFILER
#if FEXCORE_PROFILER_BACKEND == BACKEND_GPUVIS
namespace FEXCore::Profiler {
ProfilerBlock::ProfilerBlock(std::string_view const Format)
: DurationBegin {GetTime()}
, Format {Format} {
}
ProfilerBlock::~ProfilerBlock() {
auto Duration = GetTime() - DurationBegin;
TraceObject(Format, Duration);
}
}
namespace GPUVis {
// ftrace FD for writing trace data.
// Needs to be a raw FD since we hold this open for the entire application execution.
static int TraceFD {-1};
// Need to search the paths to find the real trace path
static std::array<char const*, 2> TraceFSDirectories {
"/sys/kernel/tracing",
"/sys/kernel/debug/tracing",
};
static bool IsTraceFS(char const* Path) {
struct statfs stat;
if (statfs(Path, &stat)) {
return false;
}
return stat.f_type == TRACEFS_MAGIC;
}
void Init() {
for (auto Path : TraceFSDirectories) {
if (IsTraceFS(Path)) {
std::string FilePath = fmt::format("{}/trace_marker", Path);
TraceFD = open(FilePath.c_str(), O_WRONLY | O_CLOEXEC);
if (TraceFD != -1) {
// Opened TraceFD, early exit
break;
}
}
}
}
void Shutdown() {
if (TraceFD != -1) {
close(TraceFD);
TraceFD = -1;
}
}
void TraceObject(std::string_view const Format, uint64_t Duration) {
if (TraceFD != -1) {
// Print the duration as something that began negative duration ago
std::string Event = fmt::format("{} (lduration=-{})\n", Format, Duration);
write(TraceFD, Event.c_str(), Event.size());
}
}
void TraceObject(std::string_view const Format) {
if (TraceFD != -1) {
std::string Event = fmt::format("{}\n", Format);
write(TraceFD, Format.data(), Format.size());
}
}
}
#else
#error Unknown profiler backend
#endif
#endif
namespace FEXCore::Profiler {
#ifdef ENABLE_FEXCORE_PROFILER
void Init() {
#if FEXCORE_PROFILER_BACKEND == BACKEND_GPUVIS
GPUVis::Init();
#endif
}
void Shutdown() {
#if FEXCORE_PROFILER_BACKEND == BACKEND_GPUVIS
GPUVis::Shutdown();
#endif
}
void TraceObject(std::string_view const Format, uint64_t Duration) {
#if FEXCORE_PROFILER_BACKEND == BACKEND_GPUVIS
GPUVis::TraceObject(Format, Duration);
#endif
}
void TraceObject(std::string_view const Format) {
#if FEXCORE_PROFILER_BACKEND == BACKEND_GPUVIS
GPUVis::TraceObject(Format);
#endif
}
#endif
}
+4
View File
@@ -122,6 +122,10 @@ namespace CPU {
bool IsAddressInCodeBuffer(uintptr_t Address) const;
protected:
// Max spill slot size in bytes. We need at most 32 bytes
// to be able to handle a 256-bit vector store to a slot.
constexpr static uint32_t MaxSpillSlotSize = 32;
FEXCore::Core::InternalThreadState *ThreadState;
size_t InitialCodeSize, MaxCodeSize;
+9
View File
@@ -5,6 +5,7 @@
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Core/CPUID.h>
#include <FEXCore/IR/IR.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include <istream>
@@ -280,4 +281,12 @@ namespace FEXCore::Context {
FEX_DEFAULT_VISIBILITY void ConfigureAOTGen(FEXCore::Core::InternalThreadState *Thread, std::set<uint64_t> *ExternalBranches, uint64_t SectionMaxAddress);
FEX_DEFAULT_VISIBILITY CustomIRResult AddCustomIREntrypoint(FEXCore::Context::Context *CTX, uintptr_t Entrypoint, std::function<void(uintptr_t Entrypoint, FEXCore::IR::IREmitter *)> Handler, void *Creator = nullptr, void *Data = nullptr);
/**
* @brief Allows the frontend to register its own thunk handlers independent of what is controlled in the backend.
*
* @param CTX A valid non-null context instance.
* @param Definitions A vector of thunk definitions that the frontend controls
*/
FEX_DEFAULT_VISIBILITY void AppendThunkDefinitions(FEXCore::Context::Context *CTX, std::vector<FEXCore::IR::ThunkDefinition> const& Definitions);
}
+10 -3
View File
@@ -28,9 +28,16 @@ namespace FEXCore::Core {
uint64_t rip; ///< Current core's RIP. May not be entirely accurate while JIT is active
uint64_t gregs[16];
uint16_t es, cs, ss, ds;
uint64_t gs;
uint64_t fs;
// Raw segment register indexes
uint16_t es_idx, cs_idx, ss_idx, ds_idx;
uint16_t gs_idx, fs_idx;
uint16_t _pad[2];
// Segment registers holding base addresses
uint32_t es_cached, cs_cached, ss_cached, ds_cached;
uint64_t gs_cached;
uint64_t fs_cached;
uint64_t _pad2[1];
XMMRegs xmm;
uint8_t flags[48];
uint64_t mm[8][2];
+5
View File
@@ -21,7 +21,12 @@ class HostFeatures final {
bool SupportsRCPC{};
bool SupportsTSOImm9{};
bool SupportsRAND{};
bool Supports3DNow{};
bool SupportsSSE4A{};
bool SupportsAVX{};
bool SupportsSHA{};
bool SupportsBMI1{};
bool SupportsBMI2{};
// Float exception behaviour
bool SupportsFlushInputsToZero{};
+8
View File
@@ -58,6 +58,14 @@ namespace FEXCore {
FEATURE_XTILE_DATA = 1U << 18,
};
bool HasExtendedContext() const {
return magic1 == FP_XSTATE_MAGIC;
}
bool HasYMMH() const {
return (xfeatures & FEATURE_YMM) != 0;
}
// If magic1 is set to FP_XSTATE_MAGIC, then the encompassing
// frame is an xstate frame. If 0, then it's a legacy frame.
uint32_t magic1;
+7
View File
@@ -413,6 +413,13 @@ struct SHA256Sum final {
}
};
typedef void ThunkedFunction(void* ArgsRv);
struct ThunkDefinition final {
SHA256Sum Sum;
ThunkedFunction *ThunkFunction;
};
class NodeIterator;
/* This iterator can be used to step though nodes.
+8 -7
View File
@@ -5,6 +5,7 @@
#include <cstdint>
#include <functional>
#include <sys/types.h>
#include <vector>
namespace FEXCore::Allocator {
using MMAP_Hook = void*(*)(void*, size_t, int, int, int, off_t);
@@ -24,19 +25,19 @@ namespace FEXCore::Allocator {
FEX_DEFAULT_VISIBILITY void ClearHooks();
FEX_DEFAULT_VISIBILITY size_t DetermineVASize();
// 48-bit VA handling
struct PtrCache {
uint64_t Ptr;
uint64_t Size;
struct MemoryRegion {
void *Ptr;
size_t Size;
};
FEX_DEFAULT_VISIBILITY PtrCache* StealMemoryRegion(uintptr_t Begin, uintptr_t End);
FEX_DEFAULT_VISIBILITY void ReclaimMemoryRegion(PtrCache* Regions);
FEX_DEFAULT_VISIBILITY std::vector<MemoryRegion> StealMemoryRegion(uintptr_t Begin, uintptr_t End);
FEX_DEFAULT_VISIBILITY void ReclaimMemoryRegion(const std::vector<MemoryRegion> & Regions);
// When running a 64-bit executable on ARM then userspace guest only gets 47 bits of VA
// This is a feature of x86-64 where the kernel gets a full 128TB of VA space
// x86-64 canonical addresses with bit 48 set will sign extend the address (Ignoring LA57)
// AArch64 canonical addresses are only up to bits 48/52 with the remainder being other things
// Use this to reserve the top 128TB of VA so the guest never see it
// Returns nullptr on host VA < 48bits
FEX_DEFAULT_VISIBILITY PtrCache* Steal48BitVA();
FEX_DEFAULT_VISIBILITY std::vector<MemoryRegion> Steal48BitVA();
}
+55
View File
@@ -0,0 +1,55 @@
#pragma once
#include <cstdint>
#include <string_view>
#include <time.h>
#include <FEXCore/Utils/CompilerDefs.h>
namespace FEXCore::Profiler {
#ifdef ENABLE_FEXCORE_PROFILER
FEX_DEFAULT_VISIBILITY void Init();
FEX_DEFAULT_VISIBILITY void Shutdown();
FEX_DEFAULT_VISIBILITY void TraceObject(std::string_view const Format);
FEX_DEFAULT_VISIBILITY void TraceObject(std::string_view const Format, uint64_t Duration);
static inline uint64_t GetTime() {
// We want the time in the least amount of overhead possible
// clock_gettime will do a VDSO call with the least amount of overhead
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1'000'000'000ULL + ts.tv_nsec;
}
// A class that follows scoping rules to generate a profile duration block
class ProfilerBlock final {
public:
ProfilerBlock(std::string_view const Format);
~ProfilerBlock();
private:
uint64_t DurationBegin;
std::string_view const Format;
};
#define UniqueScopeName2(name, line) name ## line
#define UniqueScopeName(name, line) UniqueScopeName2(name, line)
// Declare an instantaneous profiler event.
#define FEXCORE_PROFILE_INSTANT(name) FEXCore::Profiler::TraceObject(name)
// Declare a scoped profile block variable with a fixed name.
#define FEXCORE_PROFILE_SCOPED(name) \
FEXCore::Profiler::ProfilerBlock UniqueScopeName(ScopedBlock_, __LINE__) (name)
#else
[[maybe_unused]] static void Init() {}
[[maybe_unused]] static void Shutdown() {}
[[maybe_unused]] static void TraceObject(std::string_view const Format) {}
[[maybe_unused]] static void TraceObject(std::string_view const, uint64_t) {}
#define FEXCORE_PROFILE_INSTANT(...) do {} while(0)
#define FEXCORE_PROFILE_SCOPED(...) do {} while(0)
#endif
}
+1 -1
+33 -43
View File
@@ -1,74 +1,64 @@
add_library(FEXHeaderUtils INTERFACE)
# Check for syscall support here
check_cxx_source_compiles(
"
#include <sched.h>
int main() {
return ::getcpu(nullptr, nullptr);
}"
"
#include <sched.h>
int main() {
return ::getcpu(nullptr, nullptr);
}"
compiles)
if (compiles)
message(STATUS "Has getcpu helper")
add_definitions(-DHAS_SYSCALL_GETCPU=1)
target_compile_definitions(FEXHeaderUtils INTERFACE HAS_SYSCALL_GETCPU=1)
endif ()
check_cxx_source_compiles(
"
#include <unistd.h>
int main() {
return ::gettid();
}"
"
#include <unistd.h>
int main() {
return ::gettid();
}"
compiles)
if (compiles)
message(STATUS "Has gettid helper")
add_definitions(-DHAS_SYSCALL_GETTID=1)
target_compile_definitions(FEXHeaderUtils INTERFACE HAS_SYSCALL_GETTID=1)
endif ()
check_cxx_source_compiles(
"
#include <signal.h>
int main() {
return ::tgkill(0, 0, 0);
}"
"
#include <signal.h>
int main() {
return ::tgkill(0, 0, 0);
}"
compiles)
if (compiles)
message(STATUS "Has tgkill helper")
add_definitions(-DHAS_SYSCALL_TGKILL=1)
target_compile_definitions(FEXHeaderUtils INTERFACE HAS_SYSCALL_TGKILL=1)
endif ()
check_cxx_source_compiles(
"
#include <sys/stat.h>
int main() {
return ::statx(0, nullptr, 0, 0, nullptr);
}"
"
#include <sys/stat.h>
int main() {
return ::statx(0, nullptr, 0, 0, nullptr);
}"
compiles)
if (compiles)
message(STATUS "Has statx helper")
add_definitions(-DHAS_SYSCALL_STATX=1)
target_compile_definitions(FEXHeaderUtils INTERFACE HAS_SYSCALL_STATX=1)
endif ()
check_cxx_source_compiles(
"
#include <stdio.h>
int main() {
return ::renameat2(0, nullptr, 0, nullptr, 0);
}"
"
#include <stdio.h>
int main() {
return ::renameat2(0, nullptr, 0, nullptr, 0);
}"
compiles)
if (compiles)
message(STATUS "Has renameat2 helper")
add_definitions(-DHAS_SYSCALL_RENAMEAT2=1)
endif ()
check_cxx_source_compiles(
"
#include <stdio.h>
#include <syscall.h>
int main() {
return ::syscall(SYS_pidfd_open, ::getpid(), 0);
}"
compiles)
if (compiles)
message(STATUS "Has pidfd_open helper")
add_definitions(-DHAS_SYSCALL_PIDFD_OPEN=1)
target_compile_definitions(FEXHeaderUtils INTERFACE HAS_SYSCALL_RENAMEAT2=1)
endif ()
target_include_directories(FEXHeaderUtils INTERFACE .)
+9 -8
View File
@@ -38,10 +38,15 @@ namespace FHU::Syscalls {
#endif
#endif
// Common syscall numbers
#ifndef SYS_pidfd_open
#define SYS_pidfd_open 434
#endif
inline int32_t getcpu(uint32_t *cpu, uint32_t *node) {
// Third argument is unused
#if defined(HAS_SYSCALL_GETCPU) && HAS_SYSCALL_GETCPU
return ::getcpu(cpu, node, nullptr);
return ::getcpu(cpu, node);
#else
return ::syscall(SYS_getcpu, cpu, node, nullptr);
#endif
@@ -57,7 +62,7 @@ inline int32_t gettid() {
inline int32_t tgkill(pid_t tgid, pid_t tid, int sig) {
#if defined(HAS_SYSCALL_GETTID) && HAS_SYSCALL_GETTID
return ::tgkill(tggid, tid, sig);
return ::tgkill(tgid, tid, sig);
#else
return ::syscall(SYS_tgkill, tgid, tid, sig);
#endif
@@ -65,7 +70,7 @@ inline int32_t tgkill(pid_t tgid, pid_t tid, int sig) {
inline int32_t statx(int dirfd, const char *pathname, int32_t flags, uint32_t mask, void *statxbuf) {
#if defined(HAS_SYSCALL_STATX) && HAS_SYSCALL_STATX
return ::statx(dirfd, pathname, flags, mask, statxbuf);
return ::statx(dirfd, pathname, flags, mask, reinterpret_cast<struct statx *__restrict>(statxbuf));
#else
return ::syscall(SYS_statx, dirfd, pathname, flags, mask, statxbuf);
#endif
@@ -80,11 +85,7 @@ inline int32_t renameat2(int olddirfd, const char *oldpath, int newdirfd, const
}
inline int32_t pidfd_open(pid_t pid, unsigned int flags) {
#if defined(DHAS_SYSCALL_PIDFD_OPEN) && DHAS_SYSCALL_PIDFD_OPEN
return ::syscall(SYS_pidfd_open, pid_t pid, unsigned int flags);
#else
return -1;
#endif
return ::syscall(SYS_pidfd_open, pid, flags);
}
}
+2 -2
View File
@@ -30,11 +30,11 @@ On AArch64 hosts the user **MUST** have an x86-64 RootFS [Creating a RootFS](#Ro
See the [Source Outline](docs/SourceOutline.md) for more information.
### Building FEX
Follow the guide on the official FEX-Emu Wiki [here](https://wiki.fex-emu.org/index.php/Development:Setting_up_FEX).
Follow the guide on the official FEX-Emu Wiki [here](https://wiki.fex-emu.com/index.php/Development:Setting_up_FEX).
### RootFS generation
AArch64 hosts require a rootfs for running applications.
Follow the guide on the wiki page for seeing how to set up the rootfs from scratch
https://wiki.fex-emu.org/index.php/Development:Setting_up_RootFS
https://wiki.fex-emu.com/index.php/Development:Setting_up_RootFS
![FEX diagram](docs/Diagram.svg)
+26 -9
View File
@@ -83,6 +83,12 @@ def HashFile(file):
return int.from_bytes(x.digest(), "big")
def RemoveRootFSFolder(RootFSPath):
print("Removing previous rootfs extraction before copying")
shutil.rmtree(RootFSPath, ignore_errors = True)
# Recreate the folder
os.makedirs(RootFSPath)
def CheckFilesystemForFS(RootFSMountPath, RootFSPath, DistroFit):
# Check if rootfs mount path exists
if (not os.path.exists(RootFSMountPath) or
@@ -105,6 +111,7 @@ def CheckFilesystemForFS(RootFSMountPath, RootFSPath, DistroFit):
MountRootFSImagePath = RootFSMountPath + DistroFit[3]
RootFSImagePath = RootFSPath + "/" + os.path.basename(DistroFit[3])
NeedsExtraction = False
PreviouslyExistingRootFS = False
if not os.path.exists(MountRootFSImagePath):
print("Image {} doesn't exist".format(MountRootFSImagePath))
@@ -113,29 +120,39 @@ def CheckFilesystemForFS(RootFSMountPath, RootFSPath, DistroFit):
if not os.path.exists(RootFSImagePath):
# Copy over
print("RootFS image doesn't exist. Copying")
shutil.copyfile(MountRootFSImagePath, RootFSImagePath)
NeedsExtraction = True
# Now hash the image
RootFSHash = HashFile(RootFSImagePath)
if RootFSHash != DistroFit[4]:
print("Hash {} did not match {}, copying new image".format(hex(RootFSHash), hex(DistroFit[4])))
RemoveRootFSFolder(RootFSPath)
shutil.copyfile(MountRootFSImagePath, RootFSImagePath)
NeedsExtraction = True
# Check if the image needs to be extracted
if not os.path.exists(RootFSPath + "/usr"):
NeedsExtraction = True
else:
PreviouslyExistingRootFS = True
# Now hash the image
RootFSHash = HashFile(RootFSImagePath)
if RootFSHash != DistroFit[4]:
print("Hash {} did not match {}, copying new image".format(hex(RootFSHash), hex(DistroFit[4])))
if PreviouslyExistingRootFS:
RemoveRootFSFolder(RootFSPath)
shutil.copyfile(MountRootFSImagePath, RootFSImagePath)
NeedsExtraction = True
if NeedsExtraction:
print("Extracting rootfs")
CmdResult = subprocess.call(["unsquashfs", "-f", "-d", RootFSPath, RootFSImagePath])
if CmdResult != 0:
print("Couldn't extract squashfs")
print("Couldn't extract squashfs. Removing image file to be safe")
os.remove(RootFSImagePath)
return False
if not os.path.exists(RootFSPath + "/usr"):
print("Couldn't extract squashfs")
print("Couldn't extract squashfs. Removing image file to be safe")
os.remove(RootFSImagePath)
return False
print("RootFS successfully checked and extracted")
+8 -5
View File
@@ -148,13 +148,16 @@ def HandleFunctionDeclCursor(Arch, Cursor):
elif (Child.kind == CursorKind.PARM_DECL):
# This gives us a parameter type
Function.Params.append(Child.type.spelling)
elif (Child.kind == CursorKind.UNEXPOSED_ATTR):
# Whatever you are we don't care about you
return Arch
elif (Child.kind == CursorKind.ASM_LABEL_ATTR):
# Whatever you are we don't care about you
return Arch
elif (Child.kind == CursorKind.VISIBILITY_ATTR):
elif (Child.kind == CursorKind.WARN_UNUSED_RESULT_ATTR):
# Whatever you are we don't care about you
return Arch
elif (Child.kind == CursorKind.VISIBILITY_ATTR or
Child.kind == CursorKind.UNEXPOSED_ATTR or
Child.kind == CursorKind.CONST_ATTR or
Child.kind == CursorKind.PURE_ATTR):
pass
else:
logging.critical ("\tUnhandled FunctionDeclCursor {0}-{1}-{2}".format(Child.kind, Child.type.spelling, Child.spelling))
@@ -165,7 +168,7 @@ def HandleFunctionDeclCursor(Arch, Cursor):
def PrintFunctionDecls():
for Decl in FunctionDecls:
print("fn(\"{0} {1}({2})\")".format(Decl.Ret, Decl.Name, ", ".join(Decl.Params)))
print("template<> struct fex_gen_config<{}> {{}};".format(Decl.Name))
def FindClangArguments(OriginalArguments):
AddedArguments = ["clang"]
+7 -1
View File
@@ -59,7 +59,13 @@ SyscallArm64File = "/include/uapi/asm-generic/unistd.h"
DefinitionRenameDict = {
"pread64": "pread_64",
"pwrite64": "pwrite_64",
"prlimit64": "prlimit_64"
"prlimit64": "prlimit_64",
# Shm symbols conflict with termux defines and FEX's syscall token pasting.
# Underscore at the start to avoid name collision
"shmget": "_shmget",
"shmctl": "_shmctl",
"shmat": "_shmat",
"shmdt": "_shmdt",
}
Definitions_x64 = []
+3 -1
View File
@@ -306,12 +306,14 @@ def GetRootFSPath():
return _RootFSPath
def CheckRootFSInstallStatus():
# Matches what is available on https://rootfs.fex-emu.org/file/fex-rootfs/RootFS_links_XXH3.txt
# Matches what is available on https://rootfs.fex-emu.com/file/fex-rootfs/RootFS_links.json
UbuntuVersionToRootFS = {
"20.04": "Ubuntu_21_04.sqsh",
"21.04": "Ubuntu_21_04.sqsh",
"21.10": "Ubuntu_21_10.sqsh",
"21.10": "Ubuntu_21_10.ero",
"22.04": "Ubuntu_22_04.sqsh",
"22.04": "Ubuntu_22_04.ero",
}
return os.path.exists(GetRootFSPath() + UbuntuVersionToRootFS[GetDistro()[1]])
+25 -16
View File
@@ -10,6 +10,18 @@ import logging
logger = logging.getLogger()
logger.setLevel(logging.WARNING)
# These defines are temporarily defined since python3-clang doesn't yet support these.
# Once this tool gets switched over to C++ then this won't be an issue.
# Expression that references a C++20 concept.
CursorKind.CONCEPTSPECIALIZATIONEXPR = CursorKind(153),
# C++2a std::bit_cast expression.
CursorKind.BUILTINBITCASTEXPR = CursorKind(280)
# a concept declaration.
CursorKind.CONCEPTDECL = CursorKind(604),
@dataclass
class TypeDefinition:
TYPE_UNKNOWN = 0
@@ -268,7 +280,7 @@ def HandleTypeDefDeclCursor(Arch, Cursor):
if (len(TypeDefName) != 0):
HandleTypeDefDecl(Arch, Cursor, TypeDefName)
# Append namespace
# Append namespace
Arch.NamespaceScope.append(TypeDefName)
SetNamespace(Arch)
@@ -404,19 +416,20 @@ def HandleCursor(Arch, Cursor):
return
for Child in Cursor.get_children():
if (Child.kind == CursorKind.TRANSLATION_UNIT):
kind = Child.kind
if (kind == CursorKind.TRANSLATION_UNIT):
Arch = HandleCursor(Arch, Child)
elif (Child.kind == CursorKind.FIELD_DECL):
elif (kind == CursorKind.FIELD_DECL):
pass
elif (Child.kind == CursorKind.UNION_DECL):
elif (kind == CursorKind.UNION_DECL):
Arch = HandleUnionDeclCursor(Arch, Child)
elif (Child.kind == CursorKind.STRUCT_DECL):
elif (kind == CursorKind.STRUCT_DECL):
Arch = HandleStructDeclCursor(Arch, Child)
elif (Child.kind == CursorKind.TYPEDEF_DECL):
elif (kind == CursorKind.TYPEDEF_DECL):
Arch = HandleTypeDefDeclCursor(Arch, Child)
elif (Child.kind == CursorKind.VAR_DECL):
elif (kind == CursorKind.VAR_DECL):
Arch = HandleVarDeclCursor(Arch, Child)
elif (Child.kind == CursorKind.NAMESPACE):
elif (kind == CursorKind.NAMESPACE):
# Append namespace
Arch.NamespaceScope.append(Child.spelling)
SetNamespace(Arch)
@@ -427,7 +440,7 @@ def HandleCursor(Arch, Cursor):
# Pop namespace off
Arch.NamespaceScope.pop()
SetNamespace(Arch)
elif (Child.kind == CursorKind.TYPE_REF):
elif (kind == CursorKind.TYPE_REF):
# Safe to pass on
pass
else:
@@ -638,25 +651,21 @@ def main():
BaseArgs.append(sys.argv[ArgIndex])
args_x86_32 = [
"-I/usr/i686-linux-gnu/include/c++/10/i686-linux-gnu/",
"-I/usr/i686-linux-gnu/include/",
"-I/usr/i686-linux-gnu/include",
"-O2",
"-m32",
"--target=i686-linux-unknown",
]
args_x86_64 = [
"-I/usr/include/x86_64-linux-gnu",
"-I/usr/x86_64-linux-gnu/include/c++/10/x86_64-linux-gnu/",
"-I/usr/x86_64-linux-gnu/include/",
"-I/usr/x86_64-linux-gnu/include",
"-O2",
"--target=x86_64-linux-unknown",
"-D_M_X86_64",
]
args_aarch64 = [
"-I/usr/aarch64-linux-gnu/include/c++/10/aarch64-linux-gnu/",
"-I/usr/aarch64-linux-gnu/include/",
"-I/usr/aarch64-linux-gnu/include",
"-O2",
"--target=aarch64-linux-unknown",
"-D_M_ARM_64",
+59 -36
View File
@@ -3,44 +3,55 @@ import os
import sys
import subprocess
# Args: <Known Failures file> <ExpectedOutputsFile> <DisabledTestsFile> <TestName> <FexExecutable> <FexArgs>...
def LoadTestsFile(File):
Dict = {}
if not os.path.exists(File):
return Dict
with open(File) as dtf:
for line in dtf:
test = line.split("#")[0].strip() # remove comments and empty spaces
if len(test) > 0:
Dict[test] = 1
return Dict
def LoadTestsFileResults(File):
Dict = {}
if not os.path.exists(File):
return Dict
with open(File) as dtf:
for line in dtf:
test = line.split("#")[0].strip() # remove comments and empty spaces
if len(test) > 0:
parts = line.split(" ")
Dict[parts[0]] = int(parts[1])
return Dict
# Args: <Known Failures file> <ExpectedOutputsFile> <DisabledTestsFile> <FlakeTestsFile> <TestName> <Mode> <FexExecutable> <FexArgs>...
# fexargs should also include the test executable
if (len(sys.argv) < 6):
if (len(sys.argv) < 7):
sys.exit()
known_failures_file = sys.argv[1]
expected_output_file = sys.argv[2]
disabled_tests_file = sys.argv[3]
test_name = sys.argv[4]
mode = sys.argv[5]
fexecutable = sys.argv[6]
flake_tests_file = sys.argv[4]
test_name = sys.argv[5]
mode = sys.argv[6]
fexecutable = sys.argv[7]
StartingFEXArgsOffset = 8
known_failures = { }
expected_output = { }
disabled_tests = { }
# Open the known failures file and add it to a dictionary
with open(known_failures_file) as kff:
for line in kff:
test = line.split("#")[0].strip() # remove comments and empty spaces
if len(test) > 0:
known_failures[test] = 1
# Open expected outputs and add it to dictionary
with open(expected_output_file) as eof:
for line in eof:
line = test = line.split("#")[0].strip() # remove comments and empty spaces
if len(line) > 0:
parts = line.split(" ")
expected_output[parts[0]] = int(parts[1])
with open(disabled_tests_file) as dtf:
for line in dtf:
test = line.split("#")[0].strip() # remove comments and empty spaces
if len(test) > 0:
disabled_tests[test] = 1
# Open test expected information files and load in to dictionaries.
known_failures = LoadTestsFile(known_failures_file)
expected_output = LoadTestsFileResults(expected_output_file)
disabled_tests = LoadTestsFile(disabled_tests_file)
flake_tests = LoadTestsFile(flake_tests_file)
# run with timeout to avoid locking up
RunnerArgs = []
@@ -54,25 +65,37 @@ if (mode == "guest"):
RunnerArgs.append(ROOTFS_ENV)
# Add the rest of the arguments
for i in range(len(sys.argv) - 7):
RunnerArgs.append(sys.argv[7 + i])
for i in range(len(sys.argv) - StartingFEXArgsOffset):
RunnerArgs.append(sys.argv[StartingFEXArgsOffset + i])
#print(RunnerArgs)
ResultCode = 0
# Handle flakes
TryCount = 1
if (flake_tests.get(test_name)):
TryCount = 5
if (disabled_tests.get(test_name)):
ResultCode = -73
else:
# Run the test and wait for it to end to get the result
Process = subprocess.Popen(RunnerArgs)
Process.wait()
ResultCode = Process.returncode
# expect zero by default
if (not test_name in expected_output):
expected_output[test_name] = 0
if ResultCode == 0:
for Try in range(TryCount):
# Run the test and wait for it to end to get the result
print(RunnerArgs)
Process = subprocess.Popen(RunnerArgs)
Process.wait()
ResultCode = Process.returncode
# Break if the expected output is the result code
if (expected_output[test_name] == ResultCode):
break
if (expected_output[test_name] != ResultCode):
if (test_name in expected_output):
print("test failed, expected is", expected_output[test_name], "but got", ResultCode)
+27 -9
View File
@@ -63,8 +63,15 @@ class Mode(Flag) :
MODE_64 = 1
class HostFeatures(Flag) :
ANY = 0
AVX_ONLY = 1
FEATURE_ANY = 0
FEATURE_3DNOW = (1 << 0)
FEATURE_SSE4A = (1 << 1)
FEATURE_AVX = (1 << 2)
FEATURE_RAND = (1 << 3)
FEATURE_SHA = (1 << 4)
FEATURE_CLZERO = (1 << 5)
FEATURE_BMI1 = (1 << 6)
FEATURE_BMI2 = (1 << 7)
RegStringLookup = {
"NONE": Regs.REG_NONE,
@@ -128,8 +135,14 @@ ModeStringLookup = {
}
HostFeaturesLookup = {
"ANY" : HostFeatures.ANY,
"AVX" : HostFeatures.AVX_ONLY,
"3DNOW" : HostFeatures.FEATURE_3DNOW,
"SSE4A" : HostFeatures.FEATURE_SSE4A,
"AVX" : HostFeatures.FEATURE_AVX,
"RAND" : HostFeatures.FEATURE_RAND,
"SHA" : HostFeatures.FEATURE_SHA,
"CLZERO" : HostFeatures.FEATURE_CLZERO,
"BMI1" : HostFeatures.FEATURE_BMI1,
"BMI2" : HostFeatures.FEATURE_BMI2,
}
def parse_hexstring(s):
@@ -152,7 +165,7 @@ def parse_json(json_text, output_file):
OptionIgnore = Regs.REG_NONE
OptionABI = ABI.ABI_SYSTEMV
OptionMode = Mode.MODE_64
OptionHostFeatures = HostFeatures.ANY
OptionHostFeatures = HostFeatures.FEATURE_ANY
OptionStackSize = 4096
OptionEntryPoint = 1
OptionRegData = {}
@@ -208,10 +221,15 @@ def parse_json(json_text, output_file):
if ("HOSTFEATURES" in json_object):
data = json_object["HOSTFEATURES"]
data = data.upper()
if not (data in HostFeaturesLookup):
sys.exit("Invalid host feature")
OptionHostFeatures = HostFeaturesLookup[data]
if not (type(data) is list):
sys.exit("HostFeatures value must be list of features")
for data_key in data:
data_key = data_key.upper()
if not (data_key in HostFeaturesLookup):
sys.exit("Invalid host feature")
OptionHostFeatures |= HostFeaturesLookup[data_key]
if ("STACKSIZE" in json_object):
data = json_object["STACKSIZE"]
+13 -7
View File
@@ -4,7 +4,7 @@ import subprocess
import os.path
from os import path
# Args: <Known Failures file> <DisabledTestsFile> <DisabledTestsTypeFile> <DisabledTestsRunnerFile> <TestName> <Test Harness Executable> <Args>...
# Args: <Known Failures file> <Known Failures Type File> <DisabledTestsFile> <DisabledTestsTypeFile> <DisabledTestsRunnerFile> <TestName> <Test Harness Executable> <Args>...
if (len(sys.argv) < 7):
sys.exit()
@@ -12,19 +12,25 @@ if (len(sys.argv) < 7):
known_failures = {}
disabled_tests = {}
known_failures_file = sys.argv[1]
disabled_tests_file = sys.argv[2]
disabled_tests_type_file = sys.argv[3]
disabled_tests_runner_file = sys.argv[4]
known_failures_type_file = sys.argv[2]
disabled_tests_file = sys.argv[3]
disabled_tests_type_file = sys.argv[4]
disabled_tests_runner_file = sys.argv[5]
current_test = sys.argv[5]
runner = sys.argv[6]
args_start_index = 7
current_test = sys.argv[6]
runner = sys.argv[7]
args_start_index = 8
# Open the known failures file and add it to a dictionary
with open(known_failures_file) as kff:
for line in kff:
known_failures[line.strip()] = 1
if path.exists(known_failures_type_file):
with open(known_failures_type_file) as dtf:
for line in dtf:
known_failures[line.strip()] = 1
with open(disabled_tests_file) as dtf:
for line in dtf:
disabled_tests[line.strip()] = 1
Loaded 100 of 333 files, more files were not shown because too many files have changed in this diff. Show more